Since the last study design update, further administrative challenges have arisen that require a re-think of the design of the proposed study and how such design will be operationalised.

To date, there is still no contractual agreement in place between the different organisations involved in the study1. This means that there is no binding agreement that will enable the allocation and disbursement of funds needed to fully start-up the study.
 

Implications

As per previous update, the baseline data collection has been earmarked to start on the second week of April 2016 (around the 9th or 10th of April 2016). This is now untenable as there are no available resources that can be used to start the baseline.

There are two alternatives that can address this issue.

First is to push the data collection plan by another month. This would mean conducting baseline in May instead of April (one month later than planned) and then subsequent data collection for each step of the study at 2-month intervals. Keeping the number of steps at 42, this would mean having the last data collection round for the stepped wedge study at the end of January 2016. This scenario will not involved any sample size increase as compared to previous designs that had a baseline data collection along with 4 steps of data collection. However, this scenario will not be feasible, as this will significantly reduce the time allocated for data analysis given that a March 2016 deadline has been specified by 3ie. Requesting an extension of deadline is most likely not an option because the current timeframe we have is already based on a two-month extension that has been requested by Valid International given the previous delays experienced.

Second option is to drop the baseline altogether and keep the start of interventions in the three clusters needing to be implemented in by WFP to May 2016. We would then start with incidence data collection by the first week of May 2016 for 2 weeks and then continue with the second round of incidence data collection for June and then have the first round of stepped wedge data collection by the end of June 2016. Keep the number of steps to 4, this would mean that the final stepped wedge data collection will be on the last 2 weeks of December 2016. This option will roughly maintain the amount of time previously allocated for data analysis and will ensure deliver out outputs to 3ie by the March 2016 deadline. However, dropping baseline has a sample size implication. It should be remembered that a baseline round has two benefits to the study. It reduces the overall study sample size requirement and per cluster sample size requirement. Also, it increases the power of the study to detect variances and differences. In general, a baseline makes the study so much stronger and better. Losing baseline would require a relevant increase in sample size to make up for the variance lost by giving up baseline. The sample size increase can be seen in the latest sample size estimates.
 

Limitations of current design

Though this current re-design is the most plausible option given the various issues faced with the starting up of this study, it should also be noted that it has limitations and there are still other issues that will not be solved by this current re-design.

  1. Given that WFP needs to implement in three specific localities in Kassala (i.e., Telkuk, Aroma and North Delta) first, the randomisation of rollout into the different clusters is still not going to be possible. So, this study loses its experimental nature (though it is still of a higher order of evaluation approach given that it is as close to experimental as it can be). However, this was something that the investigators knew to be difficult in a practical project implementation setting to begin with.
  2.  

  3. The number of clusters in each step is highly uneven. The current design intends to rollout in 3 clusters in the first step, 1 cluster in the second step, 1 cluster in the third step and 2 clusters in the fourth step. Whilst we tried to address this issue in the most recent re-design, the implication has been that we had to reduce the number of steps to 3 so that we can balance out the number of clusters in each step without having to increase the number of clusters to include in the study. However, as already noted earlier, reducing the study to three steps instead of four would reduce the number of data collection rounds for the incidence study to four from the six rounds that we originally planned for. Reducing to four rounds of data collection for the incidence study has its own sample size implications that at this point we will be unable to account for given resource limitations. Balancing everything else, we can deal with the unevenness in the cluster allocation for each step much reasonably as compared to having to deal with the resource implications of having to either add more clusters to the study or having to reduce the number of steps to three. The main setback with uneven cluster allocation is that the samples obtained in each step will most likely be very uneven as well. This will be most pronounced in the samples for steps 2 and 3 where there will only be 1 cluster each. We can most likely account for this in the statistical model that will be used for data analysis.
  4.  

  5. Related to point 2 above and after review of the number of villages within the study clusters in steps 2 and 3 (El Girba and River Atbara localities) while performing stage 1 spatial sampling, it will be likely that we will have to exhaustively sample both localities to ensure adequate sample size for each. Originally, we have had these two localities form a single study cluster as we have anticipated the relatively low number of potential sampling villages in each. The implication of this would be a relatively higher data collection overhead in each of these localities.
  6.  

  7. Because baseline data collection will not be done anymore, questions for specific indicator sets will need to be reconsidered as they were planned to have been collected only during baseline and during the last data collection step (step 4)3. These questions are for the multi-dimensional poverty index (MPI). These question sets will most likely have to be taken out from the data collection. The potential benefit of this is that the stepped wedge questionnaire will be a lot quicker to administer as a significant number of questions4 will not be included anymore.

Despite these limitations, this current re-design is our best option given the current circumstances and is still the most suited approach to evaluating the impact of the WFP programme. As such, we think this is the best way forward.
 
 
 

Endnotes

1 As of 7th April 2016, relevant MoU have been agreed in principle but are still undergoing finalisation in preparation for signatures. Once signatures are in place, it will still take some time for the details of the agreement, specifically those pertaining to financials, to be implemented and to take effect. Realistically, we are looking at a 2 week window before the appropriate administrative and financial structures and systems to be in place and operational.

2 The most recent re-design has shown the weakness of just doing 3 steps instead of the original 4 steps planned. As can be noted, three steps of the stepped wedge would mean a reduction of 2 months in the incidence study data collection (down to 4 months only instead of 6 months). Hence, we are now insisting that we keep a minimum of 4 steps in the stepped wedge study to ensure a minimum of 6 rounds of data collection for the incidence study.

3 These indicators are not seasonal and it takes some time before any of these indicators show any significant change if any. Measuring these indicators frequently over a period of time would therefore not be useful.

4 These questions include the question set for household roster which includes education attained by each household member, household assets, house structure characteristics and short birth history.