• By Dartington SRU
• Posted on Thursday 29^th May, 2014

The proof of the pudding is in the pudding

A good cookbook not only lists ingredients and quantities but also goes into the detail of when to add them and describes minutely how they should be combined.

Why? Simply because it makes a difference to the outcome; in the case of good eating, to the taste and texture of the meal. Much the same goes for evidence-based programmes: the detail of the implementation matters.

The reasons why programmes fail in the real world often have more to do with how they were implemented than with any inherent weakness in the recipe. Meta-analysis, the study of studies on a specific topic, shows that implementation is the most important feature where the success or otherwise of programme outcomes is concerned.

Clinical psychologist Joseph Durlak at Loyola University, Chicago, and his colleagues found as much when they examined over 500 research studies of evidence-based programmes. There is credible and extensive empirical evidence that the level of implementation affects the programme outcomes, they report.

They also found that positive results were achieved when an implementation level of between 60-80% was reached. Just as well, since no study scored near-perfect implementation by their measures.

Influential factors were found in every aspect of the implementation process. At the community level, for example, political support and adequate funding were important. As predictably perhaps, implementers who acknowledged the need for a particular programme and believed it worked were more likely to do the job well.

In the same vein, programmes that fitted the implementing organisation's mission and values and also possessed some inbuilt flexibility were more likely to be implemented faithfully.

Durlak's verdict on fidelity confirms previous findings about the balancing importance of adaptability. It is true, he argues, that higher levels of fidelity are directly linked to better programme outcomes, but since no programme is implemented with 100% fidelity, some scope for intentional adaptation needs to be allowed.

The bottom line is that adaptation should not tamper with the core components of the programme. Hence the importance of monitoring. Efficient monitoring was consistently linked with better outcomes; where it occurred effect sizes were as much as three times larger.

So measuring the effectiveness of implementation and establishing a feedback loop with the goal of improving implementation quality is becoming an increasingly familiar requirement.

Durlak J A, and DuPre E P (2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation American Journal of Community Psychology, 41, pp 327-350.