Just before Christmas, Megan McArdle posted interesting article the Atlantic blog site (here) explaining why  “promising pilot projects often don’t scale “.  She argues that pilot programs often “don’t scale for corporations, and they don’t scale for government agencies.” She continues that   “even when you put super smart people with expert credentials in charge of them” they don’t scale.  And “even when you make sure to provide ample budget resources,” success is not guaranteed.

She then goes on to offer more specific reason as to why “Rolling something out across an existing system is substantially different from even a well run test, and often, it simply doesn’t translate.”

1. Sometimes the “success” of the earlier project was simply a result of random chance, or what researchers call the Hawthorne Effect.  The effect is named after a factory outside of Chicago which ran tests to see whether workers were more productive at higher or lower levels of light.  When researchers raised the lights, productivity went up.  When researchers lowered the lights, productivity also went up.  Obviously, it wasn’t the light that boosted productivity, but something else–the change from the ordinary, or the mere act of being studied.
2. Sometimes the success was due to what you might call a “hidden parameter”, something that researchers don’t realize is affecting their test.   Remember the New Coke debacle?  That was not a hasty, ill-thought out decision by managers who didn’t care about their brand.  They did the largest market research study in history, and repeated it several times, before they made the switch.  People invariably told researchers they loved the stuff.  And they did, in the taste test.  But they didn’t love the stuff when it cost them the option of drinking old Coke.
3. Sometimes the success was due to the high quality, fully committed staff … and [doesn’t work when we are] recruiting from a pool that included folks who were just looking for a job, not a life’s mission to … while adding to the sum of human knowledge.
4. Sometimes the program becomes unmanageable as it gets larger.
5. Sometimes the results are survivor bias.  This is an especially big problem with studying health care, and the poor. Health care, because compliance rates are quite low (by one estimate I heard, something like 3/4 of the blood pressure medication prescribed is not being taken 9 months in) and the poor, because their lives are chaotic and they tend to move around a lot, so they may have to drop out, or may not be easy to find and re-enroll if they stop coming.  In the end, you’ve got a study of unusually compliant and stable people (who may be different in all sorts of ways) and oops! that’s not what the general population looks like.

So what does this mean for our life in Christ? Continue reading →