A really interesting discussion here from Patrick Dunleavy:
There are also now some very specific and increasingly influential methods for re-aggregating and re-understanding what whole literatures tell us. ‘Systematic review’ is an especially key approach now across the social sciences, spreading in from medicine and the health sciences. It starts by the reviewer clearly delineating (i) the subject, focus and boundaries of the review; and (ii) explicit criteria to be applied in evaluating sources and texts as being high value, medium value or low value. The review begins by considering the full pool of sources, leaving nothing out. The reviewer then systematically applies the criteria already set out, filtering down the studies progressively to focus on the higher value materials. Within the high value studies alone, those with the best evidence or methods employed, the systematic review considers what exactly are the strength of any evidential connections made and tries to reconcile as far as possible any divergences in estimating effects across the high value studies. Finally the conclusions (a) sum up the central findings; (b) make clear and assess the level of evidence and weight of the materials underpinning the findings; (c) give a sensitivity analysis of how far the conclusions might differ if the assessment criteria used for focusing down on key studies had been different.
We face a problem here though. The disciplines which would benefit most from systematic review are the ones least likely to have clearly agreed upon standards about what constitutes ‘better’ or ‘worse’ scholarship. Without this, the systematic review then starts to involve precisely the contentious implicit judgements which it’s intended to address. Nonetheless, I find it hard to deny the value here. How could we encourage more systematic review in Sociology?