Do literature reviews mislead?

Do literature reviews mislead end users? Our answer to this question is: yes, and often. Moreover, no one seems to be paying much attention to this fact, at least not amongst the researchers and end users involved in fire and emergency management in Australia.

We believe it’s time for the humble and familiar literature review to get some direct and critical scrutiny in this sector. To help kick it off, we recently wrote a discussion paper on the topic. The paper outlines some of the problems with the traditional approach to literature reviews from both research and end user perspectives. It then discusses some of the things that other sectors are doing to move past these problems.

We’d also like to hear what others think about this issue, both researchers and end users.

As grist for the discussion mill, consider a key argument made in the discussion paper. Summarised, it is this: because of a lack of close attention to the way they are carried out, literature reviews reveal as much about the particular views and interests of the reviewer as they do about the status of current scientific knowledge.

Why is this? In large part it is a by-product of the low status that researchers (and their universities, funding agencies and publishers) assign to literature reviews. Reviews are seen as ‘just’ summaries of other people’s work rather than as potentially important and legitimate pieces of integrative research in their own right. Because of this, the methodological attention that is demanded of all primary research is almost completely absent when it comes to literature reviews. End users might be surprised to hear that, as a general rule, researchers get very little direct training or guidance on how to review literature. It is also highly uncommon - at least in this sector - for a researcher to describe how they conducted a literature review. In our experience, when researchers do attempt to describe the methodology used to conduct a major literature review, editors, peer reviewers and end users alike consider it extraneous. We can only assume that because these parties see literature reviews as stock standard (and not ‘real’ research anyway) they consider the review process to be more or less self-explanatory and uniform.

But we would argue that the process of conducting a literature review is anything but standard. Most are idiosyncratic and highly selective given the amount of material they cover. In reality, a reviewer makes numerous choices about what studies to include (and exclude), and how to evaluate and integrate their findings. The underlying problem with this – one that is generally overlooked by everyone involved – is that the reviewer brings their own perspectives, knowledge and interests to bear on these choices. This influences what research is covered in the review and how it is interpreted and communicated. As a result, two reviews on the same topic conducted by two different researchers can cover different literature and reach very different conclusions.

What does this mean for end users? Most importantly, without an awareness of how a literature review was carried out and what choices the reviewer made (and why), end users have no basis for evaluating the review’s quality, partiality or relevance for their own practical purposes. Without this, they have three general options. They can trust in the reviewer’s perspective of what research counts; uncritically accept the review as a true, accurate and complete assessment; or regard it as a possibly interesting but generally partial and uninfluential product. Unfortunately, none of these options can substitute for the value end users can get from an informed evaluation of a thorough, well-constructed and transparent literature review.

Authors: Blythe McLennan and John Handmer, Bushfire CRC researchers at RMIT University.

See also:

McLennan B. and Handmer, J. 2011. Reviewing research for policy making and practice: a discussion paper for the Australian emergency management industry. RMIT University and the Bushfire CRC.