What to do now that the references for inclusion in the systematic review have been selected?

You are now nearly ready to write up the systematic review. You have reviewed the full text versions of any references that looked promising for inclusion, culling out those that were inappropriate (remembering to record reasons and numbers). You have read the references that have been selected and are sitting down ready to write about the subject.

For an initial systematic review, it is acceptable to tabulate the relevant data and discuss it, without performing any further analysis. This is a purely descriptive approach and although you will provide your writing groups opinions on the results, you may not want to do any further analysis. However, to get the most out of a systematic review you will probably want to go further. This could involve one of two approaches, either a meta-analysis or a narrative synthesis.

What is a meta-analysis?

If the references detail studies that are similarly designed, with similar duration, populations and study drug/intervention use, then a meta-analysis is possible. Meta-analysis are typically used in medicine and are a statistical approach to estimating the treatment effect. By combining a number of different studies, larger populations can be explored and this can provide insight into rare complications or adverse events, or provide a better overview of clinical efficacy. A good introduction to what a meta-analysis is can be found here.

What is a narrative synthesis?

If the references all include very small study populations, or the study designs are different, for example the study lengths vary widely, then a narrative synthesis may be more appropriate. This approach involves the identification of themes and trends in the research, highlighting aspects that appear to influence outcomes or are important for the intervention. A good introduction to what a narrative synthesis is can be found here.

Do you have any tips of writing a good systematic review?

First off, write the manuscript with the PRISMA checklist next to you, to ensure that all the information that it requires is included. Also, even though it may appear that the conduct of a systematic review is rigid and does not allow any creativity it is important that the final manuscript is not overly dry, otherwise no-one will read it and it is wasted effort. Ensure that the story of why the systematic review was conducted is included and that each step or question builds logically from the previous one. You do not want to make any sudden jumps, as although you may find them intuitive, your readers quite likely will not. They have not been immersed in the review and discussions around it. To avoid this, you should get a colleague who has not been involved in any discussions about the systematic review at any point to review a near-final draft. They are likely to identify any gaps in the narrative and you should listen to them and if you disagree you discuss your reasons with that person. Also, don’t forget to thank them in the acknowledgements.

What should happen after the search is complete, should we immediately start writing the systematic review?

Once you have done the first version of the search, you may find that you have too many results or not many results at all.

Too many results?

If you have too many results it is likely that your search was too broad and that rather than manually checking each reference, using the filters provided by the search engines could help refine the results. Some possibilities for refining the search are to limit it to a specific language (English, for example), or to just searching the abstract and title, rather than the full text.

Too few results?

This might be what you expected and could be the basis for the systematic review, highlighting the need for more studies in a particular area. However, you may want to broaden your search if you feel that not all relevant manuscripts have been captured. This could be done by including additional keywords, or removing limits (e.g. looking for all studies and not just those in humans).

What if the number of results seems fine?

Once you are happy that the search has captured all the appropriate results, you will want to save a list of the results, including references, title and abstract, as well as recording the initial number of results found. You will need to keep track of this for the PRISMA flowchart. These results should then be distributed to the individuals that have been allocated as reviewers and they should review each abstract and decide whether it meets the inclusion criteria defined for the review. These criteria should have been defined before the search and it is important that both reviewers use them. The reason why each abstract is rejected should also be recorded. This can be done according to the PICOS, with abstracts rejected according to not containing appropriate participants, intervention, comparator, outcome or setting, or for another reason which should be recorded.

In a perfect world both reviewers will select the same abstracts at this stage, however, if they do not, an additional reviewer should arbitrate whether the abstract should be kept. If there is confusion, the abstract should be kept, for full text review (the next step).

After the abstracts have been filtered, should we start writing?

No. You will want to get the full version of each reference and these will also need to be reviewed, in a similar manner to the abstracts, to ensure that no inappropriate references are included. At this stage, you can note any manuscripts in the references that also look appropriate for the systematic review. This will enable you to check that they have been included and if not they can be added (and noted on the PRISMA flowchart).

This process of filtering can be long and tedious, however, it should enable all the relevant literature for the systematic review to be identified. It is at this stage that the writing can commence!

What next for your systematic review – Fully define your research hypothesis or question.

So you have decided that you are going to put together a systematic review and you have managed to persuade some of your colleagues and peers to be co-authors. What might you do next to get the review done?

Have a meeting.

The first thing I would recommend is that you get everyone who is to be involved together to define the problem. This is important because everybody will come to the table with different mind sets and different backgrounds, and also prior knowledge. Together you can make sure that everyone is agreed as to what problem will be addressed.

However, you should not go beyond that in the meeting, rather once the problem has been clarified you should end the meeting and get everybody to think of what hypothesis or question your review will be addressing. This way you will not end up with just the point of view of the most respected author or the person who shouts the loudest. Also, I find that my best ideas come when I am not really thinking directly about them and by leaving time to think you provide the opportunity for people to synthesise what was discussed and come up with any solutions or points that need clarification that were missed during the meeting.

This thinking time should not be extensive and should be defined during the meeting by booking the next meeting in for a couple of days later.

Have another meeting

Everybody has hopefully now thought about the issues and come up with ideas for the hypothesis. At this second meeting you will review everybody’s ideas. Let each person present their ideas in turn without allowing any questions to ensure that people are not intimidated and only present part of what they have come up with.

Once everybody’s ideas have been presented you can then discuss the merits of them all and finesse the hypothesis until you have it agreed by all authors. At this point you will need to discuss any limits people feel will be necessary for the systematic review, will it be limited for example by date, inclusion criteria, size or specific endpoint measures. These limits will be very important once the search has been run as you will typically need to filter huge numbers of abstracts to determine the few that will be included and as suggested by the type of review your process needs to be systematic.

What next?

The next step, once the hypothesis or question has been defined, is to define the search and that will be discussed in the next post.

References

I have taken a lot of the ideas for this post from an Accidental Creative podcast episode I listened to on brainstorming. This episode can be found here and is well worth a listen.

Some thoughts on systematic reviews

I remember that in my first year of university (in 2000) we were taken to the library and shown how to search the Beilstein Handbook of Organic Chemistry to identify previous instances of reactions we might be interested in. There was a wall of books and it was quite intimidating. However, I never had to look at those books because there was also an online searchable database which made the task much easier. I cannot imagine how much time must have been spent by chemists searching for reactions and reaction conditions, that was suddenly freed up by taking the data online.

This has happened with most data sources and we can now search through records of journal publications and patents with ease. This I believe has led to the increase in the publication of the systematic review and the decrease in the importance placed on narrative reviews. It has changed so much that, nowadays, before many research projects are fully initiated, a systematic review is undertaken to explore, among other things, what is already known about the topic and whether the research project is novel. Owing to their inclusion of all available, relevant data systematic reviews are considered the pinnacle of evidence in evidence-based medicine [Wikipedia]. This is, in my mind, embodied by the Cochrane Collaboration, which has a handbook on how to perform systematic reviews for inclusion in their database.

It is, therefore, important that all researchers understand what a systematic review is and how to perform one. In addition, it is important to know what to do with the output. Is it okay to summarize the data, or should it be synthesised in some way, either through meta-analysis or narrative synthesis. Furthermore, we need to be able to judge the quality of published systematic reviews so that we can judge their quality and determine whether there is any bias inherent in them, or flaws in the methods used.

The PRISMA Statement

The PRISMA Statement is a great help, as it details everything to be included in a systematic review and its planning and I have found this useful when planning, performing and writing up systematic reviews, as it provides a useful checklist to bear in mind.

I want to start my systematic review. What should I do?

The very first thing you need to do is ensure you are asking an interesting question that is relevant and that you are asking it in the correct fashion. You should discuss your idea with colleagues who work in the same field and have similar interests, and also read any relevant papers that you are aware off. This might highlight aspects that you are unaware of or have overlooked. It will also provide the groundwork for the next step, which is putting together your hypothesis and developing your search strategy which will identify the references that will potentially be included in the review. You should also work out who your co-authors will be and each person’s role. For example, you will need somebody to do the search and at least two people to go through the results separately and someone to adjudicate any conflicting views about whether or not to include a manuscript in the review.

Once this is done you are ready to think about actually starting work on your systematic review, and I shall offer my thoughts on those steps in the next few posts.