What should happen after the search is complete, should we immediately start writing the systematic review?

Once you have done the first version of the search, you may find that you have too many results or not many results at all.

Too many results?

If you have too many results it is likely that your search was too broad and that rather than manually checking each reference, using the filters provided by the search engines could help refine the results. Some possibilities for refining the search are to limit it to a specific language (English, for example), or to just searching the abstract and title, rather than the full text.

Too few results?

This might be what you expected and could be the basis for the systematic review, highlighting the need for more studies in a particular area. However, you may want to broaden your search if you feel that not all relevant manuscripts have been captured. This could be done by including additional keywords, or removing limits (e.g. looking for all studies and not just those in humans).

What if the number of results seems fine?

Once you are happy that the search has captured all the appropriate results, you will want to save a list of the results, including references, title and abstract, as well as recording the initial number of results found. You will need to keep track of this for the PRISMA flowchart. These results should then be distributed to the individuals that have been allocated as reviewers and they should review each abstract and decide whether it meets the inclusion criteria defined for the review. These criteria should have been defined before the search and it is important that both reviewers use them. The reason why each abstract is rejected should also be recorded. This can be done according to the PICOS, with abstracts rejected according to not containing appropriate participants, intervention, comparator, outcome or setting, or for another reason which should be recorded.

In a perfect world both reviewers will select the same abstracts at this stage, however, if they do not, an additional reviewer should arbitrate whether the abstract should be kept. If there is confusion, the abstract should be kept, for full text review (the next step).

After the abstracts have been filtered, should we start writing?

No. You will want to get the full version of each reference and these will also need to be reviewed, in a similar manner to the abstracts, to ensure that no inappropriate references are included. At this stage, you can note any manuscripts in the references that also look appropriate for the systematic review. This will enable you to check that they have been included and if not they can be added (and noted on the PRISMA flowchart).

This process of filtering can be long and tedious, however, it should enable all the relevant literature for the systematic review to be identified. It is at this stage that the writing can commence!

Performing the search for the systematic review

I am going to assume that you have developed your basic research question and are frothing at the mouth with anticipation that you will be able to search the literature and get to review possibly thousands of abstracts. Like many people I would suggest that you use PICOS to help determine your search.

What does PICOS mean?

From the capitalization you have probably realised that this is an acronym. It stands for:


By working your way through each item and defining it, you should be able to clarify your question and also define your search. I feel that the best way to show how to do this is to give an example.


For this example my research question is going to be:
“Based upon differences in microvascular outcomes seen in people of South Asian origin with T2DM compared with the Western population, especially the relationship of outcomes with BMI, would SGLT2 inhibitors be more effective in this population?”

The first step is to define the population being investigated. This is “people of South Asian origin with T2DM”. However, when we perform the search we have to remember that not everyone with use the exact phrase that we have, for example South Asia includes a number of countries and we may want to include the populations of the separate countries and the names of the countries in the search as well. We will also want to add a search term for type 2 diabetes. These can then be combined as a search strategy:
(T2DM OR Type 2 diabetes) AND (South Asian OR South Asia OR India OR Pakistan OR…)

The next step is to define the intervention and for our question it would be SGLT2 inhibitors. The search strategy should also include the names of the individual drugs as well as the class. This would make the following search:
(SGLT2 OR sodium-glucose cotransporter 2 OR sodium-glucose co-transporter 2 OR canagliflozin OR dapagliflozin OR empagliflozin OR sotagliflozin…)

We then move onto the comparison. We are comparing different populations, rather than interventions, so this would be Western people with T2DM. For the first search I would not include this as a search term, and instead would keep it in mind when I was filtering the results following the search. However, the decision to include only studies comparing the two populations would help filter the results.

Next is outcomes which is undefined in our search and so would need discussion amongst the authoring group. Would we be looking at efficacy endpoints or cardiovascular outcomes, for example. Again this could be used for filtering the results and might not be included in the original search.

Finally is setting. This question could include large observational clinical data sets or clinical trial data and both would be interesting. Reviews would not be relevant to teh article but could be a useful way to ensure that all relevant references are found.

What next?

We have now defined our search and also some criteria to filter the results. So the next step would be to perform the search and the databases you use are dependent upon the subject you are researching. For example, in medicine it would include PubMed, Embase, Cochrane reviews and clinicaltrials.gov. It might also include results highlighted in the reference lists for any reviews identified.

The numbers of results should be recorded following teh search and before filtering as this is important for the PRISMA flow diagram.

A good handout to help define a search with PICOS is available from consortiumlibrary.org, which can be found here.

What next for your systematic review – Fully define your research hypothesis or question.

So you have decided that you are going to put together a systematic review and you have managed to persuade some of your colleagues and peers to be co-authors. What might you do next to get the review done?

Have a meeting.

The first thing I would recommend is that you get everyone who is to be involved together to define the problem. This is important because everybody will come to the table with different mind sets and different backgrounds, and also prior knowledge. Together you can make sure that everyone is agreed as to what problem will be addressed.

However, you should not go beyond that in the meeting, rather once the problem has been clarified you should end the meeting and get everybody to think of what hypothesis or question your review will be addressing. This way you will not end up with just the point of view of the most respected author or the person who shouts the loudest. Also, I find that my best ideas come when I am not really thinking directly about them and by leaving time to think you provide the opportunity for people to synthesise what was discussed and come up with any solutions or points that need clarification that were missed during the meeting.

This thinking time should not be extensive and should be defined during the meeting by booking the next meeting in for a couple of days later.

Have another meeting

Everybody has hopefully now thought about the issues and come up with ideas for the hypothesis. At this second meeting you will review everybody’s ideas. Let each person present their ideas in turn without allowing any questions to ensure that people are not intimidated and only present part of what they have come up with.

Once everybody’s ideas have been presented you can then discuss the merits of them all and finesse the hypothesis until you have it agreed by all authors. At this point you will need to discuss any limits people feel will be necessary for the systematic review, will it be limited for example by date, inclusion criteria, size or specific endpoint measures. These limits will be very important once the search has been run as you will typically need to filter huge numbers of abstracts to determine the few that will be included and as suggested by the type of review your process needs to be systematic.

What next?

The next step, once the hypothesis or question has been defined, is to define the search and that will be discussed in the next post.


I have taken a lot of the ideas for this post from an Accidental Creative podcast episode I listened to on brainstorming. This episode can be found here and is well worth a listen.

Some thoughts on systematic reviews

I remember that in my first year of university (in 2000) we were taken to the library and shown how to search the Beilstein Handbook of Organic Chemistry to identify previous instances of reactions we might be interested in. There was a wall of books and it was quite intimidating. However, I never had to look at those books because there was also an online searchable database which made the task much easier. I cannot imagine how much time must have been spent by chemists searching for reactions and reaction conditions, that was suddenly freed up by taking the data online.

This has happened with most data sources and we can now search through records of journal publications and patents with ease. This I believe has led to the increase in the publication of the systematic review and the decrease in the importance placed on narrative reviews. It has changed so much that, nowadays, before many research projects are fully initiated, a systematic review is undertaken to explore, among other things, what is already known about the topic and whether the research project is novel. Owing to their inclusion of all available, relevant data systematic reviews are considered the pinnacle of evidence in evidence-based medicine [Wikipedia]. This is, in my mind, embodied by the Cochrane Collaboration, which has a handbook on how to perform systematic reviews for inclusion in their database.

It is, therefore, important that all researchers understand what a systematic review is and how to perform one. In addition, it is important to know what to do with the output. Is it okay to summarize the data, or should it be synthesised in some way, either through meta-analysis or narrative synthesis. Furthermore, we need to be able to judge the quality of published systematic reviews so that we can judge their quality and determine whether there is any bias inherent in them, or flaws in the methods used.

The PRISMA Statement

The PRISMA Statement is a great help, as it details everything to be included in a systematic review and its planning and I have found this useful when planning, performing and writing up systematic reviews, as it provides a useful checklist to bear in mind.

I want to start my systematic review. What should I do?

The very first thing you need to do is ensure you are asking an interesting question that is relevant and that you are asking it in the correct fashion. You should discuss your idea with colleagues who work in the same field and have similar interests, and also read any relevant papers that you are aware off. This might highlight aspects that you are unaware of or have overlooked. It will also provide the groundwork for the next step, which is putting together your hypothesis and developing your search strategy which will identify the references that will potentially be included in the review. You should also work out who your co-authors will be and each person’s role. For example, you will need somebody to do the search and at least two people to go through the results separately and someone to adjudicate any conflicting views about whether or not to include a manuscript in the review.

Once this is done you are ready to think about actually starting work on your systematic review, and I shall offer my thoughts on those steps in the next few posts.