What should happen after the search is complete, should we immediately start writing the systematic review?

Once you have done the first version of the search, you may find that you have too many results or not many results at all.

Too many results?

If you have too many results it is likely that your search was too broad and that rather than manually checking each reference, using the filters provided by the search engines could help refine the results. Some possibilities for refining the search are to limit it to a specific language (English, for example), or to just searching the abstract and title, rather than the full text.

Too few results?

This might be what you expected and could be the basis for the systematic review, highlighting the need for more studies in a particular area. However, you may want to broaden your search if you feel that not all relevant manuscripts have been captured. This could be done by including additional keywords, or removing limits (e.g. looking for all studies and not just those in humans).

What if the number of results seems fine?

Once you are happy that the search has captured all the appropriate results, you will want to save a list of the results, including references, title and abstract, as well as recording the initial number of results found. You will need to keep track of this for the PRISMA flowchart. These results should then be distributed to the individuals that have been allocated as reviewers and they should review each abstract and decide whether it meets the inclusion criteria defined for the review. These criteria should have been defined before the search and it is important that both reviewers use them. The reason why each abstract is rejected should also be recorded. This can be done according to the PICOS, with abstracts rejected according to not containing appropriate participants, intervention, comparator, outcome or setting, or for another reason which should be recorded.

In a perfect world both reviewers will select the same abstracts at this stage, however, if they do not, an additional reviewer should arbitrate whether the abstract should be kept. If there is confusion, the abstract should be kept, for full text review (the next step).

After the abstracts have been filtered, should we start writing?

No. You will want to get the full version of each reference and these will also need to be reviewed, in a similar manner to the abstracts, to ensure that no inappropriate references are included. At this stage, you can note any manuscripts in the references that also look appropriate for the systematic review. This will enable you to check that they have been included and if not they can be added (and noted on the PRISMA flowchart).

This process of filtering can be long and tedious, however, it should enable all the relevant literature for the systematic review to be identified. It is at this stage that the writing can commence!

Might checklists improve your manuscript?

I started off as an editor, editing manuscripts to house style, as well as copy editing them to some degree to hopefully improve their readability. Each time I saw a document I would have to check for different aspects of style or language. Often this was not the most exciting thing to do, especially when I had seen the document previously. To ensure that crucial steps were not missed we had a series of checklists that could be used enabling you to tick of each task as it was done.

These checklists were not prescriptive with hundreds of elements, rather they had a few elements (typically up to 10) with the emphasis on steps that were often missed or left incomplete. For example when proof reading a manuscript the checklist items included:

  1. Check that all references are mentioned in the text
  2. Check that all tables and figures are referenced appropriately in the text
  3. Double check the title for typos

The title was included on the list because it is often difficult to spot a typo when you know what should be there, it is a short piece of text and it is often very large.

This use of checklists is discussed extremely well in The Checklist Manifesto: How To Get Things Right and I highly recommend that you read it.

What are the next steps after the first draft is written?

Once you have produced your first draft and your co-authors have reviewed it, your aim should be to address the comments and get the manuscript submitted to your chosen journal as soon as possible. How you manage this will depend upon the comments you receive.

If they are minimal

  • Make the suggested changes, or ignore comments if you have a reason to do so and can explain it to the person who made the comment
  • Ensure the manuscript is in the style requested by the journal
  • Confirm that all authors are happy to submit
  • Submit to the journal

If there are major or conflicting comments

This will necessitate the development of a second draft and a further round of review. If the comments are conflicting then it is often best to have a teleconference to discuss them and ensure that everyone is in agreement before making any changes. If agreement cannot be reached then the lead author (or the guarantor for the manuscript) should decide what to do.

Depending on the comments on the next draft the cycle will then continue. However, it is important to remember that the manuscript is being written to be published and you should not have innumerable review rounds, demanding perfection if this is at the cost of submitting.

What is an extended outline and why might it be beneficial?

Once an outline has been written, discussed with co-authors, revised as appropriate in line with any comments and been agreed upon, the next step is to bulk out the outline. I believe that producing an extended outline in bulleted form, rather than a first draft written in prose is the best next step. This ‘extended’ outline should be the first draft in bullet form and contain everything that is expected to go into the first full draft.

Why an additional step?

In my experience, it is much easier to make revisions to the order information is presented in, when it is in bullet form. It might be that as you write you realise that the logical flow from one point to the next is wrong and that it is might be improved by shifting bits around. Or it could be that when your co-authors read the manuscript they highlight places where more information is needed, which could again change the flow of the document. If the document is written in full sentences, it can be hard to make these changes as it is difficult to see where it might fit. It is much easier to revise a ‘work in progress’ that a ‘fait accompli’. Overall, I have found that adding in this step saves time in the long run and often results in a better manuscript.

If the message is incorrect, then the document fails…

I have just started perusing The Craft of Editing  by Michael Alley, which is subtitled A guide for managers, scientists, and engineers. Right near the start three different types of editing are highlighted, editing for content, editing for style and editing for form. However, the most important thing emphasized at this point is that whilst a lot of time might be spent discussing editing for style or form, these are relatively unimportant compared with editing for content.

Editing for content is defined as ensuring that the information included is correct, complete, appropriate for the audience, appropriate for the purpose and acceptable for distribution. This is generally managed by a different person to the other types of review as it will often require knowledge of the topic being discussed.

This is extremely important as we might have a beautifully written piece with no grammatical or spelling errors that is completely wrong for its intended use, and does not convey the correct key message, or even worse contains incorrect information and misleads rather than educates. This is very well put in the book where it states:

 If the message is incorrect, then the document fails, no matter how well the message is communicated or what form it is in.

I think this is something we all need to remember as we try to write or edit scientific literature.

In case you want to read this book it can be found on Amazon here: The Craft of Editing: A Guide for Managers, Scientists, and Engineers

What are the four types or reviewing?

In my thinking there are four main types of reviewing:

  • General review for story flow and sense
  • Copy edit
  • Proofread
  • Data check

Each of these is done for a specific reason and aims to potentially identify different issues with a piece of writing. However, they should not be thought of as exclusive activities. We might copy edit to some degree whilst undertaking a general review, or data check whilst proofreading. Nonetheless, it should be noted that if you are asked to do one type of review you should not undertake another without ensuring that the author of the piece is happy for you to do so. It can be distracting, as well as dispiriting, if lots of minor grammatical and spelling errors are pointed out when someone is expecting you to highlight major issues with story.

General review

This will check that the ‘story’ flows well and that everything makes sense. At this stage you will be ensuring that each argument builds upon a previous one and links to it, without anything being missed or detours being taken. To do this it is important that the objective of the piece of writing has been communicated to the reviewer as well as any issues that the author might be aware of that they want the reviewer to concentrate on. In order to provide the most useful review an understanding of what is being discussed is needed.

Copy edit

This is performed to check that sentences flow well and the most appropriate language is used to make it easy to both read and understand. This might include trimming the piece to make it more streamlined.

Proofread

This should check that spelling and grammar are correct and that the document makes sense. It can also be a final check before publishing to ensure that typeset text is correct and that layout is appropriate.

Data check

This is exactly what you might think and should ensure that all references are appropriate and any data from analyses and study reports have been input correctly

Whilst writing this I was listening to Hyperdub 10.3.