Information

Are there guidelines in the peer reviewing process on assessing methodology?

Are there guidelines in the peer reviewing process on assessing methodology?



We are searching data for your request:

Forums and discussions:
Manuals and reference books:
Data from registers:
Wait the end of the search in all databases.
Upon completion, a link will appear to access the found materials.

Usually an article goes through a peer-review process before it is published, and from what I here it is quite common that the reviewers "demand" some changes.

  • Are there guidelines for peer- reviewers, especially as the methodological side of things is concerned?
  • Are "methodology specialists" incorporated in the process?

I get the impression that good journal editors will get at least one reviewer who is skilled in the methodology used in the paper. The importance of this reviewer role would presumably vary with the statistical or other methodological complexity of the paper.

That said, reviewing is well known to be imperfect particularly when it comes to checking all the possible smaller errors that can be made. Errors in statistical analysis and reporting are wide-spread in published articles in psychology. For example, Bakker and Wicherts (2011) did a review, where to quote the abstract

we checked the consistency of reported test statistics, degrees of freedom, and p values in a random sample of high- and low-impact psychology journals… On the basis of 281 articles… [we estimate that] around 18% of statistical results in the psychological literature are incorrectly reported. Inconsistencies were more common in low-impact journals than in high-impact journals. Moreover, around 15% of the articles contained at least one statistical conclusion that proved, upon recalculation, to be incorrect; that is, recalculation rendered the previously significant result insignificant, or vice versa.

References

  • Bakker, M., & Wicherts, J. M. (2011). The (mis) reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666-678. FULL-TEXT

This does not really answer your question (because it doesn't deal in detail with methodological issues), but there is a nice journal article that deals with problems of the peer review process in general by evaluating "great scientific works of the past" from the perspective of current social sciences:

  • Trafimov, D., & Rice, S. (2009). What if social scientists had reviewed great scientific works of the past? Perspectives on Psychological Science, 4, 65-78. doi:10.1111/j.1745-6924.2009.01107.x Available online at http://cda.psych.uiuc.edu/writing_class_material/perspectives_articles/trafimow.pdf

The reference list has some articles on statistical/mathematical methodology questions.


In another interesting article, Gerd Gigerenzer shows how p-values are misinterpreted in current research and null hypothesis testing is made a false requirement for the acceptance of papers by major journals:

  • Gigerenzer, G., Krauss, S., & Vitouch, O. (2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In D. Kaplan (Ed.), The Sage Handbook of Quantitative Methodology for the Social Sciences (pp. 391-408). Thousand Oaks, CA: Sage. Available online at http://www.sozialpsychologie.uni-frankfurt.de/wp-content/uploads/2010/09/GG_Null_20042.pdf

A Google search for "peer review guidelines" brings many hits from publishers and institutions. Maybe some of them deal with methodological issues in more detail.


U.S. government peer review policies [ edit | edit source ]

Most federal regulatory agencies in the United States government must comply with specific peer review requirements before the agencies publicly disseminate certain scientific information. These requirements were published in a Peer Review Bulletin issued by the White House Office of Management and Budget ("OMB"), which establishes "government-wide standards concerning when peer review is required and, if required, what type of per review processes are appropriate."

OMB’s peer review bulletin requires that US federal regulatory agencies submit all "influential scientific information" to peer review before the information is publicly disseminated. The Bulletin defines "scientific information" as:

"factual inputs, data, models, analyses, technical information, or scientific assessments related to such disciplines as the behavioral and social sciences, public health and medical sciences, life and earth sciences, engineering, or physical sciences."

The OMB peer review Bulletin defines "influential scientific information" as

"scientific information the agency reasonably can determine will have or does have a clear and substantial impact on important public policies or private sector decisions. In the term 'influential scientific information,' the term 'influential' should be interpreted consistently with OMB's government-wide information quality guidelines and the information quality guidelines of the agency."

As noted in the preceding quotation, the Peer Review Bulletin must be read in conjunction with "OMB's government-wide information quality guidelines and the information quality guidelines of the agency." These guidelines govern the quality of all information disseminated by most US government regulatory agencies. These guidelines are required by a US statute enacted in 2001 called the Data Quality Act and also known as the Information Quality Act ("IQA"). OMB states that it prepared the peer review Bulletin pursuant to OMB's authority under the DQA.

The peer review Bulletin provides detailed guidelines for peer review of influential scientific information. The Bulletin applies more stringent peer review requirements to "highly influential scientific assessments,"

"which are a subset of influential scientific information. A scientific assessment is an evaluation of a body of scientific or technical knowledge that typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information."

While the peer review Bulletin's specific guidelines will not be discussed here in detail, one should note that the guidelines differ in several respects from traditional peer review practices at most journals. For example, the Bulletin requires public disclosure of peer reviewers' identities when they are reviewing highly influential scientific assessments. The Bulletin's summary of some of these requirements is set forth below:

"In general, an agency conducting a peer review of a highly influential scientific assessment must ensure that the peer review process is transparent by making available to the public the written charge to the peer reviewers, the peer reviewers’ names, the peer reviewers’ report(s), and the agency’s response to the peer reviewers’ report(s). . This Bulletin requires agencies to adopt or adapt the committee selection policies employed by the National Academy of Sciences(NAS)."

The peer review Bulletin specifically addresses the effect of publication in a refereed scientific journal as well the variations and limitations with peer review:


Types of peer review

Single blind

In this process, the names of the reviewers are not known to the author(s). The reviewers do know the name of the author(s).

Double blind

Here, neither reviewers or authors know each other's names.

In the open review process, both reviewers and authors know each other's names.

What about editorial review?

Journals also use an editorial review process. This is not the same as peer review. In an editorial review process an article is evaluated for style guidelines and for clarity. Reviewers here do not look at technical accuracy or errors in data or methodology, but instead look at grammar, style, and whether an article is well written.


Good practice guidelines for decision-analytic modelling in health technology assessment: a review and consolidation of quality assessment

The use of decision-analytic modelling for the purpose of health technology assessment (HTA) has increased dramatically in recent years. Several guidelines for best practice have emerged in the literature however, there is no agreed standard for what constitutes a 'good model' or how models should be formally assessed. The objective of this paper is to identify, review and consolidate existing guidelines on the use of decision-analytic modelling for the purpose of HTA and to develop a consistent framework against which the quality of models may be assessed. The review and resultant framework are summarised under the three key themes of Structure, Data and Consistency. 'Structural' aspects relate to the scope and mathematical structure of the model including the strategies under evaluation. Issues covered under the general heading of 'Data' include data identification methods and how uncertainty should be addressed. 'Consistency' relates to the overall quality of the model. The review of existing guidelines showed that although authors may provide a consistent message regarding some aspects of modelling, such as the need for transparency, they are contradictory in other areas. Particular areas of disagreement are how data should be incorporated into models and how uncertainty should be assessed. For the purpose of evaluation, the resultant framework is applied to a decision-analytic model developed as part of an appraisal for the National Institute for Health and Clinical Excellence (NICE) in the UK. As a further assessment, the review based on the framework is compared with an assessment provided by an independent experienced modeller not using the framework. It is hoped that the framework developed here may form part of the appraisals process for assessment bodies such as NICE and decision models submitted to peer review journals. However, given the speed with which decision-modelling methodology advances, there is a need for its continual update.


Conclusion

A Peer review is an important way to get feedback from the members of your team. It also provides useful information that can be used to further drive organisational growth and development.

When carrying out a peer review of any kind, remember to maintain a high level of professionalism. Ask questions that encourage honest feedback from employees. Do not squirrel or ask questions with predetermined responses.

More importantly, use workplace tools that help you gather and process feedback effectively. Formplus has numerous form templates for gathering and processing information.

You can also create custom forms for your organisation using Formplus builder.


What are the ethical guidelines for peer reviewers?

All peer reviewers must follow these ethical guidelines for Taylor & Francis journal articles in review:

  • Reviewers must give unbiased consideration to each manuscript submitted. They should judge each on its merits, without regard to race, religion, nationality, sex, seniority, or institutional affiliation of the author(s).
  • Reviewers must declare any conflict of interest before agreeing to review a manuscript. This includes any relationship with the author that may bias their review.
  • Reviewers must keep the peer review process confidential. They must not share information or correspondence about a manuscript with anyone outside of the peer review process.
  • Reviewers should provide a constructive, comprehensive, evidenced, and appropriately substantial peer review report.
  • Reviewers must avoid making statements in their report which might be construed as impugning any person’s reputation.
  • Reviewers should make all reasonable effort to submit their report and recommendation on time. They should inform the editor if this is not possible.
  • Reviewers should call to the journal editor’s attention any significant similarity between the manuscript under consideration and any published paper or submitted manuscripts of which they are aware.

Taylor & Francis recommend that reviewers also adhere to the COPE Ethical Guidelines for Peer Reviewers.

Read key findings from our white paper research – Peer review: a global view


Improving peer review: What reviewers can do

Robert J. DiDomenico, Pharm.D., BCPS-AQ Cardiology, FCCP, FHFSA, William L. Baker, Pharm.D., FCCP, FACC, FAHA, Stuart T. Haines, Pharm.D., FASHP, FCCP, FAPhA, Improving peer review: What reviewers can do, American Journal of Health-System Pharmacy, Volume 74, Issue 24, 15 December 2017, Pages 2080–2084, https://doi.org/10.2146/ajhp170190

As an author, there is no better feeling than to receive correspondence from a journal editor that your paper has been accepted for publication. However, that sense of elation can quickly turn to dread if the author has to respond to dozens, even hundreds, of comments from peer reviewers. We have observed that the quality of peer reviews has declined over the past decade, and their focus is often misguided, emphasizing grammar and style over science and methodology. This commentary provides guidance to reviewers on the best practices for conducting a peer review for biomedical journals.

Benefits of being a peer reviewer. Volunteering to be a peer reviewer is an important first step in the peer-review process. Reviewers are typically identified based on their expertise in a particular field. Peer reviewers provide benefit to authors, editors, and journals by providing useful guidance related to a scholarly work, evaluating the methodology and importance of the science, and making recommendations regarding publication decisions. 1 , 2 There are benefits to serving as peer reviewers as well. These include keeping abreast of the latest developments in their field and improving their skills to critically evaluate scientific papers, which may improve their own scientific writing. Lastly, it is an honor and a privilege to be a member of a profession. Thus, there is a moral obligation to give back to one’s professional and scientific community. 3 , 4 For these reasons, participation in peer review is a professional expectation in many settings.

Characteristics of a good peer reviewer. Most biomedical journals use the feedback provided during the peer-review process to make editorial decisions on manuscripts submitted for publication. Typically, editors of peer-reviewed journals seek content experts to perform a critical analysis of the manuscript and provide (1) specific comments regarding the manuscript’s scientific merit, (2) recommendations to authors on how to improve the manuscript, and (3) recommendations to the editor regarding manuscript disposition. Invitations to serve as a peer reviewer are often accompanied by Web links to reviewer resources (e.g., reviewer guidelines), but formal training is usually not provided nor required.

The characteristics of a good peer reviewer are not well defined. While several studies have attempted to identify characteristics that predict reviewer performance, there is little consistency in the findings, and the associations are weak. 5 , – 9 Younger age of the reviewer is 1 of the only characteristics that has been consistently associated with stronger peer reviews. 5 , – 8 Some researchers have found a positive association between the duration of time spent reviewing the article (up to 3 hours) and review quality. 5 , 7

The quality of peer review varies greatly, perhaps partly because few professionals receive training in this area. 1 , 10 However, formal training alone likely only marginally affects the quality of peer reviews. Investigators who conducted a study of more than 300 reviewers who performed 2,856 reviews for the Annals of Emergency Medicine did not find an association between more than 10 hours of training in critical appraisal skills and review quality. 6 However, a BMJ study of 609 reviewers found a small, short-term benefit (<6 months) associated with participation in a peer-reviewer training program. 9 Nevertheless, the variation in the quality of peer reviews suggests that peer reviewers may benefit from some guidance.

What reviewers can do: A step-by-step approach. Although other approaches are acceptable, we recommend a step-by-step process for peer reviewers ( Figure 1). Regardless of the approach used, the best peer reviews are systematic and comprehensive.

Step-by-step process for performing a peer review. 1 , – 4 , 10 , – 13

Step-by-step process for performing a peer review. 1 , – 4 , 10 , – 13

Responding to the invitation to perform a peer review. Peer reviewers should consider several issues before deciding to accept or decline an invitation to perform a peer review. Reviewers should first assess whether they are qualified to review the manuscript. 2 , 10 , 11 Reviewers are typically provided an abstract of the manuscript with the invitation to review. Reviewers should only accept invitations to review if they have expertise in the manuscript’s area of focus. In some cases, the reviewer may have the expertise to review a key section of the manuscript (e.g., research methodology, statistical analysis) but not other sections. In this instance, the reviewer should be clear about the scope of the review before accepting the invitation. 2 , 10

If the manuscript is within the reviewer’s area of expertise, potential or actual conflicts of interest should be considered. 2 , 10 , 11 Potential reviewer conflicts may include close personal (e.g., spouse) or professional (e.g., mentor–mentee, institutional colleague) relationships with 1 or more of the authors. 10 , 11 In addition, the presence of conflicting or competing interests that may create the potential for personal, financial, intellectual, professional, political, or religious benefit by the reviewer may be considered conflicts of interest. 2 , 10 , 11 Reviewers should ask themselves this simple but powerful question: Would others—the authors, editors, or readers—believe that my positive or negative review of this paper is in any way motivated by self-interest? If the answer is yes, it is critical to discuss the potential conflict with the editor before accepting the invitation to review. 2 , 10 , 11

Timeliness is critically important to the peer-review process. To avoid unnecessary delays in the publication process, the reviewer must consider his or her ability to meet the journal’s deadline for completing the review before accepting the invitation. 2 , 10 , 11 A quality peer-review can take several hours to perform. 3 , 5 , 7 , 10 Reviewers are typically given 2 weeks to complete their peer review, though the timeline may be as brief as 48 hours and as long as 4 weeks. 11 Reviewers should respond to peer-review invitations in a timely manner, especially if the invitation is declined. 2 , 12 In some cases, journals will rescind their invitations if a timely response (e.g., within 3–4 days) is not received. Throughout the process, reviewers must maintain confidentiality and should never circulate the manuscript to or discuss its contents with others. 1 , 12 If reviewers would like to include others (e.g., trainees) in the peer-review process, permission should be requested to do so, or at a minimum, the editor should be notified when the critique is submitted. 2 , 12

Performing the peer review. Quick read. Many agree that the first step in completing a peer review is to perform a quick read. 1 , 3 , 4 , 10 , 13 During this first read, reviewers should skim the article to gain a basic understanding of the manuscript’s contents. 13 During the quick read, reviewers should once again assess whether they have the expertise to perform the review and whether any real or potential conflicts of interest exist. Reviewers should notify the editor immediately if these problems are identified. 4 Reviewers should note ambiguities and questions that arise during the quick read and begin to form opinions regarding the importance and relevance of the work. 3 , 10

Another important reason for the quick read is to identify critical or “fatal” flaws of the manuscript. 1 , 4 , 10 , 13 In general, a manuscript’s flaws can be categorized as minor (do not require author action), major (require satisfactory resolution before publication is considered), and fatal (deem the manuscript—even with major revisions—irreparable). 1 , 10 , 13 If reviewers identify a fatal flaw during the quick read, they may recommend to reject the manuscript at this point, forgoing a more-detailed read and written critique. 1 , 4 , 10 , 13 Common examples of manuscript flaws are listed in Table 1.

Manuscript Flaws That May Warrant Revision or Rejection During Peer Review 1 , 10 , 13

Type of Flaw . Characteristics .
Minor flaws Writing style is poor, lack of clarity, manuscript organization and flow are poor, tables or figures duplicate information in the text, referencing errors
Major flaws Methodology omits analyses that may affect results (e.g., control for critical variables), methods are poorly described, unclear data tables or figures, confirmatory data that add little to current understanding in the absence of strong arguments justifying the repetition, important or relevant research is not cited, limitations are poorly acknowledged, conclusions are overstated or contradicted by author’s own data
Fatal flaws Consent not obtained when needed, methods are unethical, methodology has been discredited, methodology is flawed leading to inconclusive or inaccurate results (e.g., ignores a process known to strongly influence the topic of interest)
Type of Flaw . Characteristics .
Minor flaws Writing style is poor, lack of clarity, manuscript organization and flow are poor, tables or figures duplicate information in the text, referencing errors
Major flaws Methodology omits analyses that may affect results (e.g., control for critical variables), methods are poorly described, unclear data tables or figures, confirmatory data that add little to current understanding in the absence of strong arguments justifying the repetition, important or relevant research is not cited, limitations are poorly acknowledged, conclusions are overstated or contradicted by author’s own data
Fatal flaws Consent not obtained when needed, methods are unethical, methodology has been discredited, methodology is flawed leading to inconclusive or inaccurate results (e.g., ignores a process known to strongly influence the topic of interest)

Manuscript Flaws That May Warrant Revision or Rejection During Peer Review 1 , 10 , 13

Type of Flaw . Characteristics .
Minor flaws Writing style is poor, lack of clarity, manuscript organization and flow are poor, tables or figures duplicate information in the text, referencing errors
Major flaws Methodology omits analyses that may affect results (e.g., control for critical variables), methods are poorly described, unclear data tables or figures, confirmatory data that add little to current understanding in the absence of strong arguments justifying the repetition, important or relevant research is not cited, limitations are poorly acknowledged, conclusions are overstated or contradicted by author’s own data
Fatal flaws Consent not obtained when needed, methods are unethical, methodology has been discredited, methodology is flawed leading to inconclusive or inaccurate results (e.g., ignores a process known to strongly influence the topic of interest)
Type of Flaw . Characteristics .
Minor flaws Writing style is poor, lack of clarity, manuscript organization and flow are poor, tables or figures duplicate information in the text, referencing errors
Major flaws Methodology omits analyses that may affect results (e.g., control for critical variables), methods are poorly described, unclear data tables or figures, confirmatory data that add little to current understanding in the absence of strong arguments justifying the repetition, important or relevant research is not cited, limitations are poorly acknowledged, conclusions are overstated or contradicted by author’s own data
Fatal flaws Consent not obtained when needed, methods are unethical, methodology has been discredited, methodology is flawed leading to inconclusive or inaccurate results (e.g., ignores a process known to strongly influence the topic of interest)

Detailed read. After the quick read, reviewers should perform a more detailed, critical read of the manuscript. 3 , 4 , 10 , 13 Some scholars have suggested that this occur hours to days after the first read, as this may allow the reviewer to better prepare for a more critical read by reviewing ancillary materials (e.g., reviewer instructions, figures, tables, supplementary files). 2 , 3 , 13 The major purpose of the detailed read is to determine the scientific merits and relevance of the manuscript. 3 , 4 , 13 To help formulate this evaluation, reviewers should ask themselves 3 questions about the manuscript: Do I understand it? Do I believe it? Do I care? 11 Manuscripts should clearly state the research question or clinical problem and then describe how the scholarly work was conducted. If valid methods are used and the conclusions are supported by the data, the results should be credible. Finally, if the work is original, the question or problem important, or the intervention innovative, it should stimulate interest in the reviewer and readers alike. Reviewers should also note ambiguities and questions that arise during their detailed review and provide suggestions for improvement. 3 , 4 , 13 Importantly, the reviewer should not focus on language, writing style, and grammar unless these affect the clarity of a particular point or the frequency of these issues significantly impairs the manuscript’s overall readability. 3 , 4 , 11 , 13 In such cases, the reviewer should not attempt to rewrite the paper for the authors but rather point out the sections of the manuscript that need to be addressed.

The detailed read should evaluate all aspects of the manuscript, including the title, keywords, abstract, introduction, methods, results, discussion, conclusion, graphs, tables, figures, and references. Most minor and major flaws ( Table 1) are identified during this more-critical read. For research manuscripts, manuscript flaws are commonly found in the methods section. Reviewers should consider using a systematic process (e.g., R andomized, A ttrition, M easurement by b linded assessors or objective measurements [RAMbo]) or follow reporting guidelines when critically evaluating randomized clinical trials (e.g., CONsolidated Standards of Reporting Trials [CONSORT]) or meta-analyses (e.g., QUality Of Reporting Of Meta-analyses [QUORUM]), particularly for reviewers with limited experience. 1 , 4 , 14 , – 16 When evaluating the conclusion section, reviewers should ensure that the authors do not overstate their findings and identify any statements that may be perceived as biased. Although not an exhaustive list, Table 2 lists the characteristics of quality manuscripts worthy of acceptance. Reviewers should not be afraid to recommend conditional acceptance or rejection for those manuscripts that they believe have significant flaws or are insufficiently unique or important.

Characteristics of Quality Manuscripts Worthy of Acceptance 1 , 3 , 4 , 10 , 11 , 14

Section of Manuscript . Characteristics .
Title, keywords, abstract Title is succinct and clearly indicates the nature of the scholarly work, abstract appropriately and accurately summarizes the manuscript contents, no new information is introduced
Introduction Summarizes recent developments in the area of interest highlights existing gaps in knowledge clarifies the originality, novelty, and importance of the work
Methods Follows acceptable standards or uses a novel technique that improves the strength of the research, approaches used are described sufficiently to allow replication of the scholarly work
Results and discussion Summarize data coherently, significance described in the context of current knowledge, limitations and gaps that remain are appropriately described
Conclusion Succinct, reflects upon the aims of the manuscript, consistent with the data
Graphs, tables, figures, references Complement data in the text, does not repeat nor contradict data in the text, relevant literature cited
Section of Manuscript . Characteristics .
Title, keywords, abstract Title is succinct and clearly indicates the nature of the scholarly work, abstract appropriately and accurately summarizes the manuscript contents, no new information is introduced
Introduction Summarizes recent developments in the area of interest highlights existing gaps in knowledge clarifies the originality, novelty, and importance of the work
Methods Follows acceptable standards or uses a novel technique that improves the strength of the research, approaches used are described sufficiently to allow replication of the scholarly work
Results and discussion Summarize data coherently, significance described in the context of current knowledge, limitations and gaps that remain are appropriately described
Conclusion Succinct, reflects upon the aims of the manuscript, consistent with the data
Graphs, tables, figures, references Complement data in the text, does not repeat nor contradict data in the text, relevant literature cited

Characteristics of Quality Manuscripts Worthy of Acceptance 1 , 3 , 4 , 10 , 11 , 14

Section of Manuscript . Characteristics .
Title, keywords, abstract Title is succinct and clearly indicates the nature of the scholarly work, abstract appropriately and accurately summarizes the manuscript contents, no new information is introduced
Introduction Summarizes recent developments in the area of interest highlights existing gaps in knowledge clarifies the originality, novelty, and importance of the work
Methods Follows acceptable standards or uses a novel technique that improves the strength of the research, approaches used are described sufficiently to allow replication of the scholarly work
Results and discussion Summarize data coherently, significance described in the context of current knowledge, limitations and gaps that remain are appropriately described
Conclusion Succinct, reflects upon the aims of the manuscript, consistent with the data
Graphs, tables, figures, references Complement data in the text, does not repeat nor contradict data in the text, relevant literature cited
Section of Manuscript . Characteristics .
Title, keywords, abstract Title is succinct and clearly indicates the nature of the scholarly work, abstract appropriately and accurately summarizes the manuscript contents, no new information is introduced
Introduction Summarizes recent developments in the area of interest highlights existing gaps in knowledge clarifies the originality, novelty, and importance of the work
Methods Follows acceptable standards or uses a novel technique that improves the strength of the research, approaches used are described sufficiently to allow replication of the scholarly work
Results and discussion Summarize data coherently, significance described in the context of current knowledge, limitations and gaps that remain are appropriately described
Conclusion Succinct, reflects upon the aims of the manuscript, consistent with the data
Graphs, tables, figures, references Complement data in the text, does not repeat nor contradict data in the text, relevant literature cited

Final scan. Reviewers should perform a final scan of the manuscript, focusing on organization, flow, and other issues that affect clarity. 10 Reviewers should keep in mind that their assessment of the scientific merit of the manuscript is more valuable to the journal editors than their assessment of grammar and writing style. 3 , 4 , 10 , 13 The final quick read also provides the reviewer an opportunity to highlight minor flaws ( Table 1) and offer suggestions for improvement. 1

Written critique. Reviewers should prepare their written critique to the journal editors by first summarizing the manuscript and then providing specific comments related to the identified flaws. 4 , 10 , 11 , 13 Although some reviewers use the first paragraph of the written critique to summarize the manuscript’s contents and findings, we suggest that this summary focus on the reviewer’s overall analysis of the manuscript. 4 , 10 , 11 , 13 When providing their overall analysis of the manuscript, reviewers should highlight the strengths, point out significant flaws, and comment on the importance or significance of the work. 4 , 10 , 11 , 13 Throughout the critique, reviewers should be honest and direct but considerate when providing feedback to authors. 1 , 13 Grammatical issues identified by the reviewer that are pervasive (e.g., numerous spelling errors) should be summarized, leaving detailed review of grammar, spelling, and writing style to the journal’s copyeditors and technical staff. 4 , 10 , 11 , 13 The summary should balance both positive and constructive feedback consistent with the overall evaluation of the manuscript (e.g., the majority of comments should be positive if the recommendation is to accept or accept with minor revisions). 4 , 10 , 11 , 13 If a reviewer reviewed only specific portions of the manuscript, this fact should be clearly stated in the opening summary. 2

Following the summary, reviewers should provide specific comments related to each section of the manuscript. Reviewers should organize their comments in a logical manner and number them. 10 , 11 , 13 In addition, reviewers should specify where within the manuscript (e.g., page and line numbers) the comments are directed. 13 Many reviewers neglect to provide comments about the title and abstract, but these are perhaps the most important elements of the manuscript. Many readers will read or have access to only the title and abstract when conducting a literature search. It is helpful to the authors and the editors to indicate whether a comment identifies a major issue that must be addressed or is merely a recommendation (i.e., minor issue). 4 , 13 Given that one of the purposes of peer review is to improve the quality of manuscripts, reviewer feedback should be objective and constructive, with actionable suggestions for addressing flaws. 2 , 4 , 10 , 11 , 13 Because the classification of a manuscript’s flaws is subjective, if reviewers determine that a fatal flaw exists, an explanation of why the manuscript is beyond repair is warranted. 1 , 10 , 13 When appropriate, feedback should be supported with evidence (e.g., supporting literature) to assist the editors in their decision and the authors in their revisions. 2 , 4 , 10 When a statement in the manuscript is ambiguous and may lead to reader confusion, the reviewer may offer specific suggestions to improve clarity. 4 , 10 , 11 , 13 Reviewers should refrain from divulging their publication recommendation to the authors in the critique. The responsibility to determine the manuscript’s disposition rests with the journal editors.

Reviewers are often asked to provide confidential feedback to the editors. These comments should include a brief explanation to justify the reviewer’s publication recommendation. 13 Other confidential comments that reviewers may wish to include are unvarnished opinions on specific aspects of the manuscript, concerns regarding ethics (e.g., plagiarism, fraud, unethical research practices), and the acknowledgment of individuals who may have assisted in conducting the review. 1 , 4 , 11 , 13

Publication recommendation. In most instances, reviewers are asked to provide their recommendations regarding publication as follows: accept, accept with revisions, or reject. Some journals delineate “accept with minor revisions” from “accept with major revisions.” In general, manuscripts that are well written, novel, scientifically powerful, and relevant to the journal’s readers warrant acceptance. 3 , 11 Manuscripts that do not meet these criteria should be considered for conditional acceptance (pending revisions) or rejection. 3 Manuscripts with only minor flaws should be recommended for conditional acceptance (accept with revisions or accept with minor revisions) with the understanding that the reviewer’s suggestions for improving the manuscript are discretionary and may ultimately be ignored by authors. 4 , 10 In contrast, manuscripts with major flaws should be recommended for conditional acceptance (accept with revisions or accept with major revisions) only if the authors satisfactorily address the reviewers’ concerns. 1 , 4 , 10 When deliberating their publication recommendation, reviewers should give priority to innovative works that may have methodological flaws (minor or major) over manuscripts that are merely confirmatory or likely to be of low interest to the journal’s readership. 4 The use of research methods that are unethical, do not follow contemporary norms (e.g., outdated or discredited methods), or cannot be replicated is grounds for recommending the rejection of a manuscript. 1 , 11 , 13

Conclusion. The quality of peer reviews varies greatly and is often misguided, resulting in challenges for journal editors and frustrations for authors. The use of a stepwise approach when conducting a peer review can improve the quality of reviews, ultimately leading to high-quality articles for publication.


Steps and Standards of a Systematic Review

A systematic review begins with a clearly defined question accompanied by established inclusion and exclusion criteria.

Once you have an initial idea, search for already published literature on the topic. If there is already a systematic review article, then another review on the same topic is not needed. Additionally, reviewing published systematic reviews can help you to better frame your own research question.

If you find a research article that matches your topic, save it. Articles that exemplify your research topic are called "gold-standard" articles, and are used as examples during the search process (Step 3).

Databases for Finding Already Published Reviews

    (Free): A search strategy to retrieve citations identified as systematic reviews or meta-analyses. (NetID required): It's EBM tab can help find systematic reviews or meta-analyses. (Free): A tool to find systematic reviews using PICO (Population, Intervention, Comparison, and Outcome) (NetID required): The resource includes the full text of the regularly updated systematic reviews of the effects of healthcare prepared by The Cochrane Collaboration.

What is a systematic review protocol?

Based on Cochrane, it is a pre-defined plan and the proposed approach for a systematic review. It outlines the question that the review authors are addressing, detailing the criteria against which studies will be assessed for inclusion in the review, and describing how the authors will manage the review process. The protocol also serves as a notification of your plans to other researchers, so that no one will attempt the same project. It's strongly suggested to register your systematic review protocol once it's written. The purpose of registering a systematic review is to reduce publication bias, enhance transparency, and avoid duplication of effort.

A good way to familiarize yourself with writing systematic review research protocols is to take a look at those registered on PROSPERO- the International Prospective Register of Systematic Reviews. Once you have written a draft of your protocol, have it peer reviewed by someone outside of your research team. If it is registered in PROSPERO, it will also be open for peer review by other researchers.

Key information for writing a systematic review protocol include:

  • A research question including patients and population, intervention or exposure, comparison (if relevant), and outcomes.
  • Databases used for the searches and search strategy
  • Criteria for inclusion and exclusion.
  • Methods used to assess risk of bias.
  • Method of data extraction (selection and coding form).
  • Method of data analysis including statistical analysis if relevant.
  • Anticipated or actual start date.
  • Forming a systematic review team and team members' information.
  • Review project timeline.
  • Funding sources/sponsors
  • Conflict of interests.
  • Date of registration.

Besides PROSPERO, there are other organizations that you might search for a registered systematic review protocol:


Effective peer assessment processes: Research findings and future directions

Despite the popularity of peer assessment (PA), gaps in the literature make it difficult to describe exactly what constitutes effective PA. In a literature review, we divided PA into variables and then investigated their interrelatedness. We found that (a) PA's psychometric qualities are improved by the training and experience of peer assessors (b) the development of domain-specific skills benefits from PA-based revision (c) the development of PA skills benefits from training and is related to students' thinking style and academic achievement, and (d) student attitudes towards PA are positively influenced by training and experience. We conclude with recommendations for future research.


Optimal strategies to consider when peer reviewing a systematic review and meta-analysis

Systematic reviews are popular. A recent estimate indicates that 11 new systematic reviews are published daily. Nevertheless, evidence indicates that the quality of reporting of systematic reviews is not optimal. One likely reason is that the authors’ reports have received inadequate peer review. There are now many different types of systematic reviews and peer reviewing them can be enhanced by using a reporting guideline to supplement whatever template the journal editors have asked you, as a peer reviewer, to use. Additionally, keeping up with the current literature, whether as a content expert or being aware of advances in systematic review methods is likely be make for a more comprehensive and effective peer review. Providing a brief summary of what the systematic review has reported is an important first step in the peer review process (and not performed frequently enough). At its core, it provides the authors with some sense of what the peer reviewer believes was performed (Methods) and found (Results). Importantly, it also provides clarity regarding any potential problems in the methods, including statistical approaches for meta-analysis, results, and interpretation of the systematic review, for which the peer reviewer can seek explanations from the authors these clarifications are best presented as questions to the authors.


Optimal strategies to consider when peer reviewing a systematic review and meta-analysis

Systematic reviews are popular. A recent estimate indicates that 11 new systematic reviews are published daily. Nevertheless, evidence indicates that the quality of reporting of systematic reviews is not optimal. One likely reason is that the authors’ reports have received inadequate peer review. There are now many different types of systematic reviews and peer reviewing them can be enhanced by using a reporting guideline to supplement whatever template the journal editors have asked you, as a peer reviewer, to use. Additionally, keeping up with the current literature, whether as a content expert or being aware of advances in systematic review methods is likely be make for a more comprehensive and effective peer review. Providing a brief summary of what the systematic review has reported is an important first step in the peer review process (and not performed frequently enough). At its core, it provides the authors with some sense of what the peer reviewer believes was performed (Methods) and found (Results). Importantly, it also provides clarity regarding any potential problems in the methods, including statistical approaches for meta-analysis, results, and interpretation of the systematic review, for which the peer reviewer can seek explanations from the authors these clarifications are best presented as questions to the authors.


Types of peer review

Single blind

In this process, the names of the reviewers are not known to the author(s). The reviewers do know the name of the author(s).

Double blind

Here, neither reviewers or authors know each other's names.

In the open review process, both reviewers and authors know each other's names.

What about editorial review?

Journals also use an editorial review process. This is not the same as peer review. In an editorial review process an article is evaluated for style guidelines and for clarity. Reviewers here do not look at technical accuracy or errors in data or methodology, but instead look at grammar, style, and whether an article is well written.


Good practice guidelines for decision-analytic modelling in health technology assessment: a review and consolidation of quality assessment

The use of decision-analytic modelling for the purpose of health technology assessment (HTA) has increased dramatically in recent years. Several guidelines for best practice have emerged in the literature however, there is no agreed standard for what constitutes a 'good model' or how models should be formally assessed. The objective of this paper is to identify, review and consolidate existing guidelines on the use of decision-analytic modelling for the purpose of HTA and to develop a consistent framework against which the quality of models may be assessed. The review and resultant framework are summarised under the three key themes of Structure, Data and Consistency. 'Structural' aspects relate to the scope and mathematical structure of the model including the strategies under evaluation. Issues covered under the general heading of 'Data' include data identification methods and how uncertainty should be addressed. 'Consistency' relates to the overall quality of the model. The review of existing guidelines showed that although authors may provide a consistent message regarding some aspects of modelling, such as the need for transparency, they are contradictory in other areas. Particular areas of disagreement are how data should be incorporated into models and how uncertainty should be assessed. For the purpose of evaluation, the resultant framework is applied to a decision-analytic model developed as part of an appraisal for the National Institute for Health and Clinical Excellence (NICE) in the UK. As a further assessment, the review based on the framework is compared with an assessment provided by an independent experienced modeller not using the framework. It is hoped that the framework developed here may form part of the appraisals process for assessment bodies such as NICE and decision models submitted to peer review journals. However, given the speed with which decision-modelling methodology advances, there is a need for its continual update.


U.S. government peer review policies [ edit | edit source ]

Most federal regulatory agencies in the United States government must comply with specific peer review requirements before the agencies publicly disseminate certain scientific information. These requirements were published in a Peer Review Bulletin issued by the White House Office of Management and Budget ("OMB"), which establishes "government-wide standards concerning when peer review is required and, if required, what type of per review processes are appropriate."

OMB’s peer review bulletin requires that US federal regulatory agencies submit all "influential scientific information" to peer review before the information is publicly disseminated. The Bulletin defines "scientific information" as:

"factual inputs, data, models, analyses, technical information, or scientific assessments related to such disciplines as the behavioral and social sciences, public health and medical sciences, life and earth sciences, engineering, or physical sciences."

The OMB peer review Bulletin defines "influential scientific information" as

"scientific information the agency reasonably can determine will have or does have a clear and substantial impact on important public policies or private sector decisions. In the term 'influential scientific information,' the term 'influential' should be interpreted consistently with OMB's government-wide information quality guidelines and the information quality guidelines of the agency."

As noted in the preceding quotation, the Peer Review Bulletin must be read in conjunction with "OMB's government-wide information quality guidelines and the information quality guidelines of the agency." These guidelines govern the quality of all information disseminated by most US government regulatory agencies. These guidelines are required by a US statute enacted in 2001 called the Data Quality Act and also known as the Information Quality Act ("IQA"). OMB states that it prepared the peer review Bulletin pursuant to OMB's authority under the DQA.

The peer review Bulletin provides detailed guidelines for peer review of influential scientific information. The Bulletin applies more stringent peer review requirements to "highly influential scientific assessments,"

"which are a subset of influential scientific information. A scientific assessment is an evaluation of a body of scientific or technical knowledge that typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information."

While the peer review Bulletin's specific guidelines will not be discussed here in detail, one should note that the guidelines differ in several respects from traditional peer review practices at most journals. For example, the Bulletin requires public disclosure of peer reviewers' identities when they are reviewing highly influential scientific assessments. The Bulletin's summary of some of these requirements is set forth below:

"In general, an agency conducting a peer review of a highly influential scientific assessment must ensure that the peer review process is transparent by making available to the public the written charge to the peer reviewers, the peer reviewers’ names, the peer reviewers’ report(s), and the agency’s response to the peer reviewers’ report(s). . This Bulletin requires agencies to adopt or adapt the committee selection policies employed by the National Academy of Sciences(NAS)."

The peer review Bulletin specifically addresses the effect of publication in a refereed scientific journal as well the variations and limitations with peer review:


Conclusion

A Peer review is an important way to get feedback from the members of your team. It also provides useful information that can be used to further drive organisational growth and development.

When carrying out a peer review of any kind, remember to maintain a high level of professionalism. Ask questions that encourage honest feedback from employees. Do not squirrel or ask questions with predetermined responses.

More importantly, use workplace tools that help you gather and process feedback effectively. Formplus has numerous form templates for gathering and processing information.

You can also create custom forms for your organisation using Formplus builder.


Improving peer review: What reviewers can do

Robert J. DiDomenico, Pharm.D., BCPS-AQ Cardiology, FCCP, FHFSA, William L. Baker, Pharm.D., FCCP, FACC, FAHA, Stuart T. Haines, Pharm.D., FASHP, FCCP, FAPhA, Improving peer review: What reviewers can do, American Journal of Health-System Pharmacy, Volume 74, Issue 24, 15 December 2017, Pages 2080–2084, https://doi.org/10.2146/ajhp170190

As an author, there is no better feeling than to receive correspondence from a journal editor that your paper has been accepted for publication. However, that sense of elation can quickly turn to dread if the author has to respond to dozens, even hundreds, of comments from peer reviewers. We have observed that the quality of peer reviews has declined over the past decade, and their focus is often misguided, emphasizing grammar and style over science and methodology. This commentary provides guidance to reviewers on the best practices for conducting a peer review for biomedical journals.

Benefits of being a peer reviewer. Volunteering to be a peer reviewer is an important first step in the peer-review process. Reviewers are typically identified based on their expertise in a particular field. Peer reviewers provide benefit to authors, editors, and journals by providing useful guidance related to a scholarly work, evaluating the methodology and importance of the science, and making recommendations regarding publication decisions. 1 , 2 There are benefits to serving as peer reviewers as well. These include keeping abreast of the latest developments in their field and improving their skills to critically evaluate scientific papers, which may improve their own scientific writing. Lastly, it is an honor and a privilege to be a member of a profession. Thus, there is a moral obligation to give back to one’s professional and scientific community. 3 , 4 For these reasons, participation in peer review is a professional expectation in many settings.

Characteristics of a good peer reviewer. Most biomedical journals use the feedback provided during the peer-review process to make editorial decisions on manuscripts submitted for publication. Typically, editors of peer-reviewed journals seek content experts to perform a critical analysis of the manuscript and provide (1) specific comments regarding the manuscript’s scientific merit, (2) recommendations to authors on how to improve the manuscript, and (3) recommendations to the editor regarding manuscript disposition. Invitations to serve as a peer reviewer are often accompanied by Web links to reviewer resources (e.g., reviewer guidelines), but formal training is usually not provided nor required.

The characteristics of a good peer reviewer are not well defined. While several studies have attempted to identify characteristics that predict reviewer performance, there is little consistency in the findings, and the associations are weak. 5 , – 9 Younger age of the reviewer is 1 of the only characteristics that has been consistently associated with stronger peer reviews. 5 , – 8 Some researchers have found a positive association between the duration of time spent reviewing the article (up to 3 hours) and review quality. 5 , 7

The quality of peer review varies greatly, perhaps partly because few professionals receive training in this area. 1 , 10 However, formal training alone likely only marginally affects the quality of peer reviews. Investigators who conducted a study of more than 300 reviewers who performed 2,856 reviews for the Annals of Emergency Medicine did not find an association between more than 10 hours of training in critical appraisal skills and review quality. 6 However, a BMJ study of 609 reviewers found a small, short-term benefit (<6 months) associated with participation in a peer-reviewer training program. 9 Nevertheless, the variation in the quality of peer reviews suggests that peer reviewers may benefit from some guidance.

What reviewers can do: A step-by-step approach. Although other approaches are acceptable, we recommend a step-by-step process for peer reviewers ( Figure 1). Regardless of the approach used, the best peer reviews are systematic and comprehensive.

Step-by-step process for performing a peer review. 1 , – 4 , 10 , – 13

Step-by-step process for performing a peer review. 1 , – 4 , 10 , – 13

Responding to the invitation to perform a peer review. Peer reviewers should consider several issues before deciding to accept or decline an invitation to perform a peer review. Reviewers should first assess whether they are qualified to review the manuscript. 2 , 10 , 11 Reviewers are typically provided an abstract of the manuscript with the invitation to review. Reviewers should only accept invitations to review if they have expertise in the manuscript’s area of focus. In some cases, the reviewer may have the expertise to review a key section of the manuscript (e.g., research methodology, statistical analysis) but not other sections. In this instance, the reviewer should be clear about the scope of the review before accepting the invitation. 2 , 10

If the manuscript is within the reviewer’s area of expertise, potential or actual conflicts of interest should be considered. 2 , 10 , 11 Potential reviewer conflicts may include close personal (e.g., spouse) or professional (e.g., mentor–mentee, institutional colleague) relationships with 1 or more of the authors. 10 , 11 In addition, the presence of conflicting or competing interests that may create the potential for personal, financial, intellectual, professional, political, or religious benefit by the reviewer may be considered conflicts of interest. 2 , 10 , 11 Reviewers should ask themselves this simple but powerful question: Would others—the authors, editors, or readers—believe that my positive or negative review of this paper is in any way motivated by self-interest? If the answer is yes, it is critical to discuss the potential conflict with the editor before accepting the invitation to review. 2 , 10 , 11

Timeliness is critically important to the peer-review process. To avoid unnecessary delays in the publication process, the reviewer must consider his or her ability to meet the journal’s deadline for completing the review before accepting the invitation. 2 , 10 , 11 A quality peer-review can take several hours to perform. 3 , 5 , 7 , 10 Reviewers are typically given 2 weeks to complete their peer review, though the timeline may be as brief as 48 hours and as long as 4 weeks. 11 Reviewers should respond to peer-review invitations in a timely manner, especially if the invitation is declined. 2 , 12 In some cases, journals will rescind their invitations if a timely response (e.g., within 3–4 days) is not received. Throughout the process, reviewers must maintain confidentiality and should never circulate the manuscript to or discuss its contents with others. 1 , 12 If reviewers would like to include others (e.g., trainees) in the peer-review process, permission should be requested to do so, or at a minimum, the editor should be notified when the critique is submitted. 2 , 12

Performing the peer review. Quick read. Many agree that the first step in completing a peer review is to perform a quick read. 1 , 3 , 4 , 10 , 13 During this first read, reviewers should skim the article to gain a basic understanding of the manuscript’s contents. 13 During the quick read, reviewers should once again assess whether they have the expertise to perform the review and whether any real or potential conflicts of interest exist. Reviewers should notify the editor immediately if these problems are identified. 4 Reviewers should note ambiguities and questions that arise during the quick read and begin to form opinions regarding the importance and relevance of the work. 3 , 10

Another important reason for the quick read is to identify critical or “fatal” flaws of the manuscript. 1 , 4 , 10 , 13 In general, a manuscript’s flaws can be categorized as minor (do not require author action), major (require satisfactory resolution before publication is considered), and fatal (deem the manuscript—even with major revisions—irreparable). 1 , 10 , 13 If reviewers identify a fatal flaw during the quick read, they may recommend to reject the manuscript at this point, forgoing a more-detailed read and written critique. 1 , 4 , 10 , 13 Common examples of manuscript flaws are listed in Table 1.

Manuscript Flaws That May Warrant Revision or Rejection During Peer Review 1 , 10 , 13

Type of Flaw . Characteristics .
Minor flaws Writing style is poor, lack of clarity, manuscript organization and flow are poor, tables or figures duplicate information in the text, referencing errors
Major flaws Methodology omits analyses that may affect results (e.g., control for critical variables), methods are poorly described, unclear data tables or figures, confirmatory data that add little to current understanding in the absence of strong arguments justifying the repetition, important or relevant research is not cited, limitations are poorly acknowledged, conclusions are overstated or contradicted by author’s own data
Fatal flaws Consent not obtained when needed, methods are unethical, methodology has been discredited, methodology is flawed leading to inconclusive or inaccurate results (e.g., ignores a process known to strongly influence the topic of interest)
Type of Flaw . Characteristics .
Minor flaws Writing style is poor, lack of clarity, manuscript organization and flow are poor, tables or figures duplicate information in the text, referencing errors
Major flaws Methodology omits analyses that may affect results (e.g., control for critical variables), methods are poorly described, unclear data tables or figures, confirmatory data that add little to current understanding in the absence of strong arguments justifying the repetition, important or relevant research is not cited, limitations are poorly acknowledged, conclusions are overstated or contradicted by author’s own data
Fatal flaws Consent not obtained when needed, methods are unethical, methodology has been discredited, methodology is flawed leading to inconclusive or inaccurate results (e.g., ignores a process known to strongly influence the topic of interest)

Manuscript Flaws That May Warrant Revision or Rejection During Peer Review 1 , 10 , 13

Type of Flaw . Characteristics .
Minor flaws Writing style is poor, lack of clarity, manuscript organization and flow are poor, tables or figures duplicate information in the text, referencing errors
Major flaws Methodology omits analyses that may affect results (e.g., control for critical variables), methods are poorly described, unclear data tables or figures, confirmatory data that add little to current understanding in the absence of strong arguments justifying the repetition, important or relevant research is not cited, limitations are poorly acknowledged, conclusions are overstated or contradicted by author’s own data
Fatal flaws Consent not obtained when needed, methods are unethical, methodology has been discredited, methodology is flawed leading to inconclusive or inaccurate results (e.g., ignores a process known to strongly influence the topic of interest)
Type of Flaw . Characteristics .
Minor flaws Writing style is poor, lack of clarity, manuscript organization and flow are poor, tables or figures duplicate information in the text, referencing errors
Major flaws Methodology omits analyses that may affect results (e.g., control for critical variables), methods are poorly described, unclear data tables or figures, confirmatory data that add little to current understanding in the absence of strong arguments justifying the repetition, important or relevant research is not cited, limitations are poorly acknowledged, conclusions are overstated or contradicted by author’s own data
Fatal flaws Consent not obtained when needed, methods are unethical, methodology has been discredited, methodology is flawed leading to inconclusive or inaccurate results (e.g., ignores a process known to strongly influence the topic of interest)

Detailed read. After the quick read, reviewers should perform a more detailed, critical read of the manuscript. 3 , 4 , 10 , 13 Some scholars have suggested that this occur hours to days after the first read, as this may allow the reviewer to better prepare for a more critical read by reviewing ancillary materials (e.g., reviewer instructions, figures, tables, supplementary files). 2 , 3 , 13 The major purpose of the detailed read is to determine the scientific merits and relevance of the manuscript. 3 , 4 , 13 To help formulate this evaluation, reviewers should ask themselves 3 questions about the manuscript: Do I understand it? Do I believe it? Do I care? 11 Manuscripts should clearly state the research question or clinical problem and then describe how the scholarly work was conducted. If valid methods are used and the conclusions are supported by the data, the results should be credible. Finally, if the work is original, the question or problem important, or the intervention innovative, it should stimulate interest in the reviewer and readers alike. Reviewers should also note ambiguities and questions that arise during their detailed review and provide suggestions for improvement. 3 , 4 , 13 Importantly, the reviewer should not focus on language, writing style, and grammar unless these affect the clarity of a particular point or the frequency of these issues significantly impairs the manuscript’s overall readability. 3 , 4 , 11 , 13 In such cases, the reviewer should not attempt to rewrite the paper for the authors but rather point out the sections of the manuscript that need to be addressed.

The detailed read should evaluate all aspects of the manuscript, including the title, keywords, abstract, introduction, methods, results, discussion, conclusion, graphs, tables, figures, and references. Most minor and major flaws ( Table 1) are identified during this more-critical read. For research manuscripts, manuscript flaws are commonly found in the methods section. Reviewers should consider using a systematic process (e.g., R andomized, A ttrition, M easurement by b linded assessors or objective measurements [RAMbo]) or follow reporting guidelines when critically evaluating randomized clinical trials (e.g., CONsolidated Standards of Reporting Trials [CONSORT]) or meta-analyses (e.g., QUality Of Reporting Of Meta-analyses [QUORUM]), particularly for reviewers with limited experience. 1 , 4 , 14 , – 16 When evaluating the conclusion section, reviewers should ensure that the authors do not overstate their findings and identify any statements that may be perceived as biased. Although not an exhaustive list, Table 2 lists the characteristics of quality manuscripts worthy of acceptance. Reviewers should not be afraid to recommend conditional acceptance or rejection for those manuscripts that they believe have significant flaws or are insufficiently unique or important.

Characteristics of Quality Manuscripts Worthy of Acceptance 1 , 3 , 4 , 10 , 11 , 14

Section of Manuscript . Characteristics .
Title, keywords, abstract Title is succinct and clearly indicates the nature of the scholarly work, abstract appropriately and accurately summarizes the manuscript contents, no new information is introduced
Introduction Summarizes recent developments in the area of interest highlights existing gaps in knowledge clarifies the originality, novelty, and importance of the work
Methods Follows acceptable standards or uses a novel technique that improves the strength of the research, approaches used are described sufficiently to allow replication of the scholarly work
Results and discussion Summarize data coherently, significance described in the context of current knowledge, limitations and gaps that remain are appropriately described
Conclusion Succinct, reflects upon the aims of the manuscript, consistent with the data
Graphs, tables, figures, references Complement data in the text, does not repeat nor contradict data in the text, relevant literature cited
Section of Manuscript . Characteristics .
Title, keywords, abstract Title is succinct and clearly indicates the nature of the scholarly work, abstract appropriately and accurately summarizes the manuscript contents, no new information is introduced
Introduction Summarizes recent developments in the area of interest highlights existing gaps in knowledge clarifies the originality, novelty, and importance of the work
Methods Follows acceptable standards or uses a novel technique that improves the strength of the research, approaches used are described sufficiently to allow replication of the scholarly work
Results and discussion Summarize data coherently, significance described in the context of current knowledge, limitations and gaps that remain are appropriately described
Conclusion Succinct, reflects upon the aims of the manuscript, consistent with the data
Graphs, tables, figures, references Complement data in the text, does not repeat nor contradict data in the text, relevant literature cited

Characteristics of Quality Manuscripts Worthy of Acceptance 1 , 3 , 4 , 10 , 11 , 14

Section of Manuscript . Characteristics .
Title, keywords, abstract Title is succinct and clearly indicates the nature of the scholarly work, abstract appropriately and accurately summarizes the manuscript contents, no new information is introduced
Introduction Summarizes recent developments in the area of interest highlights existing gaps in knowledge clarifies the originality, novelty, and importance of the work
Methods Follows acceptable standards or uses a novel technique that improves the strength of the research, approaches used are described sufficiently to allow replication of the scholarly work
Results and discussion Summarize data coherently, significance described in the context of current knowledge, limitations and gaps that remain are appropriately described
Conclusion Succinct, reflects upon the aims of the manuscript, consistent with the data
Graphs, tables, figures, references Complement data in the text, does not repeat nor contradict data in the text, relevant literature cited
Section of Manuscript . Characteristics .
Title, keywords, abstract Title is succinct and clearly indicates the nature of the scholarly work, abstract appropriately and accurately summarizes the manuscript contents, no new information is introduced
Introduction Summarizes recent developments in the area of interest highlights existing gaps in knowledge clarifies the originality, novelty, and importance of the work
Methods Follows acceptable standards or uses a novel technique that improves the strength of the research, approaches used are described sufficiently to allow replication of the scholarly work
Results and discussion Summarize data coherently, significance described in the context of current knowledge, limitations and gaps that remain are appropriately described
Conclusion Succinct, reflects upon the aims of the manuscript, consistent with the data
Graphs, tables, figures, references Complement data in the text, does not repeat nor contradict data in the text, relevant literature cited

Final scan. Reviewers should perform a final scan of the manuscript, focusing on organization, flow, and other issues that affect clarity. 10 Reviewers should keep in mind that their assessment of the scientific merit of the manuscript is more valuable to the journal editors than their assessment of grammar and writing style. 3 , 4 , 10 , 13 The final quick read also provides the reviewer an opportunity to highlight minor flaws ( Table 1) and offer suggestions for improvement. 1

Written critique. Reviewers should prepare their written critique to the journal editors by first summarizing the manuscript and then providing specific comments related to the identified flaws. 4 , 10 , 11 , 13 Although some reviewers use the first paragraph of the written critique to summarize the manuscript’s contents and findings, we suggest that this summary focus on the reviewer’s overall analysis of the manuscript. 4 , 10 , 11 , 13 When providing their overall analysis of the manuscript, reviewers should highlight the strengths, point out significant flaws, and comment on the importance or significance of the work. 4 , 10 , 11 , 13 Throughout the critique, reviewers should be honest and direct but considerate when providing feedback to authors. 1 , 13 Grammatical issues identified by the reviewer that are pervasive (e.g., numerous spelling errors) should be summarized, leaving detailed review of grammar, spelling, and writing style to the journal’s copyeditors and technical staff. 4 , 10 , 11 , 13 The summary should balance both positive and constructive feedback consistent with the overall evaluation of the manuscript (e.g., the majority of comments should be positive if the recommendation is to accept or accept with minor revisions). 4 , 10 , 11 , 13 If a reviewer reviewed only specific portions of the manuscript, this fact should be clearly stated in the opening summary. 2

Following the summary, reviewers should provide specific comments related to each section of the manuscript. Reviewers should organize their comments in a logical manner and number them. 10 , 11 , 13 In addition, reviewers should specify where within the manuscript (e.g., page and line numbers) the comments are directed. 13 Many reviewers neglect to provide comments about the title and abstract, but these are perhaps the most important elements of the manuscript. Many readers will read or have access to only the title and abstract when conducting a literature search. It is helpful to the authors and the editors to indicate whether a comment identifies a major issue that must be addressed or is merely a recommendation (i.e., minor issue). 4 , 13 Given that one of the purposes of peer review is to improve the quality of manuscripts, reviewer feedback should be objective and constructive, with actionable suggestions for addressing flaws. 2 , 4 , 10 , 11 , 13 Because the classification of a manuscript’s flaws is subjective, if reviewers determine that a fatal flaw exists, an explanation of why the manuscript is beyond repair is warranted. 1 , 10 , 13 When appropriate, feedback should be supported with evidence (e.g., supporting literature) to assist the editors in their decision and the authors in their revisions. 2 , 4 , 10 When a statement in the manuscript is ambiguous and may lead to reader confusion, the reviewer may offer specific suggestions to improve clarity. 4 , 10 , 11 , 13 Reviewers should refrain from divulging their publication recommendation to the authors in the critique. The responsibility to determine the manuscript’s disposition rests with the journal editors.

Reviewers are often asked to provide confidential feedback to the editors. These comments should include a brief explanation to justify the reviewer’s publication recommendation. 13 Other confidential comments that reviewers may wish to include are unvarnished opinions on specific aspects of the manuscript, concerns regarding ethics (e.g., plagiarism, fraud, unethical research practices), and the acknowledgment of individuals who may have assisted in conducting the review. 1 , 4 , 11 , 13

Publication recommendation. In most instances, reviewers are asked to provide their recommendations regarding publication as follows: accept, accept with revisions, or reject. Some journals delineate “accept with minor revisions” from “accept with major revisions.” In general, manuscripts that are well written, novel, scientifically powerful, and relevant to the journal’s readers warrant acceptance. 3 , 11 Manuscripts that do not meet these criteria should be considered for conditional acceptance (pending revisions) or rejection. 3 Manuscripts with only minor flaws should be recommended for conditional acceptance (accept with revisions or accept with minor revisions) with the understanding that the reviewer’s suggestions for improving the manuscript are discretionary and may ultimately be ignored by authors. 4 , 10 In contrast, manuscripts with major flaws should be recommended for conditional acceptance (accept with revisions or accept with major revisions) only if the authors satisfactorily address the reviewers’ concerns. 1 , 4 , 10 When deliberating their publication recommendation, reviewers should give priority to innovative works that may have methodological flaws (minor or major) over manuscripts that are merely confirmatory or likely to be of low interest to the journal’s readership. 4 The use of research methods that are unethical, do not follow contemporary norms (e.g., outdated or discredited methods), or cannot be replicated is grounds for recommending the rejection of a manuscript. 1 , 11 , 13

Conclusion. The quality of peer reviews varies greatly and is often misguided, resulting in challenges for journal editors and frustrations for authors. The use of a stepwise approach when conducting a peer review can improve the quality of reviews, ultimately leading to high-quality articles for publication.


Steps and Standards of a Systematic Review

A systematic review begins with a clearly defined question accompanied by established inclusion and exclusion criteria.

Once you have an initial idea, search for already published literature on the topic. If there is already a systematic review article, then another review on the same topic is not needed. Additionally, reviewing published systematic reviews can help you to better frame your own research question.

If you find a research article that matches your topic, save it. Articles that exemplify your research topic are called "gold-standard" articles, and are used as examples during the search process (Step 3).

Databases for Finding Already Published Reviews

    (Free): A search strategy to retrieve citations identified as systematic reviews or meta-analyses. (NetID required): It's EBM tab can help find systematic reviews or meta-analyses. (Free): A tool to find systematic reviews using PICO (Population, Intervention, Comparison, and Outcome) (NetID required): The resource includes the full text of the regularly updated systematic reviews of the effects of healthcare prepared by The Cochrane Collaboration.

What is a systematic review protocol?

Based on Cochrane, it is a pre-defined plan and the proposed approach for a systematic review. It outlines the question that the review authors are addressing, detailing the criteria against which studies will be assessed for inclusion in the review, and describing how the authors will manage the review process. The protocol also serves as a notification of your plans to other researchers, so that no one will attempt the same project. It's strongly suggested to register your systematic review protocol once it's written. The purpose of registering a systematic review is to reduce publication bias, enhance transparency, and avoid duplication of effort.

A good way to familiarize yourself with writing systematic review research protocols is to take a look at those registered on PROSPERO- the International Prospective Register of Systematic Reviews. Once you have written a draft of your protocol, have it peer reviewed by someone outside of your research team. If it is registered in PROSPERO, it will also be open for peer review by other researchers.

Key information for writing a systematic review protocol include:

  • A research question including patients and population, intervention or exposure, comparison (if relevant), and outcomes.
  • Databases used for the searches and search strategy
  • Criteria for inclusion and exclusion.
  • Methods used to assess risk of bias.
  • Method of data extraction (selection and coding form).
  • Method of data analysis including statistical analysis if relevant.
  • Anticipated or actual start date.
  • Forming a systematic review team and team members' information.
  • Review project timeline.
  • Funding sources/sponsors
  • Conflict of interests.
  • Date of registration.

Besides PROSPERO, there are other organizations that you might search for a registered systematic review protocol:


Effective peer assessment processes: Research findings and future directions

Despite the popularity of peer assessment (PA), gaps in the literature make it difficult to describe exactly what constitutes effective PA. In a literature review, we divided PA into variables and then investigated their interrelatedness. We found that (a) PA's psychometric qualities are improved by the training and experience of peer assessors (b) the development of domain-specific skills benefits from PA-based revision (c) the development of PA skills benefits from training and is related to students' thinking style and academic achievement, and (d) student attitudes towards PA are positively influenced by training and experience. We conclude with recommendations for future research.


What are the ethical guidelines for peer reviewers?

All peer reviewers must follow these ethical guidelines for Taylor & Francis journal articles in review:

  • Reviewers must give unbiased consideration to each manuscript submitted. They should judge each on its merits, without regard to race, religion, nationality, sex, seniority, or institutional affiliation of the author(s).
  • Reviewers must declare any conflict of interest before agreeing to review a manuscript. This includes any relationship with the author that may bias their review.
  • Reviewers must keep the peer review process confidential. They must not share information or correspondence about a manuscript with anyone outside of the peer review process.
  • Reviewers should provide a constructive, comprehensive, evidenced, and appropriately substantial peer review report.
  • Reviewers must avoid making statements in their report which might be construed as impugning any person’s reputation.
  • Reviewers should make all reasonable effort to submit their report and recommendation on time. They should inform the editor if this is not possible.
  • Reviewers should call to the journal editor’s attention any significant similarity between the manuscript under consideration and any published paper or submitted manuscripts of which they are aware.

Taylor & Francis recommend that reviewers also adhere to the COPE Ethical Guidelines for Peer Reviewers.

Read key findings from our white paper research – Peer review: a global view


Watch the video: Are there any reading rules? Существуют какие либо правила чтения? (August 2022).