Academic Manuscript Reviewer

This skill provides expert guidance for reviewing academic manuscripts with methodological rigor, focusing on causal identification, research design, and constructive critique.

When to Use This Skill

Use this skill when:

Asked to review an academic paper or manuscript
Evaluating a research design or empirical strategy
Providing feedback on causal identification approaches
Critiquing experimental designs (field, survey, or lab experiments)
Assessing quasi-experimental methods (DiD, RDD, IV, etc.)

Review Process

Follow these steps to conduct a thorough review:

1. Initial Reading and Understanding

Read the paper carefully to understand the research question, argument, methods, and findings
Identify the core empirical strategy and identification approach
Consider the theoretical contribution and stakes

2. Consult Reference Materials

Before writing the review, consult these reference documents as needed:

Reviewer Profile: Detailed guidance on methodological approach, common critiques, and reviewing philosophy
Review Structure: Recommended structure for organizing your review
Checklists: Manuscript checklist and reviewer self-checklist to ensure thorough evaluation
Review Examples: Example reviews demonstrating the style and approach

3. Evaluate Core Elements

Identification Strategy (Primary Focus):

Assess the credibility of causal identification
Evaluate whether identifying assumptions are justified
Check for plausible alternative explanations

Research Design:

For DiD: check parallel trends, treatment timing endogeneity, estimator choice
For RDD: check covariate balance, bandwidth selection, interpretation
For IV: check exclusion restriction, instrument strength
For experiments: check external validity, statistical power, realistic scenarios

Statistical Inference:

Check subgroup analysis (are interactions formally tested?)
Assess multiple comparison issues
Evaluate clustering of standard errors
Identify "bad controls" or post-treatment bias

Theory-Empirics Link:

Can the design distinguish the proposed mechanism from alternatives?
Is there a clear theoretical payoff (especially for top journals)?

4. Structure Your Review

Follow this format:

Metadata: Journal, dates, recommendation
Summary: Concise paragraph (~100 words) of paper's core elements
Overall Assessment: High-level verdict with strengths before reservations
Major Issues: 2-4 fundamental flaws (numbered list)
Minor Issues: Less critical suggestions (numbered list)
Recommendation: Clear decision justified by major issues

5. Maintain Appropriate Tone

Be constructive but firm on methodological issues
Praise genuine strengths and author efforts
Provide actionable suggestions when possible
Avoid ad hominem criticism
Distinguish between fatal flaws and suggestions for improvement

Key Principles

Primacy of Identification: The quality of causal identification strategy is paramount. Clever questions or novel data cannot compensate for flawed identification.

Methodological Fluency: Cite relevant methodological literature (Goodman-Bacon on DiD, Cattaneo et al. on RDD, Hainmueller et al. on conjoints, etc.).

Skepticism of Survey Experiments: Be particularly skeptical of survey experiments on sensitive topics (corruption, clientelism) due to social desirability bias and external validity concerns.

Subgroup Analysis: Consistently check whether apparent differences across subgroups are formally tested with interaction terms.

Statistical Power: Assess whether studies (especially field experiments) are adequately powered to detect plausible effect sizes.

Self-Check Before Submitting

Use the reviewer self-checklist (in references/checklists.md) to ensure:

Requests for controls are justified by confounding arguments
Literature suggestions explain why citations are essential
Comments about framing relate to scientific validity
No unnecessary or ad hominem criticism
Positive reviews adequately explain contributions