Hypothesis Testing Skill

Purpose

Guide appropriate selection and interpretation of statistical hypothesis tests for research data analysis.

Test Selection Decision Tree

Step 1: How many variables?

One variable:

• Categorical → Chi-square goodness of fit
• Continuous → One-sample t-test

Two variables:

• Both categorical → Chi-square test of independence
• One categorical, one continuous → T-test or ANOVA
• Both continuous → Correlation or regression

Three+ variables:

• Multiple predictors → Multiple regression or ANOVA
• Complex designs → Mixed models or advanced methods

Step 2: Check assumptions

For t-tests:

1. Independence of observations
2. Normality (especially for small N)
3. Homogeneity of variance

Violations?

• Non-normal → Mann-Whitney U (non-parametric)
• Unequal variance → Welch's t-test
• Dependent observations → Paired t-test or mixed models

For ANOVA:

1. Independence
2. Normality
3. Homogeneity of variance
4. No outliers

Violations?

• Non-normal → Kruskal-Wallis test
• Unequal variance → Welch's ANOVA
• Outliers → Robust methods or transformation

Step 3: Interpret results

Always report:

1. Test statistic (t, F, χ²)
2. Degrees of freedom
3. p-value
4. Effect size with CI
5. Descriptive statistics

Example:

Independent samples t-test showed a significant difference between 
groups, t(98) = 3.45, p < .001, d = 0.69, 95% CI [0.29, 1.09]. 
The experimental group (M = 45.2, SD = 8.3) scored higher than 
control (M = 37.8, SD = 9.1).

Common Tests Reference

| Research Question | Test | Assumptions | |------------------|------|-------------| | 2 groups, continuous outcome | Independent t-test | Normality, equal variance | | 2 measurements, same people | Paired t-test | Normality of differences | | 3+ groups, one factor | One-way ANOVA | Normality, homogeneity | | 3+ groups, multiple factors | Factorial ANOVA | Normality, homogeneity | | Relationship between variables | Pearson correlation | Linearity, normality | | Predict continuous outcome | Linear regression | Linearity, normality of residuals | | 2 categorical variables | Chi-square test | Expected frequencies ≥5 | | Ordinal data, 2 groups | Mann-Whitney U | None (non-parametric) | | Ordinal data, paired | Wilcoxon signed-rank | None (non-parametric) |

Assumption Checking

Normality

Visual: Q-Q plot, histogram
Statistical: Shapiro-Wilk test (N < 50), Kolmogorov-Smirnov (N ≥ 50)
Guideline: Robust to moderate violations if N ≥ 30

Homogeneity of Variance

Visual: Box plots, residual plots
Statistical: Levene's test, Bartlett's test
Guideline: Ratio of largest/smallest variance < 4

Independence

Check: Research design, data collection
Red flags: Time series, clustered data, repeated measures
Solution: Use appropriate model (mixed effects, GEE)

Integration

Use with data-analyst agent for complete statistical analysis workflow and experiment-designer agent for planning appropriate analyses.

---

Version: 1.0.0