Nomos Test Execution Guide

This skill orchestrates test execution for the Nomos monorepo, providing command guidance and verification workflows following the project's testing standards.

When to Use This Skill

• Running the test suite (unit, integration, or all tests)
• Debugging test failures (hermetic vs integration)
• Validating test coverage
• Executing mandatory verification checklists from AGENTS.md files
• Regenerating golden files for parser/compiler tests
• Running race detector
• Testing specific modules independently

Testing Standards Overview

• 80% minimum coverage required for all modules
• 100% coverage for critical business logic paths
• Table-driven tests as standard pattern
• Integration tests require //go:build integration tag
• Test isolation using t.TempDir()
• Hermetic tests - no network/external deps in unit tests

Quick Command Reference

Basic Test Commands

# Run all tests (unit only, fast)
make test

# Run with race detector
make test-race

# Run unit tests only (explicit)
make test-unit

# Run integration tests only
make test-integration

# Run all tests (unit + integration)
make test-all

# Test specific module
make test-module MODULE=libs/parser
make test-module MODULE=libs/compiler
make test-module MODULE=apps/command-line

# Generate coverage reports
make test-coverage

Build Commands (Required Before Some Tests)

# Build all applications
make build

# Build CLI specifically
make build-cli

# Build test binaries (required for integration tests)
make build-test

Development Commands

# Format code before testing
make fmt

# Tidy dependencies
make mod-tidy

# Lint (requires golangci-lint)
make lint

# Full verification workflow
make fmt && make test && make lint

Test Execution Workflows

Workflow 1: Basic Unit Testing

When: Regular development, fast feedback loop

# 1. Run unit tests
make test

# 2. If failures, run specific module
make test-module MODULE=libs/parser

# 3. Check race conditions if suspicious
make test-race

Expected output:

ok      github.com/autonomous-bits/nomos/libs/parser    0.123s  coverage: 92.1% of statements
ok      github.com/autonomous-bits/nomos/libs/compiler  0.456s  coverage: 88.5% of statements

Workflow 2: Pre-Commit Verification

When: Before committing code

# 1. Format code
make fmt

# 2. Run all unit tests
make test

# 3. Run race detector
make test-race

# 4. Lint if available
make lint

Workflow 3: Full Integration Testing

When: Before PR, release preparation, debugging complex issues

# 1. Build test binaries first (critical!)
make build-test

# 2. Run integration tests
make test-integration

# 3. Run all tests (unit + integration)
make test-all

Common mistake: Forgetting make build-test causes integration tests to fail with "binary not found" errors.

Workflow 4: Module-Specific Testing

When: Working on specific module, faster iteration

# Parser module
make test-module MODULE=libs/parser

# Compiler module
make test-module MODULE=libs/compiler

# CLI module
make test-module MODULE=apps/command-line

Workflow 5: Coverage Validation

When: Ensuring coverage requirements met

# 1. Generate coverage report
make test-coverage

# 2. Open HTML report
open coverage.html  # macOS
xdg-open coverage.html  # Linux
start coverage.html  # Windows

# 3. Verify minimum 80% coverage
# Look for files below threshold

Check coverage manually:

go test -coverprofile=coverage.out ./...
go tool cover -func=coverage.out | grep total
# Example output: total: (statements) 88.2%

Debugging Test Failures

Failure Type 1: Unit Test Failures

Symptom: Test fails in regular make test run

Debug steps:

# 1. Run specific test verbosely
cd libs/parser
go test -v -run TestParsing

# 2. Check for table-driven test case
# Look at test output to identify failing case

# 3. Run with race detector
go test -race -run TestParsing

# 4. Check test helper usage
# Ensure helpers marked with t.Helper()

Failure Type 2: Integration Test Failures

Symptom: Test fails only with make test-integration

Debug steps:

# 1. Verify build tags present
grep -r "//go:build integration" libs/compiler/test/

# 2. Ensure test binary built
make build-test
ls -la bin/

# 3. Run integration test verbosely
cd libs/compiler
go test -v -tags=integration ./test/...

# 4. Check for external dependencies
# - Network calls?
# - File system operations?
# - Provider binary execution?

Failure Type 3: Golden File Mismatches (Parser)

Symptom: "output mismatch" errors comparing actual vs golden files

Debug steps:

# 1. Review actual output
cat testdata/fixtures/test.csl
cat testdata/golden/test.csl.json

# 2. If intentional change, regenerate golden files
cd libs/parser
rm testdata/golden/test.csl.json
go test  # Will regenerate

# 3. Review diff before committing
git diff testdata/golden/

# 4. Run tests again to verify
go test

Failure Type 4: Hermetic Test Violations

Symptom: Tests pass locally but fail in CI

Debug steps:

# 1. Check for hermetic violations:
# - Network calls in unit tests
# - Hardcoded paths
# - Time-dependent behavior
# - External dependencies

# 2. Verify test uses t.TempDir()
grep -A 10 "func Test" libs/compiler/*_test.go | grep TempDir

# 3. Check for missing build tags
# Unit tests should NOT have //go:build tags
# Integration tests MUST have //go:build integration

# 4. Isolate test
go test -run TestSpecific -count=1 -v

Failure Type 5: Race Detector Warnings

Symptom: make test-race reports data races

Debug steps:

# 1. Run race detector with failing test
go test -race -run TestConcurrent -v

# 2. Examine race report
# Shows: goroutine stack traces where race occurs

# 3. Common causes:
# - Missing mutex protection
# - Shared map access
# - Provider caching without locks

# 4. Fix and re-run
make test-race

Module-Specific Test Patterns

Parser Module (libs/parser)

Test organization:

testdata/
  fixtures/     # Input .csl files
  golden/       # Expected AST outputs
  errors/       # Error test cases

Common commands:

cd libs/parser

# Run all tests
go test ./...

# Run benchmarks
go test -bench=. -benchmem

# Regenerate golden files (after intentional changes)
rm testdata/golden/*.json
go test

# Check coverage
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html

Expected behavior:

• Golden files auto-generate if missing (review before commit!)
• Benchmarks should meet performance goals (see AGENTS.md)
• Coverage should exceed 80%

Compiler Module (libs/compiler)

Test organization:

testdata/          # Test fixtures
test/              # Integration tests
testutil/          # Fake providers, test helpers

Common commands:

cd libs/compiler

# Unit tests only
go test -run "^Test" ./...

# Integration tests (hermetic)
make build-test  # Critical first step
go test -tags=integration ./test/...

# All tests
go test ./... && go test -tags=integration ./test/...

# Test provider resolution specifically
go test -v -run TestProvider

Expected behavior:

• Hermetic tests use fake providers from testutil/
• Integration tests require real provider binaries
• Provider process cleanup verified (no zombie processes)

CLI Module (apps/command-line)

Test organization:

testdata/          # Test configs
test/              # Integration tests (CLI invocation)
internal/          # Unit tests alongside code

Common commands:

cd apps/command-line

# Build CLI first
make build-cli

# Unit tests
go test ./internal/...

# Integration tests (invoke CLI binary)
make build-test
go test -tags=integration ./test/...

# Test specific command
go test -v -run TestInit ./internal/initcmd/

Expected behavior:

• Integration tests invoke actual CLI binary
• Exit codes verified
• Stdout/stderr captured and validated

Verification Checklists (Mandatory)

Each module has verification requirements in AGENTS.md. Follow these before completing tasks:

Universal Checklist (All Modules)

# ✅ Build Verification
make build
# All code must compile without errors

# ✅ Unit Test Verification
make test
# All existing tests must pass

# ✅ Race Detector
make test-race
# No data races reported

# ✅ Linting (if golangci-lint installed)
make lint
# No errors (warnings acceptable if documented)

# ✅ Coverage Check
make test-coverage
# Minimum 80% overall, 100% for critical paths

Parser Module Checklist

cd libs/parser

# ✅ Build
go build ./...

# ✅ Unit Tests
go test ./...
# All tests pass

# ✅ Race Detector
go test -race ./...

# ✅ Golden Files
git diff testdata/golden/
# Review any changes to golden files

# ✅ Benchmarks
go test -bench=. -run=^$
# Performance meets goals

# ✅ Linting
go vet ./...
golangci-lint run

# ✅ Coverage
go test -coverprofile=coverage.out ./...
go tool cover -func=coverage.out | grep total
# Minimum 80%

Compiler Module Checklist

cd libs/compiler

# ✅ Build
go build ./...

# ✅ Unit Tests
go test ./...

# ✅ Integration Tests
go test -tags=integration ./test/...

# ✅ Race Detector
go test -race ./...

# ✅ Linting
go vet ./...
golangci-lint run

# ✅ Coverage
go test -coverprofile=coverage.out ./...
# Check: >= 80%

CLI Module Checklist

cd apps/command-line

# ✅ Build CLI
make build-cli
./nomos --help
# Binary executes and shows help

# ✅ Unit Tests
go test ./internal/...

# ✅ Integration Tests
make build-test
go test -tags=integration ./test/...

# ✅ Race Detector
go test -race ./...

# ✅ Exit Codes
go test -v -run TestExitCode ./test/...
# Verify: 0 (success), 1 (errors), 2 (usage)

# ✅ Linting
go vet ./...
golangci-lint run

# ✅ Coverage
go test -coverprofile=coverage.out ./...

Common Issues and Solutions

Issue: "Test binary not found"

Solution:

make build-test

Integration tests require pre-built binaries.

Issue: "go.work out of sync"

Solution:

make work-sync
make mod-tidy

Issue: Tests pass locally, fail in CI

Causes:

1. Missing integration build tags
2. Hermetic test violations (network calls)
3. Platform-specific assumptions
4. Uncommitted test fixtures

Solution:

# Check build tags
grep -r "//go:build integration" .

# Check for network calls in unit tests
grep -r "http\." *_test.go

# Test in clean environment
git clean -xdf
make test

Issue: Coverage below 80%

Solution:

# Identify uncovered code
go test -coverprofile=coverage.out ./...
go tool cover -func=coverage.out | grep -v "100.0%"

# Add tests for uncovered functions
# Focus on critical paths first (must be 100%)

Issue: Race detector warnings

Solution:

# Run with race detector
go test -race -v ./...

# Fix identified races:
# - Add mutex protection
# - Use channels for coordination
# - Copy data instead of sharing

Best Practices

1. Always run tests before committing:

   make fmt && make test && make test-race
   

2. Test specific module during development:

   make test-module MODULE=libs/parser
   

3. Build test binaries before integration tests:

   make build-test
   

4. Check coverage periodically:

   make test-coverage
   

5. Run full suite before PR:

   make build-test && make test-all && make test-race
   

6. Review golden file changes carefully:

   git diff testdata/golden/
   

Reference Documentation

For complete testing guidelines, see:

• docs/TESTING_GUIDE.md
• apps/command-line/AGENTS.md
• libs/compiler/AGENTS.md
• libs/parser/AGENTS.md