Differential Fuzzer

Always load Debugging skill for reference

The differential fuzzer compares Turso results against SQLite for generated SQL statements to find correctness bugs.

Location

testing/differential-oracle/fuzzer/

Running the Fuzzer

Single Run

# Basic run (100 statements, random seed)
cargo run --bin differential_fuzzer

# With specific seed for reproducibility
cargo run --bin differential_fuzzer -- --seed 12345

# More statements with verbose output
cargo run --bin differential_fuzzer -- -n 1000 --verbose

# Keep database files after run (for debugging)
cargo run --bin differential_fuzzer -- --seed 12345 --keep-files

# All options
cargo run --bin differential_fuzzer -- \
  --seed <SEED>           # Deterministic seed
  -n <NUM>                # Number of statements (default: 100)
  -t <NUM>                # Number of tables (default: 2)
  -c <NUM>                # Columns per table (default: 5)
  --verbose               # Print each SQL statement
  --keep-files            # Persist .db files to disk

Continuous Fuzzing (Loop Mode)

# Run forever with random seeds
cargo run --bin differential_fuzzer -- loop

# Run 50 iterations
cargo run --bin differential_fuzzer -- loop 50

Docker Runner (CI/Production)

# Build and run from repo root
docker build -f testing/differential-oracle/fuzzer/docker-runner/Dockerfile -t fuzzer .
docker run -e GITHUB_TOKEN=xxx -e SLACK_WEBHOOK_URL=xxx fuzzer

Environment variables for docker-runner:

TIME_LIMIT_MINUTES - Total runtime (default: 1440 = 24h)
PER_RUN_TIMEOUT_SECONDS - Per-run timeout (default: 1200 = 20min)
NUM_STATEMENTS - Statements per run (default: 1000)
LOG_TO_STDOUT - Print fuzzer output (default: false)
GITHUB_TOKEN - For auto-filing issues
SLACK_WEBHOOK_URL - For notifications

Output Files

All output goes to simulator-output/ directory:

| File | Description | |------|-------------| | test.sql | All executed SQL statements. Failed statements prefixed with -- FAILED:, errors with -- ERROR: | | schema.json | Database schema at end of run (or at failure) | | test.db | Turso database file (only with --keep-files) | | test-sqlite.db | SQLite database file (only with --keep-files) |

Reproducing Errors

Always follow these steps

Find the seed in the error output:

INFO: Starting differential_fuzzer with config: SimConfig { seed: 12345, ... }

Re-run with that seed:

cargo run --bin differential_fuzzer -- --seed 12345 --verbose --keep-files

Check output files:
- simulator-output/test.sql - Find the failing statement (look for -- FAILED:)
- simulator-output/schema.json - Check table structure at failure time
Create a minimal reproducer
- Create reproducer in .sqltest or in .rs always load Debugging skill for reference
Compare behavior manually: If needed try to compare the behaviour and produce a report in the end. Always write to a tmp file first with Edit tool to test the sql and then pass it to the binaries.
```
# Run failing SQL against SQLite
sqlite3 :memory: < simulator-output/test.sql

# Run against tursodb CLI
tursodb :memory: < simulator-output/test.sql
```

Understanding Failures

Oracle Failure Types

Row set mismatch - Turso returned different rows than SQLite
Turso errored but SQLite succeeded - Turso rejected valid SQL
SQLite errored but Turso succeeded - Turso accepted invalid SQL
Schema mismatch - Tables/columns differ after DDL

Warning (non-fatal)

Unordered LIMIT mismatch - LIMIT without ORDER BY may return different valid rows

Key Source Files

| File | Purpose | |------|---------| | main.rs | CLI parsing, entry point | | runner.rs | Main simulation loop, executes statements on both DBs | | oracle.rs | Compares Turso vs SQLite results | | schema.rs | Introspects schema from both databases | | memory/ | In-memory IO for deterministic simulation |

Tracing

Set RUST_LOG for more detailed output:

RUST_LOG=debug cargo run --bin differential_fuzzer -- --seed 12345