FDB Simulation Workloads

Patterns for designing Rust workloads that find bugs through autonomous chaos testing.

Determinism Rules (Critical)

Breaking any rule destroys reproducibility. The simulator requires identical seeds to produce identical execution paths.

| Instead of | Use | Why | |------------|-----|-----| | HashMap / HashSet | BTreeMap / BTreeSet | Iteration order must be deterministic | | rand::random() | context.rnd() | Seeded randomness for reproducibility | | SystemTime::now() | context.now() | Simulation controls time | | Manual retry loops | db.run() | Proper retry handling with maybe_committed | | tokio::spawn() | Never | Simulation uses custom executor |

Operation Alphabet

Design operations across three categories:

| Category | Purpose | Examples | |----------|---------|----------| | Normal | Mirror production traffic distribution | 80% reads, 15% writes, 5% complex updates | | Adversarial | Edge cases customers will send | Empty strings, max-length values, null bytes, boundary integers | | Nemesis | Deliberately break things | Delete random data, crash mid-batch, conflict storms, approach 10MB limit |

Invariant Patterns

Verify correctness continuously during execution, not just at the end.

| Pattern | Description | Example | |---------|-------------|---------| | Reference Model | In-memory copy of expected state; compare in check phase | BTreeMap tracking all expected key-values | | Conservation Laws | Quantities that must remain constant | Total money across accounts unchanged | | Structural Integrity | Data structure validity checks | Index entries point to existing records | | Operation Logging | Log intent in same transaction | Eliminates maybe_committed uncertainty |

Three-Crate Architecture

Separate production frameworks from simulation-testable code:

my-project/
├── my-fdb-service/      # Core FDB operations - NO tokio
├── my-grpc-server/      # Production layer (tokio + tonic)
└── my-fdb-workloads/    # Simulation tests

The service crate contains pure db.run() transaction logic. The server crate wraps it for production. The workloads crate tests actual service code under chaos.

Common Pitfalls

1. Initialization on all clients - Use if self.client_id == 0 to run setup once
2. Ignoring maybe_committed - Check the flag in db.run() closure for idempotency
3. Storing database references - Each phase receives fresh references; don't cache them
4. Wrapping FdbError - Keep errors unwrapped so retry mechanism detects retryable errors
5. Assuming setup is failure-free - FDB knobs randomize; always use db.run() with retry logic

Full Reference

For detailed examples, patterns, and rationale: https://pierrezemb.fr/posts/writing-rust-fdb-workloads-that-find-bugs/