Rust Librarian

Semantic knowledge system for Rust development. Ingests authoritative Rust sources into Qdrant and provides search tools for agents planning or writing Rust code.

Architecture

Mirrors the Go docs indexer pattern from os/agent/performance-engine/:

[Rust Book Sources (GitHub markdown)]
        |
        v
[Chunker: split by chapter/section/concept]
        |
        v
[Embedder: sentence-transformers all-mpnet-base-v2 (768D)]
        |
        v
[Qdrant Collections]
  - project-rust-stdlib-framework-docs   (std library API docs)
  - project-rust-books-guides            (The Rust Book, Rustonomicon, etc.)
  - project-rust-patterns                (Design patterns, idioms)
  - project-rust-error-examples          (Compiler errors + fixes)
  - project-rust-crate-docs              (Popular crate documentation)
        |
        v
[Search API: POST /search with metadata filtering]

Commands

/rust-librarian ingest <source>

Ingest a Rust knowledge source into Qdrant.

Supported sources:

| Source | Repository | Collection | |--------|-----------|------------| | rust-book | github.com/rust-lang/book | project-rust-books-guides | | rust-by-example | github.com/rust-lang/rust-by-example | project-rust-books-guides | | rustonomicon | github.com/rust-lang/nomicon | project-rust-books-guides | | rust-api-guidelines | github.com/rust-lang/api-guidelines | project-rust-patterns | | rust-design-patterns | github.com/rust-unofficial/patterns | project-rust-patterns | | rust-cookbook | github.com/rust-lang-nursery/rust-cookbook | project-rust-patterns | | effective-rust | github.com/brson/effective-rust (or local) | project-rust-patterns | | std | doc.rust-lang.org/std | project-rust-stdlib-framework-docs | | error-index | doc.rust-lang.org/error_codes | project-rust-error-examples | | crate:<name> | docs.rs/<name> | project-rust-crate-docs |

Ingestion pipeline:

# 1. Clone source
git clone --depth 1 https://github.com/rust-lang/book ~/lev/workshop/intake/rust-book

# 2. Run chunker (Python, reuses os/agent schema)
python3 rust_librarian/ingest.py --source rust-book --path ~/lev/workshop/intake/rust-book/src/

/rust-librarian search <query>

Search the Rust knowledge base. Returns ranked results with source attribution.

# Pattern search
/rust-librarian search "lifetime elision rules"

# Error search
/rust-librarian search "error[E0502] cannot borrow as mutable"

# Idiom search
/rust-librarian search "builder pattern in Rust"

/rust-librarian status

Show ingestion status across all collections.

Chunking Strategy

Different sources need different chunking:

Books (The Rust Book, Rustonomicon)

• Unit: Chapter section (## heading)
• Metadata: chapter_number, section_title, book_name
• Hierarchy: CONCEPT
• DocumentType: GUIDE or TUTORIAL
• Content: Full section text including code examples

API Docs (std library)

• Unit: Per item (struct, enum, fn, trait, module)
• Metadata: module_path, item_type, signature
• Hierarchy: FUNCTION / CLASS / MODULE / PACKAGE
• DocumentType: API
• Content: Signature + documentation + examples

Patterns / Guidelines

• Unit: Per pattern (one pattern = one chunk)
• Metadata: pattern_name, category, applicability
• Hierarchy: CONCEPT
• DocumentType: STANDARD
• Content: Description + when to use + code example + tradeoffs

Error Examples

• Unit: Per error code
• Metadata: error_code (E0xxx), category (borrow, lifetime, type)
• Hierarchy: CONCEPT
• DocumentType: EXAMPLE
• Content: Error message + explanation + fix + before/after code

Qdrant Schema

Reuses DocumentMetadata from os/agent/performance-engine/semantic-search/schemas/metadata.py:

DocumentMetadata(
    id="rust-book-ch04-ownership",
    content_type=ContentType.FRAMEWORK,   # or PRINCIPLE for patterns
    project="rust-knowledge",
    scope=Scope.GLOBAL,
    framework="rust",
    document_type=DocumentType.GUIDE,
    hierarchy_level=HierarchyLevel.CONCEPT,
    authority="external",                  # from official Rust docs
    source_file="src/ch04-01-what-is-ownership.md",
    source_url="https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html",
    tags=["ownership", "memory", "borrowing"],
    keywords=["ownership", "move", "copy", "drop", "stack", "heap"],
    summary="Rust ownership: each value has one owner, transfers on assignment",
    custom_fields={
        "book": "the-rust-programming-language",
        "chapter": 4,
        "section": "4.1",
        "rust_concepts": ["ownership", "move-semantics"]
    }
)

Collection config (768D, Cosine, HNSW m=16 ef=100):

VectorParams(size=768, distance=Distance.COSINE)
HnswConfigDiff(m=16, ef_construct=100)

Integration Points

SDLC Planning (workflow-rust-plan)

The workflow-rust-plan calls this skill's search before generating code plans:

1. Extract Rust concepts from task description
2. Query project-rust-patterns for relevant idioms
3. Query project-rust-error-examples for known pitfalls
4. Inject results into planning context

Compile-Fix Loop (RustCoder pattern)

When cargo build fails:

1. Extract error code from compiler output (use --message-format=json)
2. Query project-rust-error-examples for the error code
3. Return fix examples + explanation to the fixing LLM

Agent Context Enrichment

Any agent writing Rust can call:

/rust-librarian search "how to implement Display trait"

Bootstrapping (priority order)

/rust-librarian ingest rust-book            # Foundation (~20 chapters)
/rust-librarian ingest rust-by-example      # Practical examples
/rust-librarian ingest error-index          # All compiler error codes
/rust-librarian ingest rust-design-patterns  # Idioms and patterns
/rust-librarian ingest rust-api-guidelines   # API design rules
/rust-librarian ingest rustonomicon         # Unsafe Rust
/rust-librarian ingest crate:serde          # Serialization
/rust-librarian ingest crate:tokio          # Async runtime
/rust-librarian ingest crate:clap           # CLI parsing

Dependencies

• Python 3.8+ with sentence-transformers, qdrant-client
• Qdrant on localhost:6333 (or QDRANT_HOST)
• Git for cloning book sources
• Reuses schema from os/agent/performance-engine/semantic-search/

Related

• os/agent/performance-engine/ -- Go docs indexer (prior art, same architecture)
• workflow-rust-plan -- Planning workflow that consumes search
• plugins/core-sdlc/ -- SDLC plan-execute-verify flow
• RustCoder (~/lev/workshop/intake/RustCoder/) -- Compile-fix loop reference