seekdb Hybrid Search Optimizer

Optimize seekdb queries that combine MATCH ... AGAINST, scalar predicates, score ordering, and LIMIT.

Required outcome

Improve measured QPS without changing query semantics. Treat recall@10 >= 0.95 in every measured run as a hard correctness gate, not a secondary metric.

Workflow

Reproduce the baseline with the same dataset, indexes, SQL, warm-up, machine limits, and benchmark scripts used for the target run.
Record three query runs, recall@10 for each run, CPU time, allocations, I/O, and the hottest functions. Do not optimize from a single timing sample.
Trace the complete path from SQL planning through scalar filtering, inverted-index traversal, BM25 scoring, top-k maintenance, and row materialization.
Form one falsifiable bottleneck hypothesis at a time. Prefer changes that reduce unnecessary candidates, scoring work, allocations, copies, virtual calls, or repeated decoding.
Implement the smallest change that tests the hypothesis. Preserve a generic fallback when using SIMD or CPU-specific instructions.
Rebuild and rerun correctness tests before performance tests. Reject a change if any recall run falls below 0.95 or results become nondeterministic.
Compare the median and spread of three warmed runs against baseline. Keep only improvements that survive repeated runs and do not regress memory use or build portability.

High-value investigation areas

Push selective base_id and id predicates as early as the execution model safely permits.
Avoid scoring postings that cannot enter the final top-k; use bounded heaps or equivalent top-k structures where semantics allow.
Reuse buffers and decoded metadata inside hot loops; remove per-document allocation and redundant expression evaluation.
Improve posting-list intersection and iterator advancement without skipping valid candidates.
Check cache locality, branch behavior, lock contention, and accidental materialization of full result sets.
Use SIMD only for isolated hot kernels with runtime or build-time fallback to the portable implementation.

Constraints

Do not change SQL semantics, silently lower recall, hard-code benchmark answers, or bypass required filters.
Do not modify global optimization flags, project dependencies, build.sh, or broad CMake settings merely to inflate results.
Keep patches reviewable and document the mechanism behind each performance gain.
Do not claim success from compilation alone; report baseline and optimized QPS plus all recall@10 values.

Deliverable

Return a concise report containing the bottleneck evidence, changed files, why the optimization is safe, build/test commands, baseline versus optimized measurements, recall results, and any remaining risks.