polars-dovmed

Search PubMed Central Open Access and bioRxiv parquet corpora with polars-dovmed.

Use the bundled helper, skills/polars-dovmed/scripts/query_literature.py, for hosted API or local parquet-backed searches. The helper auto-loads ~/.config/polars-dovmed/.env.

Run the helper with uv run --script so its pinned Parquet fallback dependency is available. Local runs record --corpus-revision (or DOVMED_CORPUS_REVISION) and fall back from flattened.csv to processed.parquet or the legacy prcoessed.parquet only when the compact output is absent.

Current hosted API defaults:

• Treat API keys as secrets in artifacts. Do not save keys in run directories, memory records, summaries, or final answers.
• Save generated run artifacts under tasks/polars-dovmed-runs/<slug>/ by default, not under skills/polars-dovmed/.
• Prefer structured async search through /api/jobs targeting scan_literature_advanced(mode="discovery").
• Do not use the flat /api/search_literature endpoint for smoke tests or normal skill work. It is opt-in only with --allow-flat-query and may hang behind the edge proxy.
• Do not start with --corpus both. Run --corpus biorxiv and --corpus pmc as separate calls, then merge results.
• For OpenPMC/PMC, do not run a broad unbanded scan. Use the materialized clean year bands in parallel with direct FTS-backed calls: --sync --year-bands clean_split --year-band-workers 4.
• If the user names a specific publication year or narrow range, map it to the matching clean band(s) and search only those bands. Use all clean bands only when no year constraint is given.
• For recent or emerging taxa, run bioRxiv anchor-only discovery and OpenPMC clean-band discovery as separate searches. bioRxiv is small and often returns first; OpenPMC should still use parallel clean bands.
• For any entity-centric prompt with topic modifiers, such as "distribution and genomics of X", first search the exact entity/aliases only. Treat topic words as triage labels or a second-pass refinement after confirming anchor hits; do not build a large first-pass query of generic topic terms.
• Do not invent aliases by splitting compact taxon, gene, or clade names into common words unless the prompt or literature supports that synonym. Keep aliases evidence-based; when a real synonym is multi-word, keep it as one phrase in a single JSON term.
• For OpenPMC clean-band discovery, skip automatic details rerank: pass --skip-details-rerank. The FTS index searches title, abstract, and full text, but PMC details lookups still read parquet and can dominate wall time. Fetch details only for selected PMCID values after discovery.
• Citation metadata should come from the corpus response/details endpoint first. If DOI/year/journal are missing, use bounded Crossref lookup through --crossref-metadata or the crossref-lookup skill. Do not use generic web search for DOI repair except as a final publisher-page check for unresolved/ambiguous records.
• If OpenPMC clean-band search returns Database not found, stop and report that the deployment is not exposing the indexed OpenPMC bands. Do not fall back to a monolithic unbanded OpenPMC scan.
• For interactive work, pass --poll-timeout and, when available, wrap searches in timeout. Do not rerun OpenPMC with longer waits after one bounded clean-band failure.

Public access note:

• omics-skills does not provide a hosted API key or the PMC/bioRxiv parquet corpora.
• Public users can prepare local PMC searches from Uri Neri's upstream package: https://github.com/UriNeri/polars-dovmed.
• Local setup uses upstream dovmed download, dovmed build-parquet, and dovmed scan.
• If no API key and no local corpus exist, state that polars-dovmed is not configured and use another literature fallback.

Instructions

1. Route the task and decide execution mode.
   • Use the hosted API when POLARS_DOVMED_API_KEY is configured in the environment. Never pass the key in argv.
   • Use local dovmed scan when hosted access is unavailable and local parquet paths exist.
   • Do not imply that this repo ships hosted access or local corpora.
2. Create a dedicated run directory.
   • Default: tasks/polars-dovmed-runs/<date-topic>/.
   • Save prompt.txt, query.json, submitted payloads, raw responses, timing files, and any curated summary.
   • Never save secrets.
3. Author a structured query JSON directly.
   • Start with exact anchor names and aliases.
   • Add relation groups only when they improve precision.
   • Use disqualifying_terms for acronym collisions and wrong systems.
   • For "hosts of X" or other recent-taxonomy prompts, first run an anchor-only query for X and aliases; use host terms during triage or a second pass.
   • For "topic of X" prompts, such as environmental distribution, genomics, metabolism, or ecology of an entity, keep the first query anchor-only. Do not add broad words like distribution, genome, environment, host, or taxonomy as peer OR concepts.
4. Inspect the query JSON before searching.
   • Check spelling, taxonomy aliases, regex escaping, and noisy terms.
   • Keep support terms soft in discovery; avoid generic anchors such as host alone.
5. Run discovery first.
   • bioRxiv API path: --queries-file ... --mode discovery --corpus biorxiv.
   • OpenPMC API path: --queries-file ... --mode discovery --corpus pmc --sync --year-bands clean_split --year-band-workers 4 --skip-details-rerank, unless the prompt has a year constraint that maps to fewer bands.
   • Use --extract-matches none --add-group-counts primary.
   • Use bounded polling, for example --poll-timeout 75 for bioRxiv and --poll-timeout 120 for OpenPMC clean bands.
   • Keep --details-rerank-limit modest, usually 8-12, and use it automatically only for bioRxiv or non-year-band searches. For OpenPMC, fetch details by PMCID after inspecting top hits.
6. Inspect the first 5-10 hits.
   • If results are noisy, refine query.json and rerun.
   • For citation-quality answers, first use PMCID/DOI details from polars-dovmed, then --crossref-metadata or /crossref-lookup for missing DOI/year/journal. Use DOI landing pages or journal pages only when Crossref and corpus metadata disagree or remain ambiguous.
7. Record timings when the user asks about speed.
   • Capture wall time with shell time -p or timeout ....
   • Also report helper/API elapsed_ms when present.
   • Timeouts and failed endpoints are valid measured results; do not hide them.

Hosted API Reachability

Minimal reachability check:

curl -sS --max-time 20 https://api.newlineages.com/

Expected: service metadata from the root endpoint. This only proves the service is reachable, not that a corpus scan is healthy.

Structured smoke check:

RUN=tasks/polars-dovmed-runs/smoke-$(date +%Y%m%d)
mkdir -p "$RUN"
cp skills/polars-dovmed/fixtures/smoke_prompt.txt "$RUN/prompt.txt"
cp skills/polars-dovmed/fixtures/smoke_query.json "$RUN/query.json"

timeout 90s uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --queries-file "$RUN/query.json" \
  --corpus biorxiv \
  --mode discovery \
  --max-results 3 \
  --skip-details-rerank \
  --poll-timeout 75 \
  --save-payload "$RUN/payload_smoke.json" \
  --save-response "$RUN/results_smoke.json" \
  > "$RUN/summary_smoke.json" 2> "$RUN/time_smoke.txt"

If timeout is unavailable, still pass --poll-timeout and record that no outer wall-clock guard was available. Avoid /usr/bin/time; use shell time -p unless you have confirmed the path exists.

Known client pitfall: Cloudflare can reject Python urllib's default user agent with HTTP 403 and error code 1010. The helper sends a normal user agent. If raw urllib is unavoidable, send both X-API-Key and User-Agent.

Preferred Workflow

Step 1: Author Query JSON

Use compact concept groups:

{
  "anchor_entity": [
    ["primary_name"],
    ["alias_1"],
    ["alias_2"]
  ],
  "relation_or_property": [
    ["primary_name", "relation_term"],
    ["alias_1", "specific_relation_alias"]
  ],
  "disqualifying_terms": [
    ["term_to_exclude"]
  ]
}

Group logic (enforced consistently on both the FTS fast path and the parquet scan): terms inside one inner group are AND'd (all must co-occur); inner groups within a concept are OR'd; concepts are OR'd, with ranking favoring items that match more groups. A multi-word term is matched as an exact phrase. So list synonyms/aliases as separate single-term groups (each ["alias"] on its own line) — never pack them into one group, which would require them all to co-occur. Use a multi-term group only to require co-occurrence (e.g. ["entity", "relation"]).

For recent taxa or sparse terms, begin with anchor-only JSON:

{"anchor_entity": [["Mirusviricota"], ["mirusvirus"], ["mirusviruses"]]}

Step 2: Search With Hosted API

Fast first pass for emerging taxa:

RUN=tasks/polars-dovmed-runs/mirusviricota-hosts-$(date +%Y%m%d)
mkdir -p "$RUN"
printf '%s\n' "papers describing hosts of Mirusviricota" > "$RUN/prompt.txt"
# Write and inspect "$RUN/query.json" before running the search.

timeout 90s uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --queries-file "$RUN/query.json" \
  --corpus biorxiv \
  --mode discovery \
  --extract-matches none \
  --add-group-counts primary \
  --max-results 25 \
  --details-rerank-limit 12 \
  --poll-timeout 75 \
  --save-payload "$RUN/payload_biorxiv.json" \
  --save-response "$RUN/results_biorxiv.json" \
  > "$RUN/summary_biorxiv.json" 2> "$RUN/time_biorxiv.txt"

OpenPMC pass with parallel clean year bands:

timeout 120s uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --queries-file "$RUN/query.json" \
  --corpus pmc \
  --mode discovery \
  --sync \
  --year-bands clean_split \
  --year-band-workers 4 \
  --extract-matches none \
  --add-group-counts primary \
  --max-results 25 \
  --skip-details-rerank \
  --poll-timeout 120 \
  --save-payload "$RUN/payload_pmc.json" \
  --save-response "$RUN/results_pmc.json" \
  > "$RUN/summary_pmc.json" 2> "$RUN/time_pmc.txt"

For citation metadata enrichment after a shortlist, prefer bounded Crossref fallback over web search:

uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --details PMC6362216 PMC10132079 \
  --corpus pmc \
  --crossref-metadata \
  --crossref-limit 10 \
  --save-payload "$RUN/payload_details.json" \
  --save-response "$RUN/results_details.json"

If the prompt gives a year constraint, map it before searching:

• <=2009 -> pre_2010
• 2010-2020 -> 2010_2020
• 2021-2023 -> 2021_2023
• >=2024 -> 2024_plus

For a range crossing bands, pass an explicit comma list to --year-bands, for example --year-bands 2021_2023,2024_plus. Use a single --year-band only when the whole requested range is inside one band.

Fetch details directly when you already know identifiers:

uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --details PMC6912108 PMC8490762 \
  --corpus pmc \
  --save-payload "$RUN/payload_details.json" \
  --save-response "$RUN/results_details.json"

For bioRxiv details, pass DOI values with --corpus biorxiv.

Step 3: Search Locally With dovmed scan

Use local mode when hosted access is unavailable or explicitly unwanted.

uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --execution-mode local \
  --corpus pmc \
  --local-parquet-pattern "$DOVMED_PMC_PARQUET" \
  --queries-file "$RUN/query.json" \
  --save-payload "$RUN/payload_local.json" \
  --save-response "$RUN/results_local.json"

Local corpus aliases:

• --corpus pmc: set DOVMED_PMC_PARQUET or pass --local-parquet-pattern.
• --corpus biorxiv: set DOVMED_BIORXIV_PARQUET or pass --local-parquet-pattern.
• --corpus both: only with one compatible explicit parquet pattern; otherwise run separate scans.

Search Semantics

• Prefer structured JSON over ad hoc natural-language search strings.
• Build searches around anchor concepts first.
• List alternate names and spelling variants as separate single-term groups within a concept (groups are OR'd); keep a multi-word synonym as one phrase term. Multiple terms in one group are AND'd (all must co-occur) — now enforced on the FTS fast path too, so never pack synonyms into a single group.
• Treat support concepts as refiners, not anchors.
• For "X of Y" prompts, combine X and relation terms inside an OR-of-AND group only after an anchor-only discovery pass if recall is poor.
• Down-rank papers matching only generic support terms or full-text-only background mentions.

Ranking priority:

1. exact anchor hit in title
2. exact anchor hit in abstract
3. anchor plus support co-occurrence in title or abstract
4. multiple distinct relevant group matches
5. full-text-only matches

Quick Reference

| Task | Action | |------|--------| | Run directory | tasks/polars-dovmed-runs/<date-topic>/ | | Preferred query | Authored query.json, inspected before search | | Preferred hosted path | helper scan_literature_advanced(mode="discovery"); async except FTS-backed OpenPMC clean bands | | Emerging taxon first pass | --corpus biorxiv, anchor-only query, bounded timeout | | OpenPMC pass | separate --corpus pmc --sync --year-bands clean_split --year-band-workers 4 --skip-details-rerank call | | Year-constrained OpenPMC | map requested years to one or more clean bands before searching | | Avoid by default | flat /api/search_literature, --corpus both, unbanded broad OpenPMC | | Details endpoint | --details ... --corpus pmc|biorxiv | | Missing DOI/year | --crossref-metadata or skills/crossref-lookup/scripts/lookup --title ... | | Local fallback | --execution-mode local --local-parquet-pattern ... | | Quick verification | uv run --script skills/polars-dovmed/scripts/smoke_test.py --run-dir tasks/polars-dovmed-runs/smoke-test | | Timing | shell time -p, helper elapsed_ms, and timeout status |

Input Requirements

• A literature search prompt or inspected structured query.json.
• Hosted API key or local parquet files for the requested corpus.
• Writable run directory outside the skill source tree.

Output

• prompt.txt, query.json, payload JSON, raw response JSON, timing files, and optional curated summary.
• Paper list with titles, identifiers, corpus, relevance notes, and timing.
• Warnings for timeout, 502, database-not-found, missing metadata, or fallback use.

Quality Gates

• [ ] API key handled as a secret and not persisted.
• [ ] Run directory is outside skills/polars-dovmed/.
• [ ] Query JSON authored and inspected before search.
• [ ] Hosted API root or helper smoke checked before declaring outage.
• [ ] Flat endpoint, --corpus both, and broad unbanded OpenPMC avoided unless explicitly justified.
• [ ] OpenPMC broad searches use direct parallel clean year bands with --sync and --skip-details-rerank; year-constrained searches use only matching bands.
• [ ] Discovery mode used before advanced refinement.
• [ ] First 5-10 hits reviewed and noisy terms refined.
• [ ] Long PMC scans have bounded poll and wall-clock timeout.
• [ ] Timings and failure modes reported when speed is part of the request.
• [ ] Citation metadata uses DB/details first, then bounded Crossref fallback; generic web search is not used for routine DOI repair.

Examples

Mirusviricota Host Search

1. Create tasks/polars-dovmed-runs/mirusviricota-hosts-YYYYMMDD/query.json with:

{"anchor_entity": [["Mirusviricota"], ["mirusvirus"], ["mirusviruses"]], "disqualifying_terms": [["MIRU-VNTR"], ["mycobacterium"]]}

2. Run the bioRxiv hosted command from Step 2.
3. Inspect results_biorxiv.json and any companion details response.
4. Run the direct parallel OpenPMC clean-band command when PMC coverage is needed.
5. Summarize direct host evidence separately from background mentions.

Local PMC Search

RUN=tasks/polars-dovmed-runs/klosneuvirinae-hosts
uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --execution-mode local \
  --corpus pmc \
  --local-parquet-pattern "$DOVMED_PMC_PARQUET" \
  --queries-file "$RUN/query.json" \
  --save-payload "$RUN/payload_local.json" \
  --save-response "$RUN/results_local.json"

Troubleshooting

Issue: Flat /api/search_literature times out Solution: Do not use it for smoke tests. Use the root endpoint plus structured helper smoke.

Issue: --corpus both returns 502 Solution: Run separate --corpus biorxiv and --corpus pmc calls and merge results manually.

Issue: OpenPMC --year-bands clean_split returns Database not found Solution: Stop and report that the hosted deployment is not exposing the indexed OpenPMC bands. Do not retry as an unbanded full-corpus OpenPMC scan.

Issue: OpenPMC clean-band search times out Solution: Report the measured timeout and keep the bioRxiv result. Do not retry the same OpenPMC query with a longer unbanded scan; narrow by user-specified year band or refine the anchor query.

Issue: A query about "topic of X" retrieves generic topic papers Solution: Replace the first-pass query with exact entity aliases only. Use topic terms during manual triage or a focused second pass after confirming anchor hits.

Issue: DOI, year, or journal metadata is missing Solution: Fetch details by PMCID/DOI first. If still incomplete, use --crossref-metadata or the crossref-lookup skill. Avoid generic web search unless both corpus metadata and Crossref are unresolved or conflicting.

Issue: /usr/bin/time is missing Solution: Use shell time -p or record start/end timestamps; do not assume /usr/bin/time exists.

Issue: Artifacts were written under skills/polars-dovmed/runs/ Solution: Move future run artifacts to tasks/polars-dovmed-runs/ and do not commit generated run outputs.

Issue: Hosted API key is missing and local parquet files are missing Solution: State that polars-dovmed is not configured. Use another literature-search skill or ask the user to provide hosted access or local corpora.