polars-dovmed

Search PubMed Central Open Access and bioRxiv parquet corpora with polars-dovmed.

Use the bundled helper, skills/polars-dovmed/scripts/query_literature.py, for hosted API or local parquet-backed searches. The helper auto-loads ~/.config/polars-dovmed/.env.

Run the helper with uv run --script so its pinned Parquet fallback dependency is available. Local runs record --corpus-revision (or DOVMED_CORPUS_REVISION) and fall back from flattened.csv to processed.parquet or the legacy prcoessed.parquet only when the compact output is absent.

Current hosted API defaults:

Treat API keys as secrets in artifacts. Do not save keys in run directories, memory records, summaries, or final answers.
Save generated run artifacts under tasks/polars-dovmed-runs/<slug>/ by default, not under skills/polars-dovmed/.
Prefer structured async search through /api/jobs targeting scan_literature_advanced(mode="discovery").
Do not use the flat /api/search_literature endpoint for smoke tests or normal skill work. It is opt-in only with --allow-flat-query and may hang behind the edge proxy.
Do not start with --corpus both. Run --corpus biorxiv and --corpus pmc as separate calls, then merge results.
For OpenPMC/PMC, do not run a broad unbanded scan. Use the materialized clean year bands in parallel with direct FTS-backed calls: --sync --year-bands clean_split --year-band-workers 4.
If the user names a specific publication year or narrow range, map it to the matching clean band(s) and search only those bands. Use all clean bands only when no year constraint is given.
For recent or emerging taxa, run bioRxiv anchor-only discovery and OpenPMC clean-band discovery as separate searches. bioRxiv is small and often returns first; OpenPMC should still use parallel clean bands.
For any entity-centric prompt with topic modifiers, such as "distribution and genomics of X", first search the exact entity/aliases only. Treat topic words as triage labels or a second-pass refinement after confirming anchor hits; do not build a large first-pass query of generic topic terms.
Do not invent aliases by splitting compact taxon, gene, or clade names into common words unless the prompt or literature supports that synonym. Keep aliases evidence-based; when a real synonym is multi-word, keep it as one phrase in a single JSON term.
For OpenPMC clean-band discovery, skip automatic details rerank: pass --skip-details-rerank. The FTS index searches title, abstract, and full text, but PMC details lookups still read parquet and can dominate wall time. Fetch details only for selected PMCID values after discovery.
Citation metadata should come from the corpus response/details endpoint first. If DOI/year/journal are missing, use bounded Crossref lookup through --crossref-metadata or the crossref-lookup skill. Do not use generic web search for DOI repair except as a final publisher-page check for unresolved/ambiguous records.
If OpenPMC clean-band search returns Database not found, stop and report that the deployment is not exposing the indexed OpenPMC bands. Do not fall back to a monolithic unbanded OpenPMC scan.
For interactive work, pass --poll-timeout and, when available, wrap searches in timeout. Do not rerun OpenPMC with longer waits after one bounded clean-band failure.

Public access note:

omics-skills does not provide a hosted API key or the PMC/bioRxiv parquet corpora.
Public users can prepare local PMC searches from Uri Neri's upstream package: https://github.com/UriNeri/polars-dovmed.
Local setup uses upstream dovmed download, dovmed build-parquet, and dovmed scan.
If no API key and no local corpus exist, state that polars-dovmed is not configured and use another literature fallback.

Instructions

Route the task and decide execution mode.
- Use the hosted API when POLARS_DOVMED_API_KEY is configured in the environment. Never pass the key in argv.
- Use local dovmed scan when hosted access is unavailable and local parquet paths exist.
- Do not imply that this repo ships hosted access or local corpora.
Create a dedicated run directory.
- Default: tasks/polars-dovmed-runs/<date-topic>/.
- Save prompt.txt, query.json, submitted payloads, raw responses, timing files, and any curated summary.
- Never save secrets.
Author a structured query JSON directly.
- Start with exact anchor names and aliases.
- Add relation groups only when they improve precision.
- Use disqualifying_terms for acronym collisions and wrong systems.
- For "hosts of X" or other recent-taxonomy prompts, first run an anchor-only query for X and aliases; use host terms during triage or a second pass.
- For "topic of X" prompts, such as environmental distribution, genomics, metabolism, or ecology of an entity, keep the first query anchor-only. Do not add broad words like distribution, genome, environment, host, or taxonomy as peer OR concepts.
Inspect the query JSON before searching.
- Check spelling, taxonomy aliases, regex escaping, and noisy terms.
- Keep support terms soft in discovery; avoid generic anchors such as host alone.
Run discovery first.
- bioRxiv API path: --queries-file ... --mode discovery --corpus biorxiv.
- OpenPMC API path: --queries-file ... --mode discovery --corpus pmc --sync --year-bands clean_split --year-band-workers 4 --skip-details-rerank, unless the prompt has a year constraint that maps to fewer bands.
- Use --extract-matches none --add-group-counts primary.
- Use bounded polling, for example --poll-timeout 75 for bioRxiv and --poll-timeout 120 for OpenPMC clean bands.
- Keep --details-rerank-limit modest, usually 8-12, and use it automatically only for bioRxiv or non-year-band searches. For OpenPMC, fetch details by PMCID after inspecting top hits.
Inspect the first 5-10 hits.
- If results are noisy, refine query.json and rerun.
- For citation-quality answers, first use PMCID/DOI details from polars-dovmed, then --crossref-metadata or /crossref-lookup for missing DOI/year/journal. Use DOI landing pages or journal pages only when Crossref and corpus metadata disagree or remain ambiguous.
Record timings when the user asks about speed.
- Capture wall time with shell time -p or timeout ....
- Also report helper/API elapsed_ms when present.
- Timeouts and failed endpoints are valid measured results; do not hide them.

Hosted API Reachability

Minimal reachability check:

curl -sS --max-time 20 https://api.newlineages.com/

Expected: service metadata from the root endpoint. This only proves the service is reachable, not that a corpus scan is healthy.

Structured smoke check:

RUN=tasks/polars-dovmed-runs/smoke-$(date +%Y%m%d)
mkdir -p "$RUN"
cp skills/polars-dovmed/fixtures/smoke_prompt.txt "$RUN/prompt.txt"
cp skills/polars-dovmed/fixtures/smoke_query.json "$RUN/query.json"

timeout 90s uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --queries-file "$RUN/query.json" \
  --corpus biorxiv \
  --mode discovery \
  --max-results 3 \
  --skip-details-rerank \
  --poll-timeout 75 \
  --save-payload "$RUN/payload_smoke.json" \
  --save-response "$RUN/results_smoke.json" \
  > "$RUN/summary_smoke.json" 2> "$RUN/time_smoke.txt"

If timeout is unavailable, still pass --poll-timeout and record that no outer wall-clock guard was available. Avoid /usr/bin/time; use shell time -p unless you have confirmed the path exists.

Known client pitfall: Cloudflare can reject Python urllib's default user agent with HTTP 403 and error code 1010. The helper sends a normal user agent. If raw urllib is unavoidable, send both X-API-Key and User-Agent.

Preferred Workflow

Step 1: Author Query JSON

Use compact concept groups:

{
  "anchor_entity": [
    ["primary_name"],
    ["alias_1"],
    ["alias_2"]
  ],
  "relation_or_property": [
    ["primary_name", "relation_term"],
    ["alias_1", "specific_relation_alias"]
  ],
  "disqualifying_terms": [
    ["term_to_exclude"]
  ]
}

Group logic (enforced consistently on both the FTS fast path and the parquet scan): terms inside one inner group are AND'd (all must co-occur); inner groups within a concept are OR'd; concepts are OR'd, with ranking favoring items that match more groups. A multi-word term is matched as an exact phrase. So list synonyms/aliases as separate single-term groups (each ["alias"] on its own line) — never pack them into one group, which would require them all to co-occur. Use a multi-term group only to require co-occurrence (e.g. ["entity", "relation"]).

For recent taxa or sparse terms, begin with anchor-only JSON:

{"anchor_entity": [["Mirusviricota"], ["mirusvirus"], ["mirusviruses"]]}

Step 2: Search With Hosted API

Fast first pass for emerging taxa:

RUN=tasks/polars-dovmed-runs/mirusviricota-hosts-$(date +%Y%m%d)
mkdir -p "$RUN"
printf '%s\n' "papers describing hosts of Mirusviricota" > "$RUN/prompt.txt"
# Write and inspect "$RUN/query.json" before running the search.

timeout 90s uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --queries-file "$RUN/query.json" \
  --corpus biorxiv \
  --mode discovery \
  --extract-matches none \
  --add-group-counts primary \
  --max-results 25 \
  --details-rerank-limit 12 \
  --poll-timeout 75 \
  --save-payload "$RUN/payload_biorxiv.json" \
  --save-response "$RUN/results_biorxiv.json" \
  > "$RUN/summary_biorxiv.json" 2> "$RUN/time_biorxiv.txt"

OpenPMC pass with parallel clean year bands:

timeout 120s uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --queries-file "$RUN/query.json" \
  --corpus pmc \
  --mode discovery \
  --sync \
  --year-bands clean_split \
  --year-band-workers 4 \
  --extract-matches none \
  --add-group-counts primary \
  --max-results 25 \
  --skip-details-rerank \
  --poll-timeout 120 \
  --save-payload "$RUN/payload_pmc.json" \
  --save-response "$RUN/results_pmc.json" \
  > "$RUN/summary_pmc.json" 2> "$RUN/time_pmc.txt"

For citation metadata enrichment after a shortlist, prefer bounded Crossref fallback over web search:

uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --details PMC6362216 PMC10132079 \
  --corpus pmc \
  --crossref-metadata \
  --crossref-limit 10 \
  --save-payload "$RUN/payload_details.json" \
  --save-response "$RUN/results_details.json"

If the prompt gives a year constraint, map it before searching:

<=2009 -> pre_2010
2010-2020 -> 2010_2020
2021-2023 -> 2021_2023
>=2024 -> 2024_plus

For a range crossing bands, pass an explicit comma list to --year-bands, for example --year-bands 2021_2023,2024_plus. Use a single --year-band only when the whole requested range is inside one band.

Fetch details directly when you already know identifiers:

uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --details PMC6912108 PMC8490762 \
  --corpus pmc \
  --save-payload "$RUN/payload_details.json" \
  --save-response "$RUN/results_details.json"

For bioRxiv details, pass DOI values with --corpus biorxiv.

Step 3: Search Locally With dovmed scan

Use local mode when hosted access is unavailable or explicitly unwanted.

uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --execution-mode local \
  --corpus pmc \
  --local-parquet-pattern "$DOVMED_PMC_PARQUET" \
  --queries-file "$RUN/query.json" \
  --save-payload "$RUN/payload_local.json" \
  --save-response "$RUN/results_local.json"

Local corpus aliases:

--corpus pmc: set DOVMED_PMC_PARQUET or pass --local-parquet-pattern.
--corpus biorxiv: set DOVMED_BIORXIV_PARQUET or pass --local-parquet-pattern.
--corpus both: only with one compatible explicit parquet pattern; otherwise run separate scans.

Search Semantics

Prefer structured JSON over ad hoc natural-language search strings.
Build searches around anchor concepts first.
List alternate names and spelling variants as separate single-term groups within a concept (groups are OR'd); keep a multi-word synonym as one phrase term. Multiple terms in one group are AND'd (all must co-occur) — now enforced on the FTS fast path too, so never pack synonyms into a single group.
Treat support concepts as refiners, not anchors.
For "X of Y" prompts, combine X and relation terms inside an OR-of-AND group only after an anchor-only discovery pass if recall is poor.
Down-rank papers matching only generic support terms or full-text-only background mentions.

Ranking priority:

exact anchor hit in title
exact anchor hit in abstract
anchor plus support co-occurrence in title or abstract
multiple distinct relevant group matches
full-text-only matches

Quick Reference

| Task | Action | |------|--------| | Run directory | tasks/polars-dovmed-runs/<date-topic>/ | | Preferred query | Authored query.json, inspected before search | | Preferred hosted path | helper scan_literature_advanced(mode="discovery"); async except FTS-backed OpenPMC clean bands | | Emerging taxon first pass | --corpus biorxiv, anchor-only query, bounded timeout | | OpenPMC pass | separate --corpus pmc --sync --year-bands clean_split --year-band-workers 4 --skip-details-rerank call | | Year-constrained OpenPMC | map requested years to one or more clean bands before searching | | Avoid by default | flat /api/search_literature, --corpus both, unbanded broad OpenPMC | | Details endpoint | --details ... --corpus pmc|biorxiv | | Missing DOI/year | --crossref-metadata or skills/crossref-lookup/scripts/lookup --title ... | | Local fallback | --execution-mode local --local-parquet-pattern ... | | Quick verification | uv run --no-project python skills/polars-dovmed/scripts/smoke_test.py --run-dir tasks/polars-dovmed-runs/smoke-test | | Timing | shell time -p, helper elapsed_ms, and timeout status |

Input Requirements

A literature search prompt or inspected structured query.json.
Hosted API key or local parquet files for the requested corpus.
Writable run directory outside the skill source tree.

Output

prompt.txt, query.json, payload JSON, raw response JSON, timing files, and optional curated summary.
Paper list with titles, identifiers, corpus, relevance notes, and timing.
Warnings for timeout, 502, database-not-found, missing metadata, or fallback use.

Quality Gates

[ ] API key handled as a secret and not persisted.
[ ] Run directory is outside skills/polars-dovmed/.
[ ] Query JSON authored and inspected before search.
[ ] Hosted API root or helper smoke checked before declaring outage.
[ ] Flat endpoint, --corpus both, and broad unbanded OpenPMC avoided unless explicitly justified.
[ ] OpenPMC broad searches use direct parallel clean year bands with --sync and --skip-details-rerank; year-constrained searches use only matching bands.
[ ] Discovery mode used before advanced refinement.
[ ] First 5-10 hits reviewed and noisy terms refined.
[ ] Long PMC scans have bounded poll and wall-clock timeout.
[ ] Timings and failure modes reported when speed is part of the request.
[ ] Citation metadata uses DB/details first, then bounded Crossref fallback; generic web search is not used for routine DOI repair.

Examples

Mirusviricota Host Search

Create tasks/polars-dovmed-runs/mirusviricota-hosts-YYYYMMDD/query.json with:

{"anchor_entity": [["Mirusviricota"], ["mirusvirus"], ["mirusviruses"]], "disqualifying_terms": [["MIRU-VNTR"], ["mycobacterium"]]}

Run the bioRxiv hosted command from Step 2.
Inspect results_biorxiv.json and any companion details response.
Run the direct parallel OpenPMC clean-band command when PMC coverage is needed.
Summarize direct host evidence separately from background mentions.

Local PMC Search

RUN=tasks/polars-dovmed-runs/klosneuvirinae-hosts
uv run --script skills/polars-dovmed/scripts/query_literature.py \
  --execution-mode local \
  --corpus pmc \
  --local-parquet-pattern "$DOVMED_PMC_PARQUET" \
  --queries-file "$RUN/query.json" \
  --save-payload "$RUN/payload_local.json" \
  --save-response "$RUN/results_local.json"

Troubleshooting

Issue: Flat /api/search_literature times out Solution: Do not use it for smoke tests. Use the root endpoint plus structured helper smoke.

Issue: --corpus both returns 502 Solution: Run separate --corpus biorxiv and --corpus pmc calls and merge results manually.

Issue: OpenPMC --year-bands clean_split returns Database not found Solution: Stop and report that the hosted deployment is not exposing the indexed OpenPMC bands. Do not retry as an unbanded full-corpus OpenPMC scan.

Issue: OpenPMC clean-band search times out Solution: Report the measured timeout and keep the bioRxiv result. Do not retry the same OpenPMC query with a longer unbanded scan; narrow by user-specified year band or refine the anchor query.

Issue: A query about "topic of X" retrieves generic topic papers Solution: Replace the first-pass query with exact entity aliases only. Use topic terms during manual triage or a focused second pass after confirming anchor hits.

Issue: DOI, year, or journal metadata is missing Solution: Fetch details by PMCID/DOI first. If still incomplete, use --crossref-metadata or the crossref-lookup skill. Avoid generic web search unless both corpus metadata and Crossref are unresolved or conflicting.

Issue: /usr/bin/time is missing Solution: Use shell time -p or record start/end timestamps; do not assume /usr/bin/time exists.

Issue: Artifacts were written under skills/polars-dovmed/runs/ Solution: Move future run artifacts to tasks/polars-dovmed-runs/ and do not commit generated run outputs.

Issue: Hosted API key is missing and local parquet files are missing Solution: State that polars-dovmed is not configured. Use another literature-search skill or ask the user to provide hosted access or local corpora.