shield_lock
JinDaGe
← Back to skills
extension
Category: Development & EngineeringAPI key requirement unconfirmed

local-kb-qa：本地 GPU 上的私有知识库问答 Skill

团队内部文档散落在群聊、Wiki、部署记录等多个系统里，问答时既难定位出处，又不愿把资料交给云端。本作品是一个跑在本地 GPU 上的私有知识库问答 Skill，基于 RAG 架构，检索与生成都在本机完成，文档不出域、断网可用、零调用成本。 技术方案：生成模型采用 Qwen2.5-7B-Instruct（4-bit NF4 量化，显存占用约 4.3 GB），embedding 采用 bge-small-zh-v1.5（512 维，CPU 推理），向量库使用 Chroma 本地持久化。支持 NVIDIA CUDA 与 Intel OpenVINO 双后端，通过环境变量一键切换。 应用场景：面向有内部文档的小团队，通过 localhost HTTP 服务对接 WorkBuddy / Qoder / TRAE 等 Agent，返回带文件引用的回答，审计有据可查。

personAuthor: yeatliaohubgithub
open_in_newRepositorydownloadDownload package

Model: Qwen/Qwen2.5-7B-Instruct (4-bit, runs on the local NVIDIA GPU by default)
Embedding: BAAI/bge-small-zh-v1.5 (local, Chinese-friendly)
Vector store: Chroma (local persistence)
SKILL_VERSION: v1.1.0

What this skill is for: point it at a folder of your own documents (Markdown, txt, pdf, docx), ask questions in natural language, and get answers grounded in those documents — all on your own PC. Nothing is uploaded; the retrieval and the generation both happen on-device. This is the "private knowledge base" pattern: the model knows only what you put in the docs/ folder.

Network usage: Setup downloads pip dependencies from pypi.org and the models (~5 GB, resumable) from modelscope.cn. Inference is fully offline — no network calls once setup is complete.

First time? Before using this skill, run these two scripts once in a terminal (use the venv created by setup.py for every later step):

python setup.py          # creates venv, installs CUDA torch + deps (~3 min)
python download_model.py # downloads the LLM + embedding model (~5 GB, resumable)

Both scripts are in the skill directory alongside this SKILL.md. Then drop your documents into the docs/ folder (created automatically), build the index, and ask.

安装到 Agent（怎么用起来）

本 Skill 面向支持 Skill 导入的 Agent（WorkBuddy / Qoder / TRAE / 通义 / 扣子 等）。把整个技能文件夹放进去即可，<SKILL_DIR>（即 SKILL.md 所在目录）会在运行时自动解析，文件夹放在哪都能用：

  • WorkBuddy / CodeBuddy：把本文件夹放到用户级 ~/.workbuddy/skills/local-kb-qa/ 或项目级 .workbuddy/skills/local-kb-qa/ 下，对话中直接调用。
  • Qoder / TRAE / 通义 / 扣子：通过「导入 Skill / 从仓库导入」把本文件夹加入，触发词见上方 description。
  • 魔搭 Skills 中心：将本仓库发布到 modelscope.cn/skills，打「AI PC」标签。

导入后首次使用前，在技能目录下跑一次环境准备（见上方「First time?」）。

WorkBuddy 集成（评分·工具使用 20%）

本 Skill 通过本地 HTTP 服务与 WorkBuddy 对接，模型与向量库全程在本地，WorkBuddy 只负责把问题转给 localhost 服务并回显结果：

  1. 把整个技能文件夹复制到 ~/.workbuddy/skills/local-kb-qa/。
  2. 首次使用前在技能目录跑 python setup.py 与 python download_model.py，把文档放进 docs/ 后跑 python build_index.py。
  3. 在技能目录后台启动服务：python local_qa.py --server（监听 127.0.0.1:8000）。
  4. 在 WorkBuddy 对话中提问，本 SKILL.md 指示 WorkBuddy 用 Bash 调用： curl "http://127.0.0.1:8000/api/qa?q=<你的问题>" 把返回的 JSON 中 answer 与 sources 呈现给用户。

数据不出机：请求只在本机 loopback 上往返，不经过任何云 API。

Directory layout (all auto-created under <SKILL_DIR>)

<SKILL_DIR>\
├── SKILL.md                 ← this file
├── setup.py / download_model.py / build_index.py / local_qa.py / serve.py
├── requirements.txt
├── .venv\                   ← venv with dependencies (created by setup.py)
├── state.json               ← written by setup.py
├── models\
│   ├── LLM\                 ← Qwen2.5-7B-Instruct (downloaded by download_model.py)
│   └── EMB\                 ← bge-small-zh-v1.5
├── chroma\                  ← Chroma persistence (the vector index)
└── docs\                    ← put your .md/.txt/.pdf/.docx here

<SKILL_DIR> 是包含本 SKILL.md 的目录，由 Agent 在运行时解析，因此技能无论放在何处都可移植。

⚠️ Agent instructions

  1. Windows / PowerShell only. Never use Linux commands (ls, rm, cat). Never use && or call.
  2. Every step reads state.json itself — do not pass paths between steps manually.
  3. Use VENV_PY from state.json for all python calls — never use system python for inference.
  4. Goal: answer the user's question using only the local documents, and show the source snippets the answer was based on.

Auto-recovery policy — try before asking user:

  • If STATE=MISSING or VENV_PY=BROKEN: automatically run setup.py (up to 3 attempts). Only ask user if all 3 fail.
  • If LLM_STATUS=MISSING or EMB_STATUS=MISSING: automatically run download_model.py (up to 3 attempts). Stop if a single attempt exceeds 20 minutes — download supports resume.
  • Always announce before each attempt: ⚙️ Auto-installing environment (attempt N/3)…

Pipeline — follow exactly in order, no skipping:

Step 0: parse question      → QUESTION (verbatim from user)
Step 1: verify environment  → VENV_PY, model + index status confirmed
         ↳ if STATE=MISSING or VENV_BROKEN: auto-run setup.py (3 attempts)
         ↳ if LLM_STATUS/EMB_STATUS=MISSING: auto-run download_model.py (3 attempts)
Step 2: ensure index        → Chroma index built from docs/ (skipped if already indexed)
Step 3: ask + answer        → [SUCCESS] + answer + source snippets (all local)

Step 0: parse question (LLM only — no tools)

Take the user's question verbatim. No expansion needed for Q&A — pass it through as QUESTION.

If the user has not yet put any documents in docs/, tell them once:

📂 Put your documents (.md/.txt/.pdf/.docx) into the docs/ folder, then ask again.

…and stop. Do not attempt Step 3 on an empty index.

Step 1: verify environment and models

🔍 Step 1/3: checking environment and models…

python -c "
import json, subprocess
from pathlib import Path

state = None
skill_dir = Path(r'{baseDir}')
cand = skill_dir / 'state.json'
if cand.exists():
    state = json.loads(cand.read_text(encoding='utf-8'))

if not state:
    print('STATE=MISSING')
    exit(1)

venv_py   = Path(state['VENV_PY'])
skill_dir = Path(state['SKILL_DIR'])
llm_dir   = skill_dir / 'models' / 'LLM'
emb_dir   = skill_dir / 'models' / 'EMB'

r = subprocess.run([str(venv_py), '--version'], capture_output=True, timeout=10)
if r.returncode != 0:
    print('VENV_PY=BROKEN')
    exit(1)

print(f'VENV_PY={venv_py}')
print(f'SKILL_DIR={skill_dir}')

llm_ready = (llm_dir / 'config.json').exists() and any(llm_dir.glob('*.safetensors'))
emb_ready = any(emb_dir.rglob('*.safetensors'))
print('LLM_STATUS=READY' if llm_ready else 'LLM_STATUS=MISSING')
print('EMB_STATUS=READY' if emb_ready else 'EMB_STATUS=MISSING')
if not llm_ready or not emb_ready:
    exit(1)
"

On success: record VENV_PY and SKILL_DIR from output, proceed to Step 2.

If STATE=MISSING or VENV_PY=BROKEN → auto-run setup.py

python -c "
from pathlib import Path
p = Path(r'{baseDir}') / 'setup.py'
print(f'SETUP_PY={p}') if p.exists() else print('SETUP_PY=NOT_FOUND')
"

Announce and run (up to 3 attempts):

⚙️ Environment not initialized — auto-installing (attempt 1/3)…

python "<SETUP_PY path>"

Re-run Step 1's check after each attempt. If all 3 fail, show manual fallback below.

If LLM_STATUS=MISSING or EMB_STATUS=MISSING → auto-run download_model.py

python -c "
from pathlib import Path
p = Path(r'{baseDir}') / 'download_model.py'
print(f'DOWNLOAD_PY={p}') if p.exists() else print('DOWNLOAD_PY=NOT_FOUND')
"

Announce to user and ask how to proceed:

📥 Model not found — download required (~5 GB)
   Estimated time:
   • 100 Mbps → ~7 min
   •  50 Mbps → ~14 min
   •  10 Mbps → ~70 min
   Download supports resume — safe to interrupt and retry.

   ✅ Start auto-download
   📂 I'll download manually — show me the link

Auto-download (up to 3 attempts, stop if a single attempt exceeds 20 minutes):

python "<DOWNLOAD_PY path>"

Re-run Step 1's check after each attempt.

Manual download fallback:

LLM (Qwen2.5-7B-Instruct) and embedding (bge-small-zh-v1.5) both come from ModelScope. download_model.py uses modelscope.snapshot_download, so the simplest path is to just run the script. If you must do it by hand, place the model files under:

<SKILL_DIR>\models\LLM\      ← Qwen2.5-7B-Instruct (PyTorch weights)
<SKILL_DIR>\models\EMB\      ← bge-small-zh-v1.5

Manual fallback (only if all 3 setup auto-attempts fail)

python -c "
from pathlib import Path
skill_dir = Path(r'{baseDir}')
for script in ['setup.py', 'download_model.py', 'build_index.py', 'local_qa.py', 'serve.py']:
    p = skill_dir / script
    if p.exists(): print(f'{script}={p}')
"

Show user:

⚠️ Auto-install failed. Please run manually in a terminal:

① Install environment:
   python "<full path to setup.py>"
   Takes ~3 min, fully automated.

② Download models (~5 GB):
   python "<full path to download_model.py>"
   Resumable — safe to interrupt and retry.

③ Build the index (after you add documents to docs/):
   python "<full path to build_index.py>"

Come back here when done.

Step 2: ensure the index is built

🗂️ Step 2/3: checking / building the local vector index…

Run the shipped indexer (it is idempotent — re-running only adds new or changed files):

& "<VENV_PY>" "<SKILL_DIR>\build_index.py"

Expected output contains INDEX=READY and the document count. If docs/ is empty, the script prints DOCS_EMPTY — go back to Step 0's "put documents in docs/" message.

| Output | Action | |--------|--------| | INDEX=READY | ✅ proceed to Step 3 | | DOCS_EMPTY | ⛔ tell the user to add documents, then stop | | any traceback | ⛔ re-run setup.py, then retry |

Step 3: ask and answer (fully local)

Two ways to answer. Prefer the local HTTP service so the answer is produced by the on-device model and easy for the Agent to consume.

Option A — local service (recommended for Agent integration)

Start the service once in the background (if not already running):

& "<VENV_PY>" "<SKILL_DIR>\local_qa.py" --server

Then answer the question by calling it:

curl "http://127.0.0.1:8000/api/qa?q=<URL-encoded QUESTION>"

The response is JSON: {"answer": "...", "sources": ["file1", ...]}. Send the answer to the conversation via message tool, including the sources list.

Option B — one-shot CLI

& "<VENV_PY>" "<SKILL_DIR>\local_qa.py" --question "QUESTION"

Pass: stdout contains [SUCCESS] followed by the answer and a SOURCES: block listing the document snippets used.

Send the answer to the conversation via message tool:

action: "send"  message: "<the answer text, including the SOURCES block>"

Final announcement:

✅ Answered from your local knowledge base (offline).
🔎 Retrieved N chunks from: <doc names>
⚙️ device: {cuda:N / openvino-gpu / cpu} | LLM: Qwen2.5-7B-Instruct

Parameters (local_qa.py)

| Param | Default | Notes | |-------|---------|-------| | --question | required (unless --server) | Natural-language question, Chinese or English | | --top-k | 4 | Number of retrieved chunks fed to the LLM | | --server | off | Start the localhost HTTP service on :8000 | | --port | 8000 | Port for --server | | KBQA_BACKEND (env) | auto | cuda (NVIDIA GPU) / openvino (Intel iGPU) / cpu |

Troubleshooting

| Error | Cause | Fix | |-------|-------|-----| | STATE=MISSING | setup.py never run | Run python setup.py from the skill directory | | VENV_PY=BROKEN | venv corrupted | Re-run setup.py — rebuilds venv automatically | | LLM_STATUS=MISSING | download never run or interrupted | Run python download_model.py — resumes automatically | | EMB_STATUS=MISSING | embedding model not downloaded | Run python download_model.py | | DOCS_EMPTY | no documents in docs/ | Add .md/.txt/.pdf/.docx files, then re-run | | INDEX=READY but weak answers | index stale after editing docs | Re-run build_index.py | | OutOfMemoryError on GPU | VRAM too small for 7B (4-bit ~4.3 GB) | Use Qwen2.5-3B-Instruct for download_model.py, or set KBQA_BACKEND=cpu | | CUDA out of memory | other apps using the GPU | Close GPU-heavy apps, or reduce --top-k | | Very slow first load | 4-bit model + CUDA kernels initialising | Normal on first run; cached afterwards | | Garbled Chinese output | stdout encoding | The scripts force UTF-8; run from PowerShell, not cmd |