Local Knowledge Base Assistant

A locally-running intelligent question-answering system based on Intel OpenVINO optimization, supporting PDF/TXT/Markdown document Q&A with complete local execution for privacy protection.

1. Overview

本项目是一个基于 Intel OpenVINO 优化的本地知识库助手，完全在本地运行，支持 PDF、TXT、Markdown 文档的智能问答。

Core Features

• Local Document Retrieval — 自动索引本地文档，支持智能检索
• Local Model Inference — 所有计算在本地 CPU/GPU 完成，无需联网
• RAG-Augmented Q&A — 基于检索到的文档片段生成精准回答
• OpenVINO Optimization — Intel OpenVINO 推理引擎，性能提升 15-20%
• RESTful API — 标准 HTTP 接口，便于系统集成
• OpenAI-Compatible API — /v1/chat/completions 接口与 OpenAI 格式一致

2. Quick Start

Prerequisites

• Python 3.10+
• Intel CPU（推荐 Core i5 / Xeon 或更新）
• 8 GB+ RAM 最低，推荐 16 GB+
• Windows / Linux / macOS

Option 1: PowerShell Script (Recommended)

# Step 1: Navigate to the project directory
cd local-knowledge-base-assistant

# Step 2: Start the service (Mainland China mirror)
.\scripts\run.ps1 -China

# Or check status
.\scripts\run.ps1 -Status

# Or stop service
.\scripts\run.ps1 -Stop

Option 2: Manual Startup

# Create virtual environment
python -m venv .venv

# Activate virtual environment
.\.venv\Scripts\Activate.ps1

# Install dependencies
pip install -r requirements.txt

# Start the service
python .\scripts\server.py --port 18765

Add Your Documents

1. Place PDF / TXT / MD files in ./docs/ directory
2. Restart service or call /api/reload-docs endpoint
3. Set use_rag: true in queries to enable document Q&A

3. API Endpoints

| Endpoint | Method | Description | |----------|--------|-------------| | /api/health | GET | Health check (returns model loading status) | | /v1/chat/completions | POST | OpenAI-compatible chat inference | | /api/query | POST | Knowledge base query (supports RAG) | | /api/reload-docs | POST | Reload documents and rebuild index | | /api/shutdown | POST | Gracefully shutdown the inference service |

4. Usage Examples

Example 1: General Q&A (OpenAI-Compatible API)

Scenario: When you want to chat with the AI assistant without accessing any local documents. Works just like ChatGPT.

Steps:

1. Set the local service address
2. Send a chat request
3. Get the AI assistant's reply

import openai

# Step 1: Set up client pointing to local service
# No real API key needed - use any value
openai.api_key = "dummy"
openai.base_url = "http://127.0.0.1:18765/v1"

# Step 2: Send chat request
response = openai.chat.completions.create(
    model="qwen2.5-1.5b",
    messages=[
        {"role": "user", "content": "你好，请介绍一下你自己"}
    ],
    temperature=0.7,
    max_tokens=256
)

# Step 3: Get and print the reply
print(response.choices[0].message.content)

Expected Output:

🤖 Assistant: 你好！我是本地知识库助手，由通义千问模型驱动，使用 Intel OpenVINO 加速推理...

---

Example 2: Document Q&A (Knowledge Base API)

Scenario: When you want to ask questions based on local documents. If you have a PDF report or TXT notes, the system retrieves relevant document chunks first, then generates an accurate answer based on those chunks.

Prerequisite: Place PDF or TXT documents in the ./docs/ directory first.

Steps:

1. Place your documents in the ./docs/ folder
2. Call the document Q&A endpoint
3. View the answer and related document chunks

import requests

# Step 1: Send document Q&A request
response = requests.post(
    "http://127.0.0.1:18765/api/query",
    json={
        "question": "根据我的笔记，OpenVINO有什么优势？",
        "use_rag": True
    }
)

# Step 2: Extract and parse the answer
result = response.json()

print(f"📄 AI Answer: {result['answer']}")
print(f"🔍 RAG Used: {'Yes' if result['rag_used'] else 'No'}")
print(f"📊 Related Documents Found: {result['rag_docs_count']} chunks")
print(f"⏱️ Generation Time: {result['generation_time']:.2f} seconds")

Expected Output:

📄 AI Answer: According to the document content, OpenVINO has the following advantages:
1. Performance optimization: Inference speed increased by 15-20%
2. Multi-hardware support: Automatic CPU/GPU/NPU acceleration
3. Memory optimization: Supports INT4/INT8 quantization
🔍 RAG Used: Yes
📊 Related Documents Found: 3 chunks
⏱️ Generation Time: 5.46 seconds

---

Example 3: Health Check

Scenario: After starting the service, verify that the service is running properly, the model is loaded successfully, and documents are indexed correctly. Useful for quick diagnostics.

Steps:

1. Send a health check request
2. View status indicators
3. Determine if the service is ready

import requests

# Send health check request
response = requests.get("http://127.0.0.1:18765/api/health")

# Parse status information
health = response.json()

print(f"🔧 Service Status: {health['status']}")
print(f"🤖 Model Loaded: {'✅ Loaded' if health['model_loaded'] else '❌ Not Loaded'}")
print(f"📚 Vector Store Ready: {'✅ Ready' if health['vector_store_ready'] else '⚠️ Not Ready'}")
print(f"📁 Documents Indexed: {health['docs_count']} files")
print(f"🔌 OpenVINO Available: {'✅ Yes' if health['openvino_available'] else '❌ No'}")

Expected Output:

🔧 Service Status: ok
🤖 Model Loaded: ✅ Loaded
📚 Vector Store Ready: ✅ Ready
📁 Documents Indexed: 3 files
🔌 OpenVINO Available: ✅ Yes

Status Code Reference:

• ok: Service fully operational
• initializing: Service is still starting up
• error: Service encountered an error

5. Architecture

The system follows a 3-layer architecture:

1. Retrieval Layer (FAISS + sentence-transformers): Indexes document chunks and retrieves top-k most relevant passages
2. Reasoning Layer (Qwen2.5 + OpenVINO): Generates answers based on retrieved context + the user's question
3. Service Layer (FastAPI): Exposes HTTP endpoints for health checks, document reload, and Q&A

┌─────────────────────────────────────────────────────────────┐
│                     Client/Server Architecture               │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────────┐       HTTP :18765       ┌──────────────┐│
│  │   Client        │ ─────────────────────→ │   Server     ││
│  │   (Optional)    │ ←───────────────────── │  (FastAPI)   ││
│  └─────────────────┘                           └──────┬───────┘│
│                                                         │         │
│                                                         ↓         │
│                     ┌─────────────────────────┐                   │
│                     │  OpenVINO Inference     │                   │
│                     │  (Qwen2.5-1.5B-Instruct)│                   │
│                     └─────────────────────────┘                   │
│                                                         │         │
│  ┌────────────────────────────────────────────────────────┐     │
│  │  Features:                                              │     │
│  │  • OpenAI-compatible API /v1/chat/completions          │     │
│  │  • Local RAG with FAISS vector database               │     │
│  │  • Privacy-first - everything runs locally            │     │
│  └────────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────┘

Workflow

User Question
    ↓
Document Retrieval (if RAG enabled)
    ↓
LLM Inference (OpenVINO-accelerated)
    ↓
Answer Generation
    ↓
Return Result

Detailed Steps:

1. User inputs a question
2. System checks for relevant documents
3. If RAG is enabled, vector search retrieves relevant context
4. LLM (via OpenVINO) performs inference on the combined prompt
5. Natural language answer is returned

6. Supported Document Formats

| Format | Extension | Notes | |--------|-----------|-------| | PDF | .pdf | Text-searchable PDFs work best; scanned/OCR PDFs may need preprocessing | | Plain Text | .txt | Any encoding (UTF-8 recommended) | | Markdown | .md | Full structure preserved |

7. Tips for Good Results

• Better documents → better answers — use well-structured, clearly-written source material
• Adjust top_k — more passages give broader context; fewer passages give more focus
• Lower temperature (0.2-0.5) for factual Q&A; higher for creative tasks
• Prefer Markdown over PDF when possible — parsing is faster and more reliable
• Reload documents after adding or modifying files in docs/

8. Troubleshooting

• Model won't load: Verify models/ directory exists and contains the converted model files
• Out of memory: Use a smaller model or enable INT8 quantization
• Poor retrieval quality: Try adding more specific documents, or increase top_k
• Connection errors: Check that port 18765 is not blocked by a firewall

9. Security Principles

• ✅ Process only user-specified local documents
• ✅ All inference runs locally — no cloud uploads
• ✅ Do not save user conversation history (unless explicitly requested)
• ✅ Do not modify original user documents

10. Technology Stack

• Inference Engine: OpenVINO 2026.1.0
• Model: Qwen2.5-1.5B-Instruct
• Framework: LangChain
• Vector Store: FAISS
• Web Framework: FastAPI + Uvicorn

11. Dependencies

- openvino>=2026.1.0
- optimum-intel
- fastapi
- uvicorn
- langchain
- langchain-community
- transformers
- faiss-cpu
- sentence-transformers

12. Notes

1. First Run: Embedding model download required (~500 MB); use a mirror for faster download in China
2. Document Directory: Place PDF/TXT/MD documents in ./docs/ directory
3. API Usage: Supports both OpenAI-compatible API and custom knowledge base API
4. Service Addresses (start service first):
   • API base: http://127.0.0.1:18765
   • Swagger docs: http://127.0.0.1:18765/docs ⭐ Interactive test interface
5. Start Service: Run .\scripts\run.ps1 -China or python .\scripts\server.py

Quick Test Flow

1. Start service: .\scripts\run.ps1 -China
   ↓
2. Open browser: Visit http://127.0.0.1:18765/docs
   ↓
3. Test online: Click any API endpoint → "Try it out" → Fill params → "Execute"
   ↓
4. View results: Check API response at bottom of page

13. Project Structure

local-knowledge-base-assistant/
├── SKILL.md              # Skill metadata + instructions
├── info.json             # Runtime configuration
├── meta.json             # App store display info
├── requirements.txt      # Python dependencies
├── README.md             # Full documentation
│
├── scripts/
│   ├── server.py         # OpenVINO inference service (FastAPI)
│   ├── convert_model.py  # Model download + OpenVINO conversion
│   └── run.ps1           # Deployment entry (PowerShell)
│
├── models/               # Model files directory
│   └── qwen2.5-1.5b-openvino/
│       ├── openvino_model.xml
│       ├── openvino_model.bin
│       └── tokenizer.json
│
├── docs/                 # User documents directory
│   ├── example.pdf
│   └── notes.txt
│
└── logs/                 # Logs directory

14. License

Licensed under Apache License 2.0. Built-in models (Qwen2.5 / sentence-transformers) follow their respective open-source licenses.