Email Concierge — Local AI Email Assistant

A privacy-first email agent that reads, understands, and drafts replies to your emails. All inference runs locally via Ollama — no data leaves your machine.

Architecture

IMAP Mailbox ──→ read_mailbox ──→ classify_summarize (Ollama)
                                        │
iCal Files  ──→ check_calendar          │
                                        ▼
Microphone  ──→ start_voice_memo ──→ transcribe_memo (distil-whisper / Ollama)
                                        │
                                        ▼
                                 draft_reply (Ollama, 3 styles)

Models: Ollama (qwen3:4b default) for classification/drafting; distil-whisper-large-v3-int8-ov (optional) for transcription.

Phase 1: Project Setup

Create project directory <PROJECT_DIR> (user-specified, or default ./email-concierge).

1.1 Create `email_concierge.py`

The main script implementing all 6 tools as Python functions + CLI interface.

Key implementation details:

IMAP access: Use imaplib (stdlib) with email module for parsing. Supports Gmail, Outlook, Yahoo, etc. via app passwords.
Ollama API: Primary call via POST /api/generate (non-streaming), CLI fallback via ollama run. Default model: qwen3:4b.
Classification categories: todo, meeting, notification, promotion. Prompt returns JSON {"category": "...", "summary": "..."}.
Calendar parsing: icalendar lib reads .ics files from a configured directory. Checks for time overlaps with meeting emails.
Voice recording: sounddevice captures 16kHz mono WAV. Ctrl+C to stop early.
Transcription: Tries distil-whisper (OpenVINO) if whisper_model_dir configured, else Ollama whisper, else delegates to transcribe-translate skill.
Draft reply: Generates 3 styles — formal, friendly, concise. Prompt includes original email + calendar context + user voice/text input.
Config: Stored as email_concierge_config.json in script directory. Created via --setup wizard.

1.2 Create `requirements.txt`

imapclient>=2.3.0
icalendar>=5.0.0
sounddevice>=0.4.6
numpy>=1.24.0
librosa>=0.10.0
soundfile>=0.12.0
openvino>=2025.3.0
optimum-intel>=1.26.1,<1.27
transformers>=4.57.0
tokenizers>=0.22.0
modelscope
torch

1.3 Create `setup.sh`

Platform-adaptive script: detect OS → create venv → install deps → check Ollama → guide config.

Phase 2: Installation

2.1 Platform detection

[[ "$(uname)" == "Darwin" ]] && PLATFORM="macos"
[[ "$(uname)" == "Linux" ]] && PLATFORM="linux"
uname -s | grep -q MINGW && PLATFORM="windows"

2.2 Create venv & install

# macOS/Linux
python3 -m venv <PROJECT_DIR>/venv
<PROJECT_DIR>/venv/bin/pip install -r <PROJECT_DIR>/requirements.txt

# Windows
python -m venv <PROJECT_DIR>\venv
<PROJECT_DIR>\venv\Scripts\pip.exe install -r <PROJECT_DIR>\requirements.txt

2.3 Install & configure Ollama

# Install: https://ollama.com
ollama pull qwen3:4b       # Default for classification + drafting
ollama pull whisper         # Optional, for voice transcription

2.4 Run setup wizard

# macOS/Linux
<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py --setup

# Windows
<PROJECT_DIR>\venv\Scripts\python.exe <PROJECT_DIR>\email_concierge.py --setup

The wizard prompts for:

IMAP: host, port, email, app password, folder
Ollama: model names for classify and draft
Calendar: directory containing .ics files
Whisper (optional): path to distil-whisper model dir

IMAP app passwords:

Gmail: Google Account → Security → App passwords
Outlook: Microsoft → Advanced → App passwords
Yahoo: Account → Security → App passwords

Phase 3: Tools Reference

read_mailbox

Fetches unread emails via IMAP.

<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py --inbox

Returns: list of {from, subject, date, body} for each unread email.

classify_summarize

Classifies each email into todo|meeting|notification|promotion and generates a one-line summary via Ollama. Run automatically with --inbox.

check_calendar

Parses .ics files and finds events overlapping a target time.

<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py --check-cal 2026-06-14

Auto-triggered when a "meeting" category email is processed.

start_voice_memo

Records audio from the default microphone.

<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py --record 30

Saves 16kHz mono WAV. Ctrl+C stops early.

transcribe_memo

Transcribes audio to text. Priority: distil-whisper (OpenVINO) → Ollama → transcribe-translate skill.

<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py --transcribe voice_memo.wav

draft_reply

Generates 3-style replies (formal/friendly/concise) using Ollama. Combines original email + calendar context + user input.

Phase 4: Workflow Examples

Full interactive pipeline

<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py --inbox --interactive

Fetches unread emails
Classifies & summarizes each
User picks an email (by number)
If meeting: auto-checks calendar for conflicts
Optionally records voice input (--voice)
Generates 3 draft replies

With voice input

<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py \
    --inbox --interactive --voice --voice-duration 20

With text input

<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py \
    --inbox --email-index 1 --user-text "Accept the meeting, suggest 2pm instead"

Save results to file

<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py \
    --inbox --interactive --output results.json

Standalone tools

# Just check calendar
<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py --check-cal today

# Just record & transcribe
<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py --record 15
<PROJECT_DIR>/venv/bin/python <PROJECT_DIR>/email_concierge.py --transcribe voice_memo_*.wav

Phase 5: Transcribe-Translate Integration

If the transcribe-translate skill is installed in the same workspace, transcribe_memo automatically delegates to it when no local whisper model is configured.

To explicitly use distil-whisper from transcribe-translate:

Set whisper_model_dir in config to point to <TT_PROJECT>/models/distil-whisper-large-v3-int8-ov
Or install the OpenVINO whisper model directly

Troubleshooting

| Issue | Fix | |---|---| | Ollama not found | Install from https://ollama.com, then ollama pull qwen3:4b | | IMAP auth failed | Use app password, not account password (Gmail/Outlook/Yahoo) | | sounddevice import error | macOS: brew install portaudio; Linux: apt install libportaudio2 | | No calendar events found | Export .ics files from Calendar app to the configured calendar_dir | | Ollama timeout | Model may be loading first time; retry. Or use smaller model (qwen3:1.7b) | | icalendar not found | pip install icalendar inside venv | | Transcription empty | Check audio file exists and is WAV/FLAC; verify whisper_model_dir in config | | Windows venv activate fails | Use venv\Scripts\python.exe directly |