Consolidate Transcripts

Why? LLMs have context limits. This skill merges multiple transcripts into a single file with accurate token counting, so you can feed an entire channel's content to Claude or GPT without exceeding limits.

Quick Start

python scripts/consolidate_transcripts.py <channel_name>

Output: ~/Documents/YTScriber/<channel_name>/<channel_name>-consolidated.md

[!NOTE] This feature is currently a standalone script. A ytscriber consolidate CLI command is planned for a future release.

---

Workflow

1. Identify the Channel

List available channels:

ls ~/Documents/YTScriber/

2. Choose Token Limit

| Use Case | Recommended Limit | Flag | |----------|-------------------|------| | Claude (200K context) | 150000 | --limit 150000 | | GPT-4 Turbo (128K) | 100000 | --limit 100000 | | Full archive (Claude Pro) | 800000 | (default) | | Quick sample | 50000 | --limit 50000 |

[!TIP] The default 800K limit leaves ~200K tokens for prompts and responses when using Claude's 1M context.

3. Run Consolidation

python scripts/consolidate_transcripts.py <channel_name> [--limit TOKENS] [--verbose]

Examples:

# Default (800K tokens)
python scripts/consolidate_transcripts.py library-of-minds

# Custom limit for GPT-4
python scripts/consolidate_transcripts.py aws-reinvent-2025 --limit 100000

# Verbose output showing all included files
python scripts/consolidate_transcripts.py dwarkesh-patel --verbose

4. Verify Output

Check the consolidated file was created:

ls -la ~/Documents/YTScriber/<channel_name>/*-consolidated.md

---

Parameters

| Option | Description | Default | |--------|-------------|---------| | channel_name | Folder name in data directory | Required | | --limit, -l | Maximum tokens to include | 800000 | | --verbose, -v | Show detailed file list | False |

---

Output Format

The consolidated file includes:

1. Header — Generation metadata, total transcripts, token/word counts
2. Table of Contents — Dates, titles, tokens, words per transcript
3. Transcripts — Full text with title, date, author, source URL

---

Troubleshooting

| Problem | Cause | Solution | |---------|-------|----------| | ModuleNotFoundError: tiktoken | tiktoken not installed | pip install tiktoken | | No transcripts found | Empty transcripts folder | Run ytscriber download first | | FileNotFoundError | Channel doesn't exist | Check ls ~/Documents/YTScriber/ for valid names | | Output file is small | Few transcripts available | Use --verbose to see what was included | | Token count seems wrong | Old tiktoken version | pip install --upgrade tiktoken |

---

Common Mistakes

1. Wrong channel name — Use the folder name exactly as shown in ls ~/Documents/YTScriber/, not the YouTube channel name.
2. Forgetting to download transcripts first — Consolidation requires transcripts to exist. Run ytscriber download first.
3. Using too high a limit — If you exceed your LLM's context, you'll get truncation errors. Use the limit guide above.
4. Expecting real-time updates — Re-run consolidation after downloading new transcripts.

---

Reference

• Transcripts sorted newest first (descending by date)
• Files without dates in filename are placed last
• Token counting uses cl100k_base encoding (GPT-4/Claude compatible)
• Consolidated files are gitignored (not committed)
• Re-running overwrites the previous consolidated file