dsh-douyin-collector

Douyin account video collector for DeepSeek Harness, rebuilt after the 2026-08 contamination incident with provenance and adversarial verification baked in.

First-principles rules (why the design is what it is)

1. A data stream is not data. Douyin pages emit several streams (works, likes/favorites, recommendations, ads). The capture stage accepts only the post stream (/aweme/post/) with an exact author sec_uid match.
2. Provenance travels with every item. Each video stores its source API URL, author id, author name, and capture time — forever auditable.
3. Identity is the sec_uid, nothing else. Nicknames can change or collide; titles can be copied. Filtering and verification both compare author ids.
4. Capture and verification are separate stages. dyc_audit re-checks every delivered video through a second channel (the detail page author card) and writes per-video verdicts.
5. Pilot first, checkpointed, resumable. count: 5 proves the pipeline; re-runs skip completed accounts.
6. No string-interpolation of logic. Page-injected scripts are static strings; runtime config travels as a JSON blob appended to the patch.

Operational rules (learned the hard way, 2026-08)

7. Detached work must break away from the session job. Any long-running process (worker, transcribe) is launched via a PowerShell Start-Process launcher — Node spawn(detached) children get reaped at turn boundaries.
8. Launcher .ps1 files must be UTF-8 with BOM, and arguments must be single-quoted PowerShell strings containing embedded double quotes, or paths with spaces/Chinese break silently (the "can't find main" error).
9. A process that is alive is not stuck. Liveness is checked before staleness; a missing progress file only counts as dead after a launch grace period, because long videos legitimately produce no output for many minutes.
10. Create directories before writing progress files. The first item of a batch must be visible on the dashboard immediately, not after it finishes.
11. Progress must be visible and honest. Dashboards auto-refresh, and the yellow "stale" warning threshold scales with the average item duration, so long videos don't trigger false alarms.
12. Diagnosability first. Every launched process redirects stdout/stderr to log files; never launch blind.

Requirements

• Windows Chrome launched with --remote-debugging-port=9222 and a persistent user profile that is logged in to douyin.com (login persists across runs).
• Node >= 22 (native WebSocket; zero npm runtime dependencies).
• Python with faster-whisper (transcription) and openpyxl (Excel export).

Quick start (standalone, no DSH required)

1. Windows: run powershell -ExecutionPolicy Bypass -File start-chrome.ps1 (macOS: see the manual Chrome command in SKILL.md). Log in to douyin.com once in that Chrome window — the login persists in chrome-profile/.
2. pip install -r requirements.txt (transcription deps, one-time).
3. Put the target account's /user/ id into accounts.json, then: node run.js (try --count 5 first). Results land in run/output/ as an interactive HTML report plus an Excel file with transcripts.

Usage (DeepSeek Harness agent tools)

dyc_check_env → dyc_run → dyc_audit → dyc_transcribe → dyc_export. See SKILL.md for the run directory layout, accounts.json format, and the completion gate. cordis.patch.yml is only used to mount the plugin inside DSH; standalone users can ignore it.