X/Twitter CDP Tweet Scraper v2

Extracts authentication from a Chromium browser via CDP, then calls the Twitter GraphQL API directly with httpx for high-speed bulk tweet collection. Outputs JSON + Markdown.

Quick Start

# By date range (recommended)
python3 <skill-path>/scripts/cdp_tweet_fetcher.py <username> --since 2026-02 [--until 2026-02-28] [--output-dir DIR]

# By year (shorthand)
python3 <skill-path>/scripts/cdp_tweet_fetcher.py <username> --year 2026 [--output-dir DIR]

# No date specified -> defaults to current year to date
python3 <skill-path>/scripts/cdp_tweet_fetcher.py <username> [--output-dir DIR]

Arguments:

• username (required): Twitter username (without @)
• --since: Start date (inclusive). Accepts YYYY-MM-DD, YYYY-MM, YYYY
• --until: End date (inclusive), defaults to today
• --year: Target year (shorthand for --since YYYY-01-01)
• --output-dir: Output directory, defaults to current working directory
• --page-delay: Seconds between API pages, default 1.0
• --max-pages: Maximum pages to fetch, default 200
• --cdp-port: CDP debugging port, default 9222

Output files:

• {username}_tweets_{YYYYMMDD}_{YYYYMMDD}.json — Full structured data
• {username}_tweets_{YYYYMMDD}_{YYYYMMDD}.md — Human-readable Markdown report

Prerequisites

1. Chromium-based browser (Chrome / Edge / Brave / Arc / Chromium) installed and logged in to Twitter/X
2. Python dependencies: pip install playwright httpx && playwright install chromium
3. Browser must be launched with CDP enabled — the script auto-detects your OS and shows the correct launch command

Launch browser with CDP (pick your browser):

macOS:

# Chrome
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222
# Edge
/Applications/Microsoft\ Edge.app/Contents/MacOS/Microsoft\ Edge --remote-debugging-port=9222
# Brave
/Applications/Brave\ Browser.app/Contents/MacOS/Brave\ Browser --remote-debugging-port=9222
# Arc
/Applications/Arc.app/Contents/MacOS/Arc --remote-debugging-port=9222

Linux:

google-chrome --remote-debugging-port=9222
# or: chromium-browser / microsoft-edge / brave-browser

Windows (PowerShell):

& "C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222
# or: msedge.exe / brave.exe in their respective paths

Architecture — Why It's Fast

v2 minimizes browser interaction: authentication is extracted once (~5 s), then all API calls go through httpx directly.

| Change | Effect | |--------|--------| | Browser only used for cookie + query ID extraction (one-time) | Init drops from ~18 s to ~5 s | | httpx direct HTTP requests (no CDP round-trips) | Each API call 3-5x faster | | Single endpoint: UserTweetsAndReplies (superset) | Half the pagination | | count=40 (was 20) | Half the pagination again | | page-delay 1.0 s (was 2.5 s) | 60% less wait per page |

Execution Flow

1. Connect to browser: CDP connection, extract cookies and CSRF token
2. Discover API: Parse JS bundles for GraphQL query IDs (no page navigation)
3. HTTP client: Create httpx client with extracted auth
4. Resolve user ID: Via UserByScreenName API
5. Bulk fetch: UserTweetsAndReplies endpoint, 40 tweets/page, direct HTTP
6. Output: JSON + Markdown files

Output Schema

Each tweet contains:

{
  "tweet_id": "123456789",
  "text": "Full tweet text...",
  "datetime": "2026-01-15T10:30:00+00:00",
  "url": "https://x.com/user/status/123456789",
  "author": "username",
  "is_reply": false,
  "reply_to": null,
  "reply_to_tweet_id": null,
  "is_retweet": false,
  "retweet_of": null,
  "is_quote": false,
  "quoted_tweet_url": null,
  "likes": 100,
  "retweets": 20,
  "replies": 5,
  "bookmarks": 30,
  "views": 10000,
  "media": ["https://pbs.twimg.com/..."],
  "links": ["https://example.com/..."]
}

Fetching Twitter Articles (Long-Form Posts)

Links in tweets matching x.com/i/article/{article_id} are Twitter Articles (long-form posts). Article content is NOT in the tweet API response and requires additional steps to extract.

Usage

python3 <skill-path>/scripts/fetch_articles.py <tweets_json> [--output-dir DIR] [--cdp-port 9222]

• tweets_json: Path to a JSON file previously output by cdp_tweet_fetcher.py
• --output-dir: Directory for article Markdown files (default: ./articles)
• --cdp-port: CDP debug port (default: 9222)

Technical Details

1. API endpoint: TweetResultByRestId (GET), queried with the article's associated tweet_id
2. Critical parameter: fieldToggles: {"withArticleRichContentState": true, "withArticlePlainText": false}
3. Data location: data.tweetResult.result.article.article_results.result.content_state
4. Content format: Draft.js — blocks (paragraphs / headings / lists / blockquotes / code blocks / atomic) + entityMap (links / media references)
5. Type pitfall: entityMap is sometimes a dict (keyed by string index) and sometimes a list (indexed by position) — must handle both

Browser Request Parameters (Verified 2026-02)

{
  "variables": {
    "tweetId": "<tweet_id>",
    "includePromotedContent": true,
    "withBirdwatchNotes": true,
    "withVoice": true,
    "withCommunity": true
  },
  "fieldToggles": {
    "withArticleRichContentState": true,
    "withArticlePlainText": false
  }
}

Lessons Learned

| Date | Lesson | Action | |------|--------|--------| | 2026-02-27 | Twitter timeline is reverse-chronological; if all tweets on a page are before the target date, subsequent pages are even older | Added early-stop condition to script | | 2026-02-27 | DOM validation wastes time when API phase yields 0 tweets in target range | Skip DOM validation directly | | 2026-02-27 | DOM scrolling has no date filter, collects many irrelevant IDs | Snowflake ID date filtering | | 2026-02-27 | v1 architecture bottleneck: page.evaluate relay is slow, dual endpoints redundant, count=20 conservative, DOM validation heavy | v2 rewrite: httpx direct, single endpoint, count=40, DOM disabled by default | | 2026-02-28 | Twitter Article (long-form) content is NOT in the tweet text; requires separate extraction | Added fetch_articles.py script | | 2026-02-28 | TweetResultByRestId does not return article body by default; requires fieldToggles: {"withArticleRichContentState": true} | Critical parameter documented | | 2026-02-28 | Article content is in Draft.js format (content_state.blocks + entityMap); entityMap can be either a list or a dict | Script handles both types | | 2026-02-28 | Playwright CDP dialog event dismiss can throw ProtocolError: No dialog is showing and kill the Node process | Must wrap in try/except | | 2026-02-28 | When facing unknown API behavior, trace the browser's actual request parameters first, then write scraping code | Methodology: observe before guessing |