Private API Reversal — Methodology

This skill is the generic methodology. No assumptions about any specific product. For Google-specific batchexecute protocol, see google-batchexecute-rpc skill (if available).

---

What This Is (And Is Not)

What it is: Every web app has a frontend that talks to a backend over HTTP. That traffic is visible in your browser. This skill is about observing that traffic, understanding its structure, and writing code that reproduces it — replacing the human clicking in the browser with your programmatic client.

What it is not:

• It is not exploiting security vulnerabilities
• It is not bypassing encryption
• It is not accessing data you don't have permission to access
• It does not work on apps using client-side request signing with secrets not visible in the browser

Honest limits stated up front:

• Some apps use Cloudflare Bot Management, TLS fingerprinting, or behavioural analysis that detects non-browser clients. This methodology does not defeat those.
• Session cookies and tokens expire. Your client needs a refresh strategy.
• Apps change their internal APIs without notice. Private clients break silently.
• Terms of Service: check before building. Many products explicitly forbid automated access.

---

Phase 1 — Reconnaissance (What to capture in DevTools)

Setup

1. Open Chrome (not Firefox — Chrome DevTools has better request inspection)
2. Open DevTools: Cmd+Option+I / Ctrl+Shift+I
3. Go to Network tab
4. Check Preserve log (top-left checkbox) — prevents log clearing on navigation
5. Filter: start with Fetch/XHR — ignores static assets, shows only API calls

What to do

Perform each action you want to automate in the UI, one at a time, watching which network requests fire. For each action you care about:

1. Click the request in the Network tab
2. Capture from Headers tab:
   • Full request URL
   • HTTP method (GET/POST/PUT etc.)
   • All request headers — especially Cookie, Authorization, Content-Type, X-* custom headers
3. Capture from Payload tab (POST requests):
   • Request body format: JSON? Form-encoded? Multipart? Protobuf?
   • Exact field names and value shapes
4. Capture from Response tab:
   • Response format: JSON? Streamed chunks? XML? Binary?
   • Structure of the data you need
5. Right-click the request → Copy → Copy as cURL — gives you a complete runnable command you can paste into a terminal to verify the request works outside the browser

What to look for specifically

The endpoint URL pattern:

https://app.example.com/_/AppName/data/batchexecute   ← Google RPC pattern
https://api.example.com/v1/resource                    ← REST pattern
https://app.example.com/graphql                        ← GraphQL
https://app.example.com/api/chat/completions           ← OpenAI-style streaming

Auth signal locations — look in ALL of these: | Location | What you might find | |---|---| | Cookie header | Session cookies (SID, HSID, __Secure-*, cf_clearance) | | Authorization header | Bearer <jwt> or Basic <base64> | | Custom headers | X-CSRF-Token, X-Request-ID, X-App-Token | | Request body | csrf_token, at=<token>, _token fields | | URL params | ?token=, ?key=, ?sid= |

CSRF tokens specifically — always scrape from the page, not hardcode:

• View page source (Cmd+U): search for csrf, _token, SNlM0e, xsrf
• DevTools Application tab → Storage → Local Storage / Session Storage
• DevTools Application tab → Cookies

Tools beyond DevTools (when you need them)

| Tool | When to use | |---|---| | mitmproxy | Mobile apps, Electron apps, or when DevTools is insufficient | | Postman Proxy | Capture streams of requests from multiple clients | | Burp Suite | Deep inspection, request mutation, replay with modifications | | Playwright | When you need a real browser for JS execution but want to intercept traffic |

---

Phase 2 — Auth Extraction

Authentication in web apps falls into five patterns. Identify which one you're dealing with:

Pattern A — Session Cookies (most common for Google, social platforms)

What it looks like: Cookie: SID=xxx; HSID=yyy; SSID=zzz

How to extract programmatically:

• Use Playwright to log in as a real user, save the browser storage state to a JSON file
• Load that JSON in your client: parse cookies, build Cookie header string
• Store the file securely — it contains your full session

// Playwright: save after login
await context.storageState({ path: 'auth.json' });

// Your client: load
const storage = JSON.parse(fs.readFileSync('auth.json'));
const cookies = storage.cookies
  .map((c: any) => `${c.name}=${c.value}`)
  .join('; ');

Additional tokens scraped from page HTML (common in Google products):

// Fetch the app homepage with your cookies, then regex for tokens
const html = await fetch(baseUrl, { headers: { Cookie: cookies } }).then(r => r.text());
const csrfToken = html.match(/"SNlM0e":"([^"]+)"/)?.[1];  // Google CSRF pattern
const sessionId = html.match(/"FdrFJe":"([^"]+)"/)?.[1];  // Google session ID

Pattern B — Bearer JWT (REST APIs, newer products)

What it looks like: Authorization: Bearer eyJhbGc...

Extraction:

• DevTools → Application → Local Storage → look for token, access_token, jwt
• Or captured directly from a /login or /auth/token response body
• JWTs expire — your client needs to call the refresh endpoint and swap the token

// Decode without verifying (inspection only — never trust unverified JWTs for auth decisions)
const payload = JSON.parse(atob(token.split('.')[1]));
console.log('Expires:', new Date(payload.exp * 1000));

Pattern C — API Key in Header

What it looks like: X-API-Key: abc123 or api-key: abc123

Simplest case — extract once, store in env var, add to every request.

Pattern D — Form-encoded CSRF token in body

What it looks like: Request body contains at=XXXXXXXX&f.req=... (Google batchexecute)

• Scraped from page HTML (see Pattern A above for extraction)
• Re-scraped on session refresh — tokens rotate with sessions

Pattern E — Dynamic request signing

What it looks like: X-Signature: <hash> that changes every request

This is the hard case. The signature is computed in the app's JavaScript using a secret or a deterministic function. You must:

1. Find the signing function in the minified JS (DevTools → Sources → search for the header name)
2. Understand what inputs it hashes (timestamp + nonce + body + secret?)
3. Replicate the hash in your client

This is genuinely difficult. If you hit this pattern, consider using Playwright to run a real browser and intercept requests via page.on('request') instead of replicating the signing.

---

Phase 3 — Request Structure Analysis

Identify the protocol

| Signal | Protocol | How to handle | |---|---|---| | URL contains /graphql | GraphQL | Use introspection query to get schema | | Body is f.req=... URL-encoded | Google batchexecute RPC | See google-batchexecute-rpc skill | | Body is {"query": "...", "variables": {...}} | GraphQL | Standard GraphQL client | | Body is flat JSON {"key": "value"} | REST | Straightforward — replicate body shape | | Response starts with )]}'\n | Google APIs | Strip that prefix before JSON.parse | | Response is data: {...}\n\n chunks | SSE streaming | Read as stream, parse each data: line | | Response is binary chunks over single connection | WebSocket | Different tooling — use ws library | | Body is multipart/form-data | File upload endpoint | Use FormData in fetch |

What parameters are required vs optional

Test this by replaying the cURL capture with headers removed one at a time. A 200 without a header = it was optional. A 401/403/400 = it was required.

The minimum viable request is the most stable. More headers = more breakage surface.

Pagination and async jobs

Many API actions are async — you submit a job and poll for results:

POST /generate → { jobId: "abc123" }
    ↓ poll every 2s
GET /status?jobId=abc123 → { status: "pending" }
GET /status?jobId=abc123 → { status: "pending" }
GET /status?jobId=abc123 → { status: "done", outputUrl: "..." }

Capture BOTH the submission request and all the subsequent polling requests.

---

Phase 4 — Building the Client

Structure every client the same way

your-product-client/
├── auth.ts          ← load cookies/tokens, scrape page tokens, refresh logic
├── encoder.ts       ← build request body for each action
├── decoder.ts       ← parse response format, strip envelopes, handle errors
├── client.ts        ← the public API your pipeline code calls
└── types.ts         ← TypeScript types for requests and responses

auth.ts pattern

export interface AuthTokens {
  cookies: Record<string, string>;
  csrfToken?: string;
  sessionId?: string;
  bearerToken?: string;
  expiresAt?: Date;
}

export async function loadAuth(storagePath: string): Promise<AuthTokens> {
  // Load from Playwright storage state or your own JSON store
}

export async function refreshIfNeeded(auth: AuthTokens): Promise<AuthTokens> {
  // Check expiry, call refresh endpoint, return updated tokens
}

export function buildHeaders(auth: AuthTokens): Record<string, string> {
  // Build the exact headers your captured requests require
}

decoder.ts — handle every response format

export function decodeResponse(raw: string, format: 'json' | 'google-rpc' | 'sse'): any {
  if (format === 'google-rpc') {
    // Strip )]}'\n prefix, parse outer array, extract payload
    const clean = raw.replace(/^\)]\}'\n/, '');
    // ... see google-batchexecute-rpc skill for full decoder
  }
  if (format === 'sse') {
    return raw.split('\n')
      .filter(line => line.startsWith('data: '))
      .map(line => JSON.parse(line.slice(6)));
  }
  return JSON.parse(raw);
}

Error handling — the four cases you will always hit

// 1. Auth expired — re-login and retry once
if (response.status === 401 || response.status === 403) {
  auth = await refreshAuth();
  return retry(request, auth);
}

// 2. Rate limiting
if (response.status === 429) {
  const retryAfter = response.headers.get('retry-after') ?? '60';
  await sleep(parseInt(retryAfter) * 1000);
  return retry(request, auth);
}

// 3. Server error — retry with backoff
if (response.status >= 500) {
  await sleep(exponentialBackoff(attempt));
  return retry(request, auth, attempt + 1);
}

// 4. Response format changed (app deployed an update)
// — this breaks silently. Log the raw response before parsing.
console.log('[decoder] raw response:', rawText.slice(0, 200));

---

Phase 5 — Stability and Maintenance

Private APIs break without warning. Build defensively:

What breaks and how to detect it fast

| What changes | Symptom | Detection | |---|---|---| | Token name renamed | 401 on all requests | Health check endpoint, alert on auth fail | | Request body field renamed | 400 Bad Request or silent wrong result | Validate response shape, not just status 200 | | Response structure changed | JSON parse error or missing fields | Schema validation on decoded response | | New required header added | 403 or app-specific error code | Log raw responses, inspect error body | | Endpoint path changed | 404 | Monitor for 404s in production |

The minimal health check

// Run this on startup and periodically
async function healthCheck(client: YourClient): Promise<boolean> {
  try {
    const result = await client.someSimpleReadOperation();
    return result !== null;
  } catch (e) {
    console.error('[health] client broken:', e.message);
    return false;
  }
}

Auth rotation

Session cookies expire — typically 30 days to 1 year. JWTs expire faster (minutes to hours). Build a scheduled job that:

1. Checks if auth is within 24h of expiry
2. If so, uses Playwright to re-login and saves fresh cookies
3. Updates the stored auth file/record
4. Notifies you if re-login fails

---

Recon Checklist — One Action at a Time

For each UI action you want to automate, capture:

• [ ] Full URL (copy from address bar in Network tab)
• [ ] HTTP method
• [ ] All headers (Headers tab → expand Request Headers)
• [ ] Request body (Payload tab — check both Form Data and Raw views)
• [ ] Response format and shape (Response tab — prettify JSON)
• [ ] cURL copy (right-click request → Copy → Copy as cURL)
• [ ] Any subsequent polling requests after async submission
• [ ] The download/result URL format from the completed job

Minimum to capture per project:

• [ ] Login flow (to understand cookie/token acquisition)
• [ ] One read operation (list/get)
• [ ] One write operation (create/submit)
• [ ] One async job + poll cycle (if the product has async generation)
• [ ] One file upload (if the product accepts files)

---

What This Skill Does Not Cover

• Mobile app APIs: require mitmproxy + certificate pinning bypass (different skill)
• Electron app APIs: similar to web but open DevTools via --inspect flag
• WebSocket-heavy apps: require ws library and different capture approach
• Apps with client-side request signing: require JS deobfuscation (different skill)
• Anti-bot systems (Cloudflare Bot Management, DataDome, Akamai): this methodology does not defeat them — requires headless browser with real fingerprint or different approach