shield_lock
金大哥
← 返回 Skill 列表
extension
分类: AI Agent 能力无需 API Key

SkillCapital

Audits an Awesome-style dataset list (e.g. Awesome-RS-VL-Data) to determine where each dataset is actually hosted and downloadable. Parses the README tables, classifies datasets, investigates the real download platform behind every entry (HuggingFace, Google Drive, Baidu NetDisk, ModelScope, Zenodo, etc.), flags application-gated datasets, checks for ModelScope mirrors via the official API, produces per-category and merged CSVs, and runs platform-share statistics by year and venue. Use when the user wants to survey, tabulate, or analyze the publishing/hosting platforms of datasets listed in a GitHub awesome-list or curated markdown table, or asks to build a dataset-platform spreadsheet with download links.

person作者: VoyagerXhubModelScope
download下载资源包

Remote-Sensing Dataset Platform Audit

Turn a curated dataset markdown list into a rigorously verified, analyzable spreadsheet of where each dataset is really hosted.

Golden rule: Download_Link = the URL where the dataset FILES physically live and can be fetched (a HuggingFace dataset page, a Google Drive folder, a Baidu NetDisk share, a Zenodo record, …). A GitHub code repository counts as the platform only when the data files are committed inside the repo or attached to a Release. If the repo README points elsewhere (Drive / HF / NetDisk / …), that external location is the platform; keep the repo URL in Comments as Project: <url>.

Inputs & prerequisites

  • The source list: a cloned repo or a README.md containing dataset tables.
  • Network access (WebFetch / WebSearch) for verifying links.
  • Optional: a ModelScope access token (ms-...) to check ModelScope mirrors via the API. Ask the user for it if ModelScope coverage is requested.
  • Python 3 with only the standard library (all helper scripts are stdlib-only).

Output column schema

Every CSV uses exactly these columns, in order:

Name, Year, Venue, Cls1, Cls2, Platform1, Download_Link1,
Platform2, Download_Link2, Needs_Application, Comments, Error, ModelScope_Mirror
  • Cls1: top-level class — Comprehensive Data, Comprehensive Benchmarks, Task-specific Data, Meta Data.
  • Cls2: sub-class. When a top-level class has no sub-headers, set Cls2 = Cls1.
  • Platform1/Download_Link1: primary real data host + specific URL.
  • Platform2/Download_Link2: secondary host (many datasets mirror on a NetDisk). If more than two hosts exist, describe the extras in Comments.
  • Needs_Application: Yes if the primary/official route requires a form / registration / email / EULA; else No. If an open author-hosted mirror exists, prefer it as Platform1 and mark No.
  • Comments: Project: <repo/paper url> plus any notes (extra platforms, access codes/passwords, subset caveats).
  • Error: broken/placeholder/wrong links found in the source + the corrected link, or "not yet released" status.
  • ModelScope_Mirror: the ModelScope dataset URL if a genuine mirror exists, else No.

Workflow

Copy this checklist and track progress:

- [ ] 1. Parse README tables into structured records
- [ ] 2. Split by sub-category into batch files
- [ ] 3. Investigate real download platform per dataset (parallel sub-agents)
- [ ] 4. Resolve application-gated datasets to their real host
- [ ] 5. Check ModelScope mirrors via API
- [ ] 6. Assemble per-category + merged CSVs
- [ ] 7. Run platform-share statistics (by year, by venue)
- [ ] 8. Verify (counts, false-positive review, dead-link sanity)

Step 1 — Parse the README

Run scripts/parse_readme.py <README.md>. It walks ## (Cls1) and ### (Cls2) headers, then parses each markdown table row, extracting Year, Venue, Country/Keywords, Name (+ paper link), and the raw Download cell. It emits parsed_datasets.json.

Pitfall: Change Detection contains the substring Detection — test the more specific sub-header first when mapping Cls2.

Step 2 — Batch by sub-category

Group records by Cls2 and write one batch_<Cls2>.json per group. This keeps each sub-agent's workload focused and context small.

Step 3 — Investigate the real platform (use sub-agents in parallel)

Launch one general-purpose sub-agent per 2 batches (≈5 agents total for a ~270-entry list). Each agent, for every dataset:

  1. If the raw Download cell is already a direct platform URL (huggingface.co/datasets, drive.google.com, pan.baidu.com, kaggle.com, zenodo.org, opendatalab.com, modelscope.cn, …) → use it directly as Download_Link1.
  2. If it is a GitHub repo, WebFetch the repo README and find where the data actually downloads from. Repo URL → Comments: Project: ...; the real host becomes Platform1. Only keep GitHub as Platform1 when data files live in the repo/Release.
  3. If it is N/A/NaN/empty, WebFetch the paper link for a data-availability statement; WebSearch as a fallback.
  4. If a link 404s / is a placeholder (e.g. YourOrg) / is a shields.io badge URL captured by mistake / has a malformed suffix → record the problem and the corrected link in Error.

Platform-name normalization (use these canonical labels): HuggingFace, Google Drive, Baidu NetDisk, Baidu AI Studio, Kaggle, Zenodo, OpenDataLab, ModelScope, OneDrive/SharePoint, IEEE DataPort, GitHub Pages, Project Website, University Website, plus long-tail ones as needed.

Each agent writes results_<batch>.json with the full column set.

Step 4 — Resolve application-gated datasets

For any dataset behind a form/registration/email, still try to find the real storage:

  • Author-hosted open mirror exists (HuggingFace / OpenDataLab) → use it as Platform1, Needs_Application = No.
  • Only an official gated route + a third-party mirror → keep official as Platform1, surface the mirror in Platform2/Comments, Needs_Application = Yes.
  • Truly gate-only (no discoverable host) → Platform1 = Application Required with the form/email as the link, Needs_Application = Yes.

Step 5 — Check ModelScope mirrors

Use the search API (see references below). For each dataset:

  1. First reuse any modelscope.cn URL already present in the collected fields.
  2. Otherwise query by cleaned name and match on the normalized repo name (lowercase, strip non-alphanumerics). Require exact-normalized equality, or containment when the token length ≥ 5.
  3. Manually review every fuzzy hit — this is where false positives hide (e.g. a general "GAIA" agent benchmark ≠ the RS "GAIA"; "laser" ≠ "LaSeRS"; a text "choice" dataset ≠ the RS "CHOICE"; a base "LEVIR-CD" ≠ "LEVIR-CD+"; a small subset or a re-processed derivative is not a full mirror).

scripts/modelscope_search.py implements the working endpoint and matching.

Step 6 — Assemble CSVs

Merge all results_*.json, sort each group by Year then Name, and write one CSV per Cls2 plus one merged ALL_*.csv (all rows, same schema). Use utf-8-sig encoding so Excel opens Chinese/links cleanly.

Step 7 — Statistics

Compute platform-share distributions: overall Platform1, Platform1 by Year, and Platform1 by Venue (report the top venues). Save analysis_summary.json and a short markdown report. Typical finding to sanity-check against: HuggingFace share rises steeply after ~2024; NetDisk/Drive dominate earlier years; AI venues favor HuggingFace while domain journals stay more fragmented.

Step 8 — Verify

  • Row counts per category match the parse.
  • Re-review all fuzzy ModelScope hits and any Platform1 that is still a bare code repo.
  • Spot-check a sample of "dead link" flags (some fetch failures are transient; do not over-flag canonical institutional pages).

Utility scripts

  • scripts/parse_readme.py — parse README tables → parsed_datasets.json. python scripts/parse_readme.py path/to/README.md
  • scripts/modelscope_search.py — search ModelScope for a dataset name. MODELSCOPE_API_TOKEN=ms-xxx python scripts/modelscope_search.py "Git-10M"

Pitfalls

  • Do NOT equate a GitHub star badge / project page with the data host.
  • Watch normalization collisions (+, spaces, - stripped) that merge distinct datasets (LEVIR-CD vs LEVIR-CD+).
  • Placeholder links (YourOrg), shields.io badge URLs, and .code/.git suffixes are common source-list bugs — record them in Error.
  • A subset or a re-processed variant on ModelScope is not a full mirror; mark conservatively and note it.
  • ModelScope search API: PUT/POST on dolphin/datasets return 404; only the GET .../api/v1/dolphin/datasets?Query=<kw> form accepts free-text search.