ModelScope Notebook Develop

Core Workflow

Inventory the source tutorial
- Locate the source notebook(s): local .ipynb, GitHub URL/repo, Colab URL, or tutorial docs.
- Preserve official markdown/text unless the user asks for a shorter adaptation.
- Record the official data/model sources, expected outputs, runtime, dependency versions, and any optional cells.
- If the user has not provided enough context, briefly explain the workflow and ask for the missing notebook/source URL, target ModelScope namespace, and whether full execution is required.
Audit dependencies and runtime
- Identify Python version, CUDA/PyTorch versions, pip/conda dependencies, CLI tools, and shell commands.
- Read dependencies.txt, the 486-package default-image snapshot supplied by the user, before adding installation cells. Directly import already installed packages whose APIs satisfy the tutorial; remove redundant %pip install / !pip install. This inventory is not a requirements file and must not be installed wholesale.
- The image may change. When needed, inspect the actual kernel with importlib.metadata.version; a difference from the snapshot alone is not a reason to reinstall, upgrade or downgrade.
- Install only actually missing packages or dependencies with a confirmed compatibility blocker. In a Notebook, use %pip install <needed-package> rather than !pip. Pin only versions required for compatibility; avoid replacing the CUDA/PyTorch stack unnecessarily.
- After an install changes packages already loaded by the kernel, restart through the Notebook UI or supported kernel control and explain that step to the user. Do not add restart instructions when no installation occurred.
- If you encounter dependency version conflicts, CUDA/PyTorch version mismatches, or checkpoint path mapping issues, use the notebook-dependency-compatibility-fixes skill (pre-installed in ModelScope Notebook's built-in Qwen Code) for targeted fixes.
- Add early capability checks for fragile version features, e.g. inspect function signatures or enum values before expensive training.
- Separate warnings from blockers. GPU/CUDA warnings are not model-loading errors; parser/config errors often happen before GPU use.
Map assets to ModelScope
- Before mirroring any Hugging Face model or dataset, check whether the identical owner/repo_name already exists on ModelScope, using the correct model/dataset type. Verify the upstream provenance, license, needed revision, actual weight/config/tokenizer/data files, and anonymous or authorized download access; a matching name alone is insufficient. If the existing repo meets the lesson requirements, reuse that exact ModelScope repo ID and do not host another copy under the user or organization.
- If the identical ID is missing, search for an official compatible ModelScope repo before mirroring. Record the checked IDs and why existing resources cannot satisfy the lesson. Recheck live metadata rather than assuming a cache miss or transient network/auth error means the repo is absent.
- Mirror only missing resources or necessary derived artifacts (for example course-specific preprocessing); reuse the original model/dataset for everything already available. When layouts or tensor keys differ, adapt the local loader and validate equivalence instead of republishing unchanged weights. Continue to download from ModelScope in the Notebook, not from HF at runtime.
- When mirroring is needed, obtain assets from official upstream sources on a network-capable preparation machine; hf-mirror.com may be an acquisition fallback there. Restricted-network ModelScope Notebooks must use ModelScope/local assets, without Hugging Face, Kaggle or PyTorch external download fallbacks in the execution code.
- If neither local nor ModelScope has the required assets, guide the user to provide access tokens/credentials for the upstream source and for ModelScope upload.
- Never print tokens. Use environment variables and redact secrets in logs.
- Check license before mirroring. If the license forbids redistribution or is unclear, stop and tell the user; do not upload. If allowed, carry license/source info into README and metadata.
Create or update ModelScope repositories
- Enter this step only after the asset checks above establish that required resources are missing or need a course-specific derived artifact. Existing compatible repositories do not need another user/organization mirror.
- Put datasets in dataset repositories and pretrained weights in model repositories. Do not mix all assets into one umbrella repo unless the user explicitly wants that.
- Each repository must have README.md with YAML front matter and body that agree on:
  - license
  - source URL/name
  - file list
  - associated models: or datasets: when relevant
- For user-owned mirrors, use clear repo IDs such as namespace/tutorial-dataset-name and namespace/model-name.
- Keep local smoke checkpoints separate from pretrained weights unless the user explicitly asks to publish them.
Adapt notebook code
- For multi-file Gallery projects, read references/gallery-workspace.md: the updated Notebook UI prompts users to copy the Gallery project from its temporary mount into persistent storage. Preserve that copy and use a single quoted %cd to the correct project/helper directory. Do not add recursive searches, sys.path edits or Gallery API download bootstraps by default.
- Add only the model/data downloads actually used by the lesson, using dataset_snapshot_download / snapshot_download when needed; keep imports minimal.
- Use repo-specific local dirs such as tutorial_assets/datasets/<repo> and tutorial_assets/models/<repo>.
- Replace external downloads (wget, gdown, hf_hub_download, Colab drive mounts) with ModelScope downloads.
- Prefer system tar -xf for large archives with many small files; Python tarfile.extractall() can be much slower on notebook cloud storage.
- For large assets, skip re-download/re-extract when the target directory already exists, but keep a clean path for first runs.
- Preserve official cells where possible; when skipping optional online-only comparisons, leave an explicit skip cell and explain what asset is missing.
Execute and verify
- First run a narrow smoke path: download small files, load model, load one dataset sample, run one forward pass.
- Then run the full notebook when requested or when outputs must match official tutorials.
- To execute the notebook from the terminal (so the agent can capture outputs/errors without a browser), use jupyter nbconvert --execute. First check whether nbconvert and nbclient are installed:
```
python - <<'PY'
import sys, subprocess
from importlib.metadata import version, PackageNotFoundError
missing = []
for package in ("nbconvert", "nbclient"):
    try:
        print(package, version(package))
    except PackageNotFoundError:
        missing.append(package)
if missing:
    subprocess.check_call([sys.executable, "-m", "pip", "install", *missing])
PY
```
  Then run:
```
jupyter nbconvert \
  --to notebook \
  --execute input.ipynb \
  --inplace \
  --ExecutePreprocessor.kernel_name=python3
```
  Use --inplace to overwrite the input file with executed outputs, or --output executed.ipynb to write to a separate file. Inspect the terminal output and the executed notebook cells for errors. If a cell raises an exception, nbconvert stops and prints the traceback — fix the cell and re-run.
- Save debug images/outputs locally when visual quality matters; inspect them directly.
- Compare against official outputs and report exact differences, including whether differences are due to data source, model version, random seed, shorter training, or unavailable optional assets.
- If a full run is blocked by local network/sandbox limits, say exactly which command failed and whether the notebook is expected to run in ModelScope cloud.
Update reports and artifacts
- Update report files with:
  - notebook path and executed notebook path
  - data/model repo IDs and source URLs
  - license information
  - local/cloud runtime notes
  - reproduction metrics and consistency with official tutorial
  - known blockers or environment differences
- Keep generated notebook source and generator scripts in sync when the repo uses a generator.
Publish or update Gallery when requested
- Read references/gallery-api.md and the live OpenAPI before calling endpoints.
- Upload individual files with official POST /files/upload; use its optional filename field for relative paths such as folder1/test1.ipynb, then pass returned file IDs to POST /galleries.
- Select a matching entry/category: notebook → .ipynb, website → .html, pdf → .pdf, file → any suffix. A Gallery can retain all notebooks and helper files while using an HTML entry for preview.
- The official OpenAPI currently has no Gallery update/publish endpoint. Explain that existing-Gallery synchronization uses a separately documented web API compatibility flow, preserving the existing ID/URL and metadata. Do not invent official PATCH endpoints or create a new Gallery on each update.
- If HTML previews are requested, build from notebook sources with nbconvert/nbformat without executing training; preserve navigation and local assets. GitHub Actions runners have their own dependencies and do not inherit the Notebook image inventory.
- For GitHub publishing, use Actions Secrets for tokens, serialize updates to the same Gallery, validate the file manifest and draft sizes before publication, and read back the published result. Report files awaiting CSI review; empty download URLs must be skipped with filenames printed.

Default Notebook Image Dependencies

The root dependencies.txt records the user's original 2026-10-08 image pip list: 486 package/version pairs, including torch 2.13.0, torchvision 0.28.0, transformers 5.16.1, modelscope 1.40.1, requests 2.34.2, datasets 4.8.4, nbconvert 7.17.1 and nbformat 5.11.1. It is an environment reference, not an installation lockfile. Do not generate %pip install -r dependencies.txt or install packages merely because they appear in the inventory.

Check only packages used by the current notebook. Distribution names may differ from imports: scikit-learn → sklearn, pillow → PIL, beautifulsoup4 → bs4, PyYAML → yaml, opencv-python → cv2.

from importlib.metadata import version, PackageNotFoundError
for package in ("modelscope", "requests", "torch", "torchvision"):
    try:
        print(package, version(package))
    except PackageNotFoundError:
        print(package, "missing; install only if this notebook needs it")

Keep setup imports minimal and use only the helper functions needed by each notebook. For the current multi-file Gallery flow, follow the UI copy-to-persistent-storage prompt, preserve all directories, then use one quoted %cd to the directory that makes the lesson imports/data paths valid. Do not automatically download the course again or modify sys.path. The kernel working directory can differ from the opened file location. See references/gallery-workspace.md for the new flow and examples; use older API download bootstraps only for confirmed legacy environments without the copy feature.

Asset and License Rules

Reuse a verified existing ModelScope repo, checking the identical Hugging Face repo_id first for both models and datasets; create user/organization mirrors only when required resources are actually missing.
If mirroring from Hugging Face, inspect the model/dataset card and license. If no machine-readable license exists, state the uncertainty and ask before upload.
Include source provenance in both YAML front matter and README body.
Use associated metadata:

---
license: apache-2.0
repo_type: dataset
tags:
  - modelscope-notebook
source:
  - name: upstream/name
  - url: https://...
models:
  - namespace/model-repo
---

ModelScope Code Pattern

Use this pattern and adapt repo IDs:

from pathlib import Path
from modelscope.hub.snapshot_download import dataset_snapshot_download, snapshot_download

ASSET_DIR = Path("./tutorial_assets")

def _repo_local_dir(kind, repo_id):
    return ASSET_DIR / kind / repo_id.split("/", 1)[1]

def fetch_dataset(repo_id, patterns):
    return Path(dataset_snapshot_download(
        repo_id,
        local_dir=str(_repo_local_dir("datasets", repo_id)),
        allow_patterns=patterns,
        max_workers=4,
    ))

def fetch_model(repo_id, patterns):
    return Path(snapshot_download(
        repo_id=repo_id,
        repo_type="model",
        local_dir=str(_repo_local_dir("models", repo_id)),
        allow_patterns=patterns,
        max_workers=4,
    ))

For large tar archives:

import subprocess
from pathlib import Path

Path("data").mkdir(exist_ok=True)
if not Path("data/dataset").is_dir():
    archive = dataset_asset_dir / "dataset.tar"
    subprocess.run(["tar", "-xf", str(archive), "-C", "data"], check=True)

Common Debug Checks

Model file exists but loading fails: print model directory, file list, config keys, package versions, and traceback.
State dict shape mismatch: compare model config features with installed package support. Example: v1.1 models may require newer architecture fields such as linear patch embeddings; old packages can instantiate v1 architecture and fail at load time.
Notebook cloud hangs on extraction: separate download timing from extraction timing; use tar -xf, not Python extraction, for many-small-file archives.
np.concat fails on cloud: use np.concatenate; numpy 1.26 does not provide np.concat.
Parser errors before training: inspect YAML/class paths and package enum/API compatibility; these are not GPU problems.
ModelScope cache lock or proxy failures: set writable cache dirs when possible, or fall back to CLI-populated cache only as a local diagnostic path.

User Guidance

When asking the user for missing information, keep it short and remind them of the pipeline:

"I will map the notebook assets, check license, mirror/upload missing files to ModelScope if allowed, adapt downloads, run smoke/full execution, and update the report. Please provide the notebook URL/path and target ModelScope namespace."
If upload is needed: ask for ModelScope token and upstream access token only when required; tell the user the tokens will be used via environment variables and not written into repo files.

References

For the current multi-file Gallery copy-to-persistent-storage feature and minimal %cd initialization, read references/gallery-workspace.md.

For a concise checklist and command snippets, see references/checklist.md.

For preinstalled packages, see dependencies.txt. For official Gallery file upload/creation and the separately marked web API update flow, see references/gallery-api.md.