shield_lock
JinDaGe
← Back to skills
extension
Category: Data & AnalyticsAPI key required

Alibabacloud Data Agent Skill

Invoke Alibaba Cloud Apsara Data Agent for Analytics via CLI to perform natural language-driven data analysis on enterprise databases. Data Agent for Analyti...

personAuthor: sdk-teamhubclawhub
open_in_newRepositorydownloadDownload package

metadata: author: DataAgent Team version: "1.8.5"

Changelog

  • v1.8.5 — Database listing migrated to ListTagMetaAsset (dms-enterprise 2018-11-01); workspace auto-resolution (CLI --workspace-id > env DATA_AGENT_WORKSPACE_ID > InitDataAgentPersonalWorkspace); db subcommand relaxed --dms-instance-id / --instance-name to optional.
  • v1.8.4: Document project Python virtualenv (venv/) setup and activation; add end-to-end regression notes for ASK_DATA / ANALYSIS (async + attach)
  • v1.8.3: db and file subcommands now accept --session-mode CLAW
  • v1.8.2: SendChatMessage now supports per-message Mode=CLAW (injected via SessionConfig.Mode); dynamic DMSUnit resolution via GetActiveRouteUnit
  • v1.8.1: Emphasize attach-based session reuse as the core interaction mechanism; add golden workflow, capability matrix, and usage rules
  • v1.8.0: Add workspace (collaborative space) support, add custom agent support
  • v1.7.2: Use Alibaba Cloud default credential chain instead of explicit AK/SK, add User-Agent header, fix RAM policy wildcard issues
  • v1.7.1: Fix CLI ls command API response parsing (support case-insensitive field names), optimize SKILL documentation structure, separate ANALYSIS mode specification document
  • v1.7.0: API_KEY authentication support, native async execution mode, session isolation, enhanced attach mode, optimized log output

Installation

Python Environment (venv) — MUST READ

🚨 Hard Requirement: Python ≥ 3.10

The macOS system /usr/bin/python3 is typically 3.8 or 3.9 and cannot run this project (it relies on match/case, TypeAlias, | union syntax, and other 3.10+ features).

Verify your version first: python3 --version. If below 3.10, install via Homebrew or pyenv:

# Homebrew
brew install python@3.12
# Or pyenv
pyenv install 3.12.4 && pyenv local 3.12.4

⚠️ You MUST use a venv virtual environment. Never install dependencies globally. Running pip install against the system Python pollutes the environment and may fail due to permission issues.

Use Existing venv (Recommended)

The project ships a pre-built venv/ directory (all dependencies pre-installed). Use it whenever possible:

cd alibabacloud-data-agent-skill

# Option A (recommended): activate the venv
source venv/bin/activate
python3 scripts/data_agent_cli.py ls

# Option B: invoke the venv interpreter directly (no activation needed)
venv/bin/python3 scripts/data_agent_cli.py ls

Rebuild venv

If venv/ is missing or dependencies are corrupted, recreate with a 3.10+ Python:

python3.12 -m venv venv          # explicitly use a 3.10+ interpreter
source venv/bin/activate
pip install -r scripts/requirements.txt

Tip: All examples in this document write python3 scripts/data_agent_cli.py .... When venv is activated, python3 resolves to the venv interpreter automatically; otherwise prefix with venv/bin/python3.

Configure Credentials

This Skill uses Alibaba Cloud default credential chain (recommended) or API_KEY authentication.

Option 1: Default Credential Chain (Recommended)

The Skill uses Alibaba Cloud SDK's default credential chain to automatically obtain credentials, supporting environment variables, configuration files, instance roles, etc.

See Alibaba Cloud Credential Chain Documentation

Option 2: API_KEY Authentication (File Analysis Only)

export DATA_AGENT_API_KEY=your-api-key
export DATA_AGENT_REGION=cn-hangzhou

Get API_KEY: Data Agent Console

Permission Requirements

RAM users need AliyunDMSFullAccess or AliyunDMSDataAgentFullAccess permissions. See RAM-POLICIES.md for detailed permission information.

Debug Mode

DATA_AGENT_DEBUG_API=1 python3 scripts/data_agent_cli.py file example.csv -q "analyze"

💡 Getting Started Tips

  • Use the built-in demo database internal_data_employees (DataAgent's built-in test database containing employee, department, and salary data) for first-time experience
  • Or use local file assets/example_game_data.csv for file analysis experience

Data Agent CLI — Unified Command-Line Data Analysis Tool

Overview

scripts/data_agent_cli.py helps users complete the full workflow from discover data → initiate analysis → track progress → get results.

Core Concepts

⚠️ Key Prerequisite: Data Agent can only analyze databases that have been imported into Data Agent Data Center.

  • Data Center: Data Agent's data center, only databases here can be analyzed
  • DMS: Alibaba Cloud Data Management Service, stores metadata of all databases
  • Relationship: Databases registered in DMS ≠ Databases in Data Center

Usage Flow:

  1. First use ls to check if the target database exists in Data Center
  2. If not found, use dms subcommand to search for database info, then use import subcommand to import it
  3. After successful import, you can use db subcommand for analysis

Analysis Modes

  • ASK_DATA (default): Synchronous execution, sub-second response, suitable for quick Q&A
  • ANALYSIS: Deep analysis, takes 5-40 minutes, requires spawning a sub-agent for async execution or using --async-run parameter
  • INSIGHT: Insight-oriented exploration, follows the same plan-confirmation flow as ANALYSIS
  • CLAW: Agentic CLAW mode. Two entry points:
    • CLI: db --session-mode CLAW ... / file --session-mode CLAW ... (session-level)
    • SDK: pass mode="CLAW" to client.send_message(...) / AsyncDataAgentClient.send_message(...) to override mode for a single message via SessionConfig.Mode

End-to-End Regression Reference (v1.8.4 verified)

Both ASK_DATA and ANALYSIS modes are regression-tested against chinook database with the async + attach flow:

| Mode | Kickoff | Observed Chain | Typical Duration | |------|---------|----------------|------------------| | ASK_DATA | db --session-mode ASK_DATA -q "..." | async worker → live SSE → result.json={"status":"completed"} | ~15s | | ANALYSIS | db --session-mode ANALYSIS -q "..." | async worker → Plan → WAIT_INPUT → attach -q "confirm" → step-by-step execution → Excel/Chart artifacts → text report → 2nd WAIT_INPUT (webpage render) | 2-10 min (text); +10 min if rendering webpage |

Key checkpoints to look for in sessions/<SESSION_ID>/progress.log:

  • > User Query: ... — request received
  • ### Execution Plan (ID: ...) — ANALYSIS plan generated, use attach -q "confirm" to proceed
  • > ⚠️ Plan confirmed, continuing analysis... — plan approved, execution starts
  • ## Step N/M: ... — per-step progress with artifacts links
  • ### Report Render + ⚠️ Please review the report rendering request. — optional HTML report render confirmation

See ANALYSIS_MODE.md for details

Workspace (Collaborative Space)

Workspaces are collaborative spaces that enable team-based data analysis with shared sessions, data sources, and access control.

  • List workspaces: Use workspace subcommand to discover available workspaces (personal or shared)
  • Bind session to workspace: Pass --workspace-id <ID> when using db or file to create a session within a specific workspace context
  • Workspace types: MY (default, personal spaces), ALL (all accessible spaces including shared ones)

Note: When a session is created within a workspace, all subsequent API calls (describe, send message, etc.) automatically carry the workspace context.

Workspace Resolution

The workspace ID is resolved automatically in this order:

  1. CLI flag --workspace-id <id>
  2. Environment variable DATA_AGENT_WORKSPACE_ID
  3. Auto-create personal workspace via InitDataAgentPersonalWorkspace

Both AK/SK and API_KEY authentication modes support this resolution chain.

Custom Agent

Custom Agents are user-defined AI agents with specialized instructions, knowledge bases, and data scope configurations.

  • List custom agents: Use agent subcommand to discover available custom agents (RELEASED status by default)
  • View agent details: Use agent describe --custom-agent-id <ID> to see full agent configuration
  • Bind session to custom agent: Pass --custom-agent-id <ID> when using db or file to create a session powered by a specific custom agent

Note: Custom Agent sessions automatically use the prod stage. The custom agent's instructions, knowledge, and data scope will be applied to the analysis session.

Session Reuse via attach (⭐ Core Mechanism)

Best Practice: attach is the recommended way to interact with an ongoing or previously created session. Always prefer attach over creating a new session for any follow-up interaction on the same data scope.

Why Use attach

After you call db / file to start a session, all subsequent interactions on that session MUST go through attach --session-id <ID>. A single session = a single conversation context on the server side, and attach is the only way to safely re-enter it.

| Capability | Command | Scenario | |------------|---------|----------| | Follow-up questions | attach --session-id <ID> -q "..." | Continue the conversation with full context, skip data-understanding overhead | | Plan confirmation | attach --session-id <ID> -q "confirm" | Approve the execution plan generated by ANALYSIS/INSIGHT mode | | Plan modification | attach --session-id <ID> -q "simplify to 3 steps" | Refine the plan before execution | | Progress monitoring | attach --session-id <ID> (no -q) | Tail live SSE progress of a long-running session | | Resume after network drop | attach --session-id <ID> --checkpoint <N> | Precise recovery from the Nth event after interruption | | Replay full history | attach --session-id <ID> --from-start | Re-stream the entire session from event 0 |

Golden Workflow (Async + attach)

The canonical pattern for long-running analyses is async db kickoff → attach for everything else:

# 1) Kick off async analysis, returns SESSION_ID immediately
python3 scripts/data_agent_cli.py db \
    --dms-db-id <dbId> \
    --db-name <schemaName> \
    --tables "employees,departments" \
    --workspace-id <workspace_id> \
    --session-mode ANALYSIS \
    -q "Analyze salary distribution"
# -> ✅ Async task started. Session ID: abc123xyz

# 2) Watch progress live (Ctrl-C is safe, server keeps running)
python3 scripts/data_agent_cli.py attach --session-id abc123xyz

# 3) Confirm or modify the plan when agent enters WAIT_INPUT
python3 scripts/data_agent_cli.py attach --session-id abc123xyz -q "confirm"

# 4) Ask follow-up questions (reuses context, no re-import, no re-profiling)
python3 scripts/data_agent_cli.py attach --session-id abc123xyz -q "Break down by job level"

# 5) Recover precisely if the stream was cut at event #219
python3 scripts/data_agent_cli.py attach --session-id abc123xyz --checkpoint 219

# 6) Fetch generated reports / charts
python3 scripts/data_agent_cli.py reports --session-id abc123xyz

Benefits of Reusing a Session via attach

  • Context preservation — previous SQL, table profiling, and user intent are kept, answers stay consistent.
  • Cost reduction — skip re-discovering schema / re-profiling tables on every question.
  • Plan governance — ANALYSIS / INSIGHT plans require explicit confirmation; only attach -q "confirm" can unblock them.
  • Resilience — --checkpoint / --from-start make long-running tasks robust against network drops and client restarts.
  • Team collaboration — share the Session ID, teammates can attach to the same session to review progress and results.

Rules of Thumb

  1. Create session once with db / file; drive everything else with attach.
  2. Record the Session ID printed after kickoff — it is the only handle to the session.
  3. For ANALYSIS / INSIGHT mode, always use attach (not a new db) to confirm plans; creating a new session loses the plan.
  4. Session artifacts (progress log, checkpoint, result, images) are persisted under sessions/<SESSION_ID>/.

See COMMANDS.md for the full attach parameter list and WORKFLOWS.md for end-to-end scenarios.

Quick Start

# 1. List available databases
python3 scripts/data_agent_cli.py ls
# Example output:
#   chinook [mysql]  dbId=abc123  instanceResourceId=rm-xxx  catalogName=chinook
#   employees [mysql]  dbId=def456  instanceResourceId=rm-yyy  catalogName=employees

# 2. Create a session for initial analysis (record the returned Session ID!)
python3 scripts/data_agent_cli.py db \
    --dms-db-id <dbId> \
    --db-name <schemaName> \
    --tables <table1,table2> \
    --workspace-id <workspace_id> \
    -q "Which department has the highest average salary"
# -> ✅ Async task started. Session ID: abc123xyz

# 3. ⭐ Reuse the session — follow-up questions, confirm plans, monitor progress
python3 scripts/data_agent_cli.py attach --session-id abc123xyz -q "Break down by month"
python3 scripts/data_agent_cli.py attach --session-id abc123xyz -q "confirm"     # approve ANALYSIS plan
python3 scripts/data_agent_cli.py attach --session-id abc123xyz                  # tail live progress
python3 scripts/data_agent_cli.py attach --session-id abc123xyz --checkpoint 219 # resume after drop

# 4. List workspaces
python3 scripts/data_agent_cli.py workspace

# 5. Query in a specific workspace
python3 scripts/data_agent_cli.py db \
    --workspace-id <WORKSPACE_ID> \
    --dms-db-id <dbId> \
    --db-name <schemaName> \
    --tables <table1,table2> -q "Which department has the highest average salary"

# 6. List available custom agents
python3 scripts/data_agent_cli.py agent

# 7. Use a custom agent for analysis
python3 scripts/data_agent_cli.py db --custom-agent-id <AGENT_ID> --dms-instance-id ... -q "your question"

Remember: db / file create the session once; all follow-ups go through attach --session-id <ID>.

📖 See WORKFLOWS.md and COMMANDS.md for complete workflows, command reference, and best practices

Project Structure

                          # Skill root directory
├── SKILL.md              # This document
├── scripts/              # Source code
│   ├── data_agent/       # SDK module
│   ├── cli/              # CLI module
│   ├── data_agent_cli.py # CLI entry point
│   └── requirements.txt  # Dependencies
├── sessions/             # Session data
└── references/           # Reference documents