OpenTester Testing Skill

MCP-First testing execution infrastructure for AI coding tools.

Prerequisites

Install OpenTester:
```
pip install opentester
```
Start OpenTester server:
```
opentester start
```
Services:
- Web UI: http://localhost:8000
- MCP Server: http://localhost:8001/mcp

Quick Start

Available Commands

| User Says | Action | |-----------|--------| | "/test" or "/t" | Create and run tests for current changes | | "/test-quick" | Quick test without PRD | | "/test-with-prd" | Comprehensive tests using PRD | | "run tests" | Execute existing tests | | "validate dsl" | Check test syntax | | "list projects" | Show all test projects | | "list templates" | Show all DSL templates | | "use template" | Create test from a template | | "save as template" | Save test case as template |

MCP Tools

| Tool | Description | Parameters | |------|-------------|------------| | list_projects | List all test projects | - | | get_project | Get project details with test cases | project_id | | create_project | Create a new project | name, target_type | | delete_project | Delete a project permanently | project_id | | validate_dsl | MUST USE Validate DSL YAML syntax and schema | dsl_yaml | | save_case | Save a test case | project_id, name, dsl_yaml, case_id (optional) | | delete_case | Delete a test case | project_id, case_id | | run_case | Execute a single test case | project_id, case_id | | run_project | Execute all cases in a project | project_id, case_ids (optional) | | stop_execution | Stop a running execution | execution_id | | get_execution_status | Get execution status | execution_id | | get_execution_log | Get detailed execution log | execution_id |

Template Tools

| Tool | Description | Parameters | |------|-------------|------------| | list_templates | List all available templates | category, target_type, tag (optional) | | get_template | Get template details | template_id | | create_template | Create a new reusable template | name, dsl_template, description, target_type, category, tags | | update_template | Update an existing template | template_id, name, dsl_template, etc. | | delete_template | Delete a template | template_id | | instantiate_template | Create test case from template | template_id, project_id, case_name, variables | | create_template_from_case | Convert existing case to template | project_id, case_id, template_name, variable_mappings |

DSL Validation Workflow (CRITICAL)

ALWAYS validate DSL before saving!

1. Generate DSL YAML
2. Call validate_dsl(dsl_yaml) to check syntax
3. IF validation fails:
   - Read the error message
   - Fix the DSL according to error
   - Call validate_dsl again
   - REPEAT until validation passes
4. IF validation passes:
   - Call save_case to store the test
5. Call run_case to execute

Validation Error Handling

Common errors and fixes:

| Error | Cause | Fix | |-------|-------|-----| | Unknown assertion type | Using type directly in assert step | Use assertion: { type: ..., expected: ... } | | field required | Missing required field | Add the missing field | | YAML error | Indentation/syntax error | Fix YAML formatting |

DSL Format (Correct)

Basic Structure

version: "1.0"
meta:
  name: "Test Name"
  description: "What this test verifies"
  author: "Author"           # Optional
  tags: ["tag1", "tag2"]     # Optional

target:
  type: cli                  # cli, api, or tui

timeout: 30                  # Optional: Default timeout (seconds)

vars:                        # Optional: Variables
  base_url: "http://localhost:3000"

steps:
  - name: "Step description"
    action: exec|assert|wait
    # Action-specific fields below

Action: exec (Execute Command)

CLI Target:

- name: "Execute shell command"
  action: exec
  command: "echo Hello World"     # Required: Command to execute
  working_dir: "/path/to/dir"     # Optional: Working directory
  env:                            # Optional: Environment variables
    KEY: "value"
  timeout: 10                     # Optional: Timeout in seconds

API Target:

- name: "Send HTTP request"
  action: exec
  method: GET                     # HTTP method: GET, POST, PUT, DELETE
  path: "/api/endpoint"           # API path (appended to base_url)
  body:                           # Optional: Request body
    key: "value"
  headers:                        # Optional: Additional headers
    X-Custom: "value"

TUI Target:

- name: "Send input to TUI"
  action: exec
  input: "text to type"           # Text input
  # OR
  key: "enter"                    # Special key: enter, tab, escape, etc.

Action: assert (Verify Conditions)

⚠️ CRITICAL: assert MUST use assertion field, NOT type directly!

CLI Assertions:

- name: "Verify exit code"
  action: assert
  assertion:
    type: exit_code               # Assertion type
    expected: 0                   # Expected value (use 'expected', not 'value'!)

- name: "Verify stdout contains text"
  action: assert
  assertion:
    type: stdout_contains
    expected: "expected text"

- name: "Verify stderr contains error"
  action: assert
  assertion:
    type: stderr_contains
    expected: "error message"

- name: "Verify stdout matches pattern"
  action: assert
  assertion:
    type: stdout_matches
    expected: "^pattern$"

- name: "Verify stderr is empty"
  action: assert
  assertion:
    type: stderr_empty
    expected: true

API Assertions:

- name: "Verify HTTP status"
  action: assert
  assertion:
    type: status_code
    expected: 200

- name: "Verify JSON response"
  action: assert
  assertion:
    type: json_path
    path: "$.data.name"           # JSON path
    expected: "expected value"

TUI Assertions:

- name: "Verify screen content"
  action: assert
  assertion:
    type: screen_contains
    expected: "visible text"

- name: "Verify cursor position"
  action: assert
  assertion:
    type: cursor_at
    expected: "prompt"

Action: wait (Pause Execution)

- name: "Wait for process"
  action: wait
  duration: 5                     # Seconds to wait

Error Handling

- name: "Optional step that may fail"
  action: exec
  command: "optional-command"
  on_error: continue              # Options: fail (default), continue, stop

Target Types

CLI Target

target:
  type: cli
  working_dir: "/path/to/dir"     # Optional
  env:                            # Optional
    KEY: "value"

API Target

target:
  type: api
  base_url: "http://api.example.com"  # Required
  headers:                            # Optional default headers
    Authorization: "Bearer token"

TUI Target

target:
  type: tui
  start_command: "my-tui-app"     # Required: Command to start TUI
  terminal:                       # Optional
    rows: 24
    cols: 80

Variables

vars:
  base_url: "http://localhost:3000"
  timeout: 30

steps:
  - name: "Use variable"
    action: exec
    command: "curl ${vars.base_url}/api"
    timeout: "${vars.timeout}"

Complete Example (CLI)

version: "1.0"
meta:
  name: "Echo Command Test"
  description: "Test basic echo functionality"

target:
  type: cli

timeout: 10

steps:
  - name: "Execute echo command"
    action: exec
    command: echo "Hello World"

  - name: "Verify exit code is 0"
    action: assert
    assertion:
      type: exit_code
      expected: 0

  - name: "Verify output contains expected text"
    action: assert
    assertion:
      type: stdout_contains
      expected: "Hello World"

Complete Example (API)

version: "1.0"
meta:
  name: "API Health Check"
  description: "Test API health endpoint"

target:
  type: api
  base_url: "http://localhost:8000"

timeout: 10

steps:
  - name: "Call health endpoint"
    action: exec
    method: GET
    path: "/health"

  - name: "Verify status code"
    action: assert
    assertion:
      type: status_code
      expected: 200

  - name: "Verify response body"
    action: assert
    assertion:
      type: json_path
      path: "$.status"
      expected: "ok"

Workflows

Create and Run Tests Workflow

When user asks to create tests:

Discover: Call list_projects to find existing projects
Context: Read relevant source files and understand changes
Generate: Create DSL YAML following the CORRECT format above
Validate: Call validate_dsl(dsl_yaml)
Check Result:
- If ❌ validation fails:
  - Read error message carefully
  - Fix the DSL (usually: change type/value to assertion: { type, expected })
  - Call validate_dsl again
  - Repeat until ✅ validation passes
Save: Call save_case with validated DSL
Execute: Call run_case to run tests
Report: Show execution results

Delete Test Case Workflow

When user asks to delete a test case:

Find: Call get_project(project_id) to confirm project and case exist
Confirm: Show user the case details and ask for confirmation
Delete: Call delete_case(project_id, case_id)
Verify: Call get_project(project_id) to confirm deletion
Report: Show deletion result

User: "Delete test case 'login-test' from project 'auth-module'"

Agent:
1. Get project details to find the case
2. Confirm: "Delete test case 'login-test' (ID: xxx)?"
3. delete_case(project_id="auth-module", case_id="xxx")
4. Report: "✅ Test case 'login-test' deleted successfully"

Delete Project Workflow

When user asks to delete a project:

Warn: Show warning that this will delete ALL test cases in the project
List: Call get_project(project_id) to show what will be deleted
Confirm: Ask user to confirm the deletion
Delete: Call delete_project(project_id)
Report: Show deletion result

User: "Delete project 'legacy-tests'"

Agent:
1. Get project details
2. Warn: "⚠️ This will permanently delete project 'legacy-tests' with 12 test cases"
3. Confirm: "Are you sure? (yes/no)"
4. delete_project(project_id="legacy-tests")
5. Report: "✅ Project 'legacy-tests' and all its test cases deleted"

Cleanup Old Executions Workflow

When user wants to clean up old test data:

List: Call list_projects to show all projects
Identify: Ask user which projects/test cases to delete
Delete: Follow delete workflow above
Report: Summary of cleanup actions

Template Workflow - List and Use Templates

When user wants to use a template:

List: Call list_templates to show available templates
Filter (optional): Use category, target_type, or tag parameters
Get Details: Call get_template(template_id) to see variable requirements
Collect Variables: Ask user for variable values
Instantiate: Call instantiate_template with variables
Report: Show created test case

User: "Use the API health check template"

Agent:
1. list_templates(category="api")
2. get_template(template_id="api-health-check")
3. "Template requires: base_url (default: http://localhost:8000)"
4. instantiate_template(
     template_id="api-health-check",
     project_id="my-project",
     case_name="My API Health Test",
     variables={"base_url": "http://localhost:3000"}
   )
5. Report: "✅ Test case created from template"

Template Workflow - Create Template from Case

When user wants to save a test case as a template:

Get Project: Call get_project(project_id) to find the case
Identify Variables: Ask user which values should become variables
Create: Call create_template_from_case with variable mappings
Report: Show created template

User: "Save 'login-test' as a template"

Agent:
1. get_project(project_id="auth-module")
2. "Which values should become variables?"
   - User: "http://localhost:3000 -> base_url"
   - User: "test@example.com -> email"
3. create_template_from_case(
     project_id="auth-module",
     case_id="login-test",
     template_name="User Login Template",
     variable_mappings={
       "http://localhost:3000": "base_url",
       "test@example.com": "email"
     }
   )
4. Report: "✅ Template created with 2 variables"

Template Workflow - Create New Template

When user wants to create a template from scratch:

Define: Ask for template name, description, category
Create DSL: Create DSL with ${variable} placeholders
Create: Call create_template with the DSL
Report: Show created template

User: "Create a template for database backup tests"

Agent:
1. "Template name? Category?"
2. Create DSL with variables like ${db_name}, ${backup_path}
3. create_template(
     name="Database Backup Test",
     dsl_template="version:... steps:...",
     category="database"
   )
4. Report: "✅ Template created with 2 variables"

Template DSL Format

Templates use ${variable_name} placeholders that get replaced when instantiated.

version: "1.0"
meta:
  name: "${test_name}"
  description: "API test for ${endpoint}"

target:
  type: api
  base_url: "${base_url}"

steps:
  - name: "Call ${endpoint}"
    action: exec
    method: GET
    path: "${endpoint}"

  - name: "Verify status"
    action: assert
    assertion:
      type: status_code
      expected: ${expected_status}

When instantiating, provide variable values:

instantiate_template(
    template_id="...",
    project_id="...",
    case_name="My Test",
    variables={
        "test_name": "User API Test",
        "endpoint": "/api/users",
        "base_url": "http://localhost:8000",
        "expected_status": 200
    }
)

Common Mistakes to Avoid

❌ WRONG - assert with direct type/value:

- action: assert
  type: stdout_contains      # ❌ Wrong!
  value: "text"              # ❌ Wrong!

✅ CORRECT - assert with assertion object:

- action: assert
  assertion:                 # ✅ Correct!
    type: stdout_contains
    expected: "text"         # ✅ Use 'expected', not 'value'!

❌ WRONG - Missing name field:

- action: exec
  command: "echo test"

✅ CORRECT - With name:

- name: "Execute test command"  # ✅ Always add name
  action: exec
  command: "echo test"