Portman API Testing Skill

Overview

Portman converts OpenAPI 3.x specifications into Postman collections with auto-generated contract tests, variation tests, content tests, and integration tests. It runs tests via Newman (Postman's CLI runner) and integrates into CI/CD pipelines.

Installation

npm install -g @apideck/portman

Or use without installing:

npx @apideck/portman -l your-openapi-spec.yaml

IMPORTANT RULES

ALWAYS use a portman-config.json (or .yaml) for test configuration. Do not rely solely on defaults for production use.
ALWAYS target operations using openApiOperationId or openApiOperation (method::path) syntax.
USE --baseUrl to override the spec's server URL when testing against local/staging environments.
USE --envFile to inject environment variables. Variables prefixed with PORTMAN_ auto-map to Postman collection variables.
USE assignVariables to chain request/response values across operations (e.g., capture id from create, use in get/update/delete).
DO NOT hardcode secrets in portman-config. Use environment variables and .env files.

Quick Start

# Generate collection from local spec
portman -l ./openapi.yaml

# Generate and run tests against live API
portman -l ./openapi.yaml -b https://api.example.com -n true

# With custom config
portman -l ./openapi.yaml -c ./portman-config.json -b https://api.example.com -n true

Configuration File

Create portman-config.json (or .yaml):

{
  "version": 1.0,
  "tests": {
    "contractTests": [],
    "contentTests": [],
    "variationTests": [],
    "integrationTests": [],
    "extendTests": []
  },
  "assignVariables": [],
  "overwrites": [],
  "globals": {}
}

JSON Schema: https://raw.githubusercontent.com/apideck-libraries/portman/main/src/utils/portman-config-schema.json

Targeting Operations

All test and overwrite sections use the same targeting system:

// By operationId
{ "openApiOperationId": "leadsAdd" }

// By multiple operationIds
{ "openApiOperationIds": ["leadsAdd", "leadsAll"] }

// By method::path (supports wildcards)
{ "openApiOperation": "GET::/crm/leads" }
{ "openApiOperation": "*::/crm/*" }
{ "openApiOperation": "POST::/*" }

// Exclude specific operations
{ "openApiOperation": "*::/crm/*", "excludeForOperations": ["leadsDelete"] }

Contract Tests

Validate API responses conform to the OpenAPI spec:

{
  "tests": {
    "contractTests": [
      {
        "openApiOperation": "*::/*",
        "statusSuccess": { "enabled": true },
        "contentType": { "enabled": true },
        "jsonBody": { "enabled": true },
        "schemaValidation": { "enabled": true },
        "headersPresent": { "enabled": true }
      },
      {
        "openApiOperation": "*::/*",
        "responseTime": { "enabled": true, "maxMs": 300 }
      }
    ]
  }
}

| Test | Description | |------|-------------| | statusSuccess | Response returns 2xx | | statusCode | Response returns specific HTTP code | | contentType | Content-Type matches spec | | jsonBody | Body is valid JSON matching spec | | schemaValidation | Body validates against JSON schema | | headersPresent | Required headers are present | | responseTime | Response within maxMs milliseconds |

Content Tests

Validate specific response values:

{
  "tests": {
    "contentTests": [
      {
        "openApiOperationId": "leadsAll",
        "responseBodyTests": [
          { "key": "status_code", "value": 200 },
          { "key": "data[0].id", "assert": "not.to.be.null" },
          { "key": "data", "minLength": 1 },
          { "key": "resource", "oneOf": ["leads", "contacts"] }
        ],
        "responseHeaderTests": [
          { "key": "content-type", "contains": "application/json" }
        ]
      }
    ]
  }
}

Content test assertions: value (exact), contains (substring), oneOf, length, minLength, maxLength, notExist, assert (Postman assertion string).

Variation Tests

Test alternative scenarios (errors, edge cases, unauthorized access):

{
  "tests": {
    "variationTests": [
      {
        "openApiOperation": "*::/crm/*",
        "openApiResponse": "401",
        "variations": [
          {
            "name": "Unauthorized",
            "overwrites": [
              { "overwriteRequestSecurity": { "bearer": { "token": "invalid" } } }
            ],
            "tests": {
              "contractTests": [{ "statusCode": { "enabled": true } }]
            }
          }
        ]
      },
      {
        "openApiOperationId": "leadsAdd",
        "openApiResponse": "400",
        "variations": [
          {
            "name": "MissingRequiredFields",
            "overwrites": [
              { "overwriteRequestBody": [{ "key": "name", "value": "", "overwrite": true }] }
            ],
            "tests": {
              "contractTests": [
                { "statusCode": { "enabled": true } },
                { "schemaValidation": { "enabled": true } }
              ]
            }
          }
        ]
      }
    ]
  }
}

Fuzz Testing

Auto-generate invalid values based on schema constraints:

{
  "tests": {
    "variationTests": [
      {
        "openApiOperation": "*::/crm/*",
        "openApiResponse": "422",
        "variations": [
          {
            "name": "FuzzTest",
            "fuzzing": [
              {
                "requestBody": [
                  {
                    "requiredFields": { "enabled": true },
                    "minimumNumberFields": { "enabled": true },
                    "maximumNumberFields": { "enabled": true },
                    "minLengthFields": { "enabled": true },
                    "maxLengthFields": { "enabled": true }
                  }
                ]
              }
            ],
            "tests": {
              "contractTests": [{ "statusCode": { "enabled": true } }]
            }
          }
        ]
      }
    ]
  }
}

Fuzzing targets: requestBody, requestQueryParams, requestHeaders.

Integration Tests

Group operations into end-to-end workflows:

{
  "tests": {
    "integrationTests": [
      {
        "name": "Lead Lifecycle",
        "operations": [
          { "openApiOperationId": "leadsAdd" },
          { "openApiOperationId": "leadsOne" },
          { "openApiOperationId": "leadsUpdate" },
          { "openApiOperationId": "leadsDelete" }
        ]
      }
    ]
  }
}

Variable Chaining

Capture values from responses to use in subsequent requests:

{
  "assignVariables": [
    {
      "openApiOperationId": "leadsAdd",
      "collectionVariables": [
        { "responseBodyProp": "data.id", "name": "leadId" },
        { "responseHeaderProp": "x-request-id", "name": "requestId" }
      ]
    }
  ]
}

Use captured variables in overwrites: {{leadId}}, {{requestId}}.

Request Overwrites

Modify generated requests:

{
  "overwrites": [
    {
      "openApiOperationId": "leadsAdd",
      "overwriteRequestBody": [
        { "key": "name", "value": "Test Lead {{$randomInt}}", "overwrite": true }
      ],
      "overwriteRequestHeaders": [
        { "key": "x-apideck-consumer-id", "value": "{{consumerId}}", "overwrite": true }
      ]
    },
    {
      "openApiOperation": "DELETE::/crm/leads/{id}",
      "overwriteRequestPathVariables": [
        { "key": "id", "value": "{{leadId}}", "overwrite": true }
      ]
    }
  ]
}

Security overwrites: overwriteRequestSecurity supports bearer, apiKey, basic, oauth2, and remove.

Globals

{
  "globals": {
    "collectionPreRequestScripts": ["pm.collectionVariables.set('timestamp', Date.now());"],
    "securityOverwrites": {
      "bearer": { "token": "{{bearerToken}}" }
    },
    "keyValueReplacements": { "x-apideck-app-id": "{{applicationId}}" },
    "valueReplacements": { "<Bearer Token>": "{{bearerToken}}" },
    "orderOfOperations": ["leadsAdd", "leadsAll", "leadsOne", "leadsUpdate", "leadsDelete"],
    "stripResponseExamples": true,
    "variableCasing": "camelCase"
  }
}

Environment Variables

Variables prefixed with PORTMAN_ in .env are auto-injected as camelCase Postman variables:

PORTMAN_CONSUMER_ID=test_user    → {{consumerId}}
PORTMAN_API_TOKEN=abc123         → {{apiToken}}

CI/CD Integration

Store all options in a CLI options file:

{
  "local": "./specs/crm.yml",
  "baseUrl": "https://staging-api.example.com",
  "output": "./output/crm.postman.json",
  "portmanConfigFile": "./config/portman-config.json",
  "envFile": "./.env",
  "includeTests": true,
  "runNewman": true
}

portman --cliOptionsFile ./portman-cli-options.json

Testing Apideck APIs

# Test CRM API
portman -u https://specs.apideck.com/crm.yml -c ./portman-config.json -b https://unify.apideck.com -n true

# Test Accounting API
portman -u https://specs.apideck.com/accounting.yml -c ./portman-config.json -b https://unify.apideck.com -n true

CLI Reference

| Flag | Description | |------|-------------| | -l, --local | Path to local OpenAPI spec | | -u, --url | URL of remote OpenAPI spec | | -b, --baseUrl | Override base URL | | -o, --output | Output file path | | -c, --portmanConfigFile | Path to portman-config | | -n, --runNewman | Run Newman after generation | | -t, --includeTests | Include test suite (default: true) | | -d, --newmanIterationData | Path to iteration data | | --envFile | Path to .env file | | --syncPostman | Upload to Postman app | | --bundleContractTests | Separate folder for contract tests | | --cliOptionsFile | Path to CLI options file | | --init | Interactive config wizard |