Marketplace Packages: Enable Incremental Extraction

This skill guides you through creating the marketplace-packages changes needed to enable incremental extraction for a connector. It follows the established pattern from Oracle's incremental extraction (PR #22199).

When to Use This Skill

• Adding incremental extraction support to a new connector in marketplace-packages
• Modifying the Argo WorkflowTemplate YAML to pass incremental parameters
• Understanding the branching and PR strategy for marketplace-packages changes

When NOT to Use This Skill

• Implementing the app-side incremental extraction code (use implement-incremental-extraction skill)
• Modifying SDK incremental logic
• Creating a brand new connector package from scratch

Branching Strategy

One Branch, Three PRs

Marketplace-packages uses a multi-environment deployment model:

• main (production) → master (preprod staging) → preprod (preprod)

Create ONE branch and open THREE PRs with the same title structure:

# 1. Create branch from master
git checkout master
git pull origin master
git checkout -b <ticket-id>-incremental

# 2. Make changes (see YAML Modifications below)

# 3. Create 3 PRs with consistent title:
# PR 1: <ticket-id> → preprod
gh pr create --base preprod --title "APP-XXXX - [preprod] add incremental extraction support to <connector> interim app - <description>"

# PR 2: <ticket-id> → master (Automatic Master PR)
gh pr create --base master --title "APP-XXXX - [preprod] add incremental extraction support to <connector> interim app - <description> (Automatic Master PR)"

# PR 3: <ticket-id> → main
gh pr create --base main --title "APP-XXXX - [preprod] add incremental extraction support to <connector> interim app - <description>"

Title Convention

APP-XXXX - [preprod] add incremental extraction support to <connector> interim app - <brief description> (Automatic Master PR)

Example (Oracle):

APP-9439 - [preprod] add incremental extraction support to oracle interim app - pass entire current state instead of transformed diff to publish to avoid circuit break failures (Automatic Master PR)

YAML Template Modifications

All changes go in:

packages/atlan/<connector>/templates/atlan-<connector>.yaml

1. Add Workflow Step References

Add the interim extraction steps to the workflow step list:

# In the steps/configmaps section that lists valid workflow steps
"extract",
"offline-extraction",
"extract-secure-agent",
"interim-extract-and-transform",                              # NEW
"interim-extract-and-transform-app-framework-secure-agent-dag", # Existing

2. Add Incremental Parameters

Add these parameters to the connector's parameter block (alongside existing extraction parameters):

# Incremental extraction parameters
- name: incremental-extraction
  valueFrom:
    configMapKeyRef:
      name: atlan-tenant-package-runtime-config
      key: atlan-<connector>.main.params.incremental-extraction
      optional: true
    default: "false"
- name: column-batch-size
  value: "25000"
- name: column-chunk-size
  value: "100000"
- name: system-schema-name
  value: "SYS"  # Database-specific: SYS for Oracle, "" for ClickHouse
# Debug parameters (uncomment for debugging)
# - name: marker-timestamp
#   value: ""

Parameter Details

| Parameter | Source | Default | Purpose | |-----------|--------|---------|---------| | incremental-extraction | ConfigMap (runtime toggle) | "false" | Enable/disable incremental mode | | column-batch-size | Hardcoded | "25000" | Tables per batch for column extraction | | column-chunk-size | Hardcoded | "100000" | Column records per output chunk | | system-schema-name | Hardcoded | DB-specific | System schema for metadata queries | | marker-timestamp | Commented out | "" | Debug: override marker for testing |

3. Update Workflow Arguments Formatting

Change workflow-arguments from folded (>) to literal (|) block scalar for improved YAML readability:

# Before
- name: workflow-arguments
  value: >
    {
      "workflow_id": "{{workflow.labels.workflows.argoproj.io/workflow-template}}"
    }

# After
- name: workflow-arguments
  value: |
    {
      "workflow_id": "{{workflow.labels.workflows.argoproj.io/workflow-template}}"
    }

4. Temporary Publish Workaround

Important: Currently, the publish step does NOT support publishing only changed assets via incremental extraction. Until this is resolved, we need a temporary workaround that passes the full current-state instead of the transformed diff.

# In the publish step parameters
# TEMPORARY WORKAROUND: Pass current-state instead of transformed directory
# Because publish breaks circuit breaker when receiving incremental diffs
#
# The current-state contains ALL assets (including ancestral) so publish
# treats it like a full extraction
#
# TODO: Remove this once publish supports incremental diff publishing
- name: transformed-input-path
  # OLD (standard):
  # value: "artifacts/apps/{{inputs.parameters.application-name}}/workflows/{{tasks.extraction-and-transformation.outputs.parameters.workflow_id}}/{{tasks.extraction-and-transformation.outputs.parameters.run_id}}/transformed"
  # NEW (temporary workaround):
  value: "persistent-artifacts/apps/{{inputs.parameters.application-name}}/connection/{{=sprig.last(sprig.splitList(\"/\", jsonpath(inputs.parameters.connection, '$.attributes.qualifiedName')))}}/current-state"

How the Path Works

Standard path:
  artifacts/apps/oracle/workflows/{workflow_id}/{run_id}/transformed

Temporary workaround path:
  persistent-artifacts/apps/oracle/connection/{connection_epoch}/current-state

The sprig.last(sprig.splitList(...)) extracts the connection epoch ID from the connection's qualified name (e.g., default/oracle/1764230875 → 1764230875).

Checklist

Before submitting PRs:

• [ ] Branch created from master
• [ ] interim-extract-and-transform added to workflow steps list
• [ ] Incremental parameters added (incremental-extraction, column-batch-size, etc.)
• [ ] workflow-arguments uses | block scalar
• [ ] Publish step uses current-state path (temporary workaround)
• [ ] TODO comment added to publish workaround explaining it's temporary
• [ ] File ends with newline
• [ ] Three PRs created: preprod, master, main
• [ ] All three PRs have consistent titles
• [ ] Linear ticket linked in PR description

PR Description Template

## Change Summary
This pull request adds support for incremental extraction in the
<Connector> integration template, introduces new configuration parameters
for batch and chunk sizes, and temporarily adjusts the publish logic to
work around current limitations with incremental extraction.

**Incremental Extraction Support and Configuration:**
* Added new parameters to enable incremental extraction, including
  `incremental-extraction`, `column-batch-size`, and `column-chunk-size`,
  with values sourced from a config map or set to defaults.

**Workflow and Step Updates:**
* Added new workflow steps for interim app extraction.

**Temporary Workaround for Publish Logic:**
* Updated `transformed-input-path` to point to `current-state` directory
  instead of `transformed` directory, due to current limitations in
  publishing changed assets via incremental extraction.
* TODO: Remove once publish supports incremental diff publishing.

## Linear Issues Resolved
- APP-XXXX - <link>

Reference PRs

• Oracle: PR #22199
  • Branch: app-9439-incremental
  • File: packages/atlan/oracle/templates/atlan-oracle.yaml
  • Changes: +30 lines, -3 lines