Coverport Integration Skill

This skill automates the integration of coverport into Go repositories for e2e test coverage collection and upload to Codecov.

What is Coverport?

Coverport is a tool that enables e2e test coverage collection by:

Building instrumented container images with Go's -cover flag
Collecting coverage data from running containers during e2e tests
Uploading the coverage data to Codecov with appropriate flags

When to Use This Skill

Use this skill when the user:

Asks to integrate coverport into their repository
Wants to add e2e test coverage tracking
Needs to set up coverage instrumentation for Go projects
Mentions integrating coverage collection for Tekton/Konflux pipelines

Prerequisites

Before using this skill, verify the repository has:

Go codebase with a Dockerfile
Tekton pipelines for CI/CD (typically in .tekton/ directory)
E2E test pipeline (typically in integration-tests/pipelines/)
GitHub Actions workflows (optional but common)
Codecov account

Instructions

Step 0: Pre-Integration Repository Scan

Before starting, run these checks to understand the repository structure:

Find main.go location:

find . -name "main.go" -not -path "*/vendor/*" -not -path "*/test/*"

Check current Dockerfile build command:
```
grep -A5 "go build" Dockerfile
```

List Tekton pipelines:

ls .tekton/*.yaml
ls integration-tests/pipelines/*.yaml 2>/dev/null || echo "No integration-tests/pipelines found"

Check for existing coverage setup:

grep -r "ENABLE_COVERAGE\|instrumented\|coverport" . --exclude-dir=vendor --exclude-dir=.git

This helps identify potential conflicts or existing coverage infrastructure before making changes.

Step 1: Analyze the Repository

Analyze the repository structure to understand what needs to be modified:

Find the Dockerfile - Look for the main Dockerfile
Identify binaries being built - Check what Go binaries are compiled in the Dockerfile and note if main.go is in root or subdirectory
Find Tekton push pipeline - Look in .tekton/ for *-push.yaml
Find E2E test pipeline - Look in integration-tests/pipelines/ for *e2e*.yaml
Find Tekton PR pipeline - Look in .tekton/ for *-pull-request.yaml
Find GitHub Actions - Look in .github/workflows/ for pr.yaml, pr.yml, codecov.yaml, or codecov.yml
Check for existing coverage integration - Search for ENABLE_COVERAGE, instrumented, coverport

Step 2: Ask Clarifying Questions

Before making changes, ask the user:

Which binaries to instrument? - If the Dockerfile builds multiple binaries, ask which ones run during e2e tests
Tenant namespace - Confirm the namespace where their build and integration pipelines run (check .tekton/*-push.yaml for the namespace field)
Secret name - Confirm they want to use coverport-secrets or specify a different name
OCI storage - Confirm where coverage data should be stored (the quay.io repository for test artifacts)

Step 3: Add Coverport as a Go Module Dependency

Add coverport to your Go module dependencies:

go get github.com/konflux-ci/coverport/instrumentation/go

This will:

Add the coverport package to go.mod as a dependency
Update go.sum with the dependency checksums

Step 4: Create coverage_init.go File

Create a new file coverage_init.go in the root of your Go module (same directory as main.go or where the package main is):

//go:build coverage

package main

// This file is only included when building with -tags=coverage.
// It starts a coverage HTTP server that allows collecting coverage data
// from the running binary during E2E tests.

import _ "github.com/konflux-ci/coverport/instrumentation/go" // starts coverage server via init()

Important:

The //go:build coverage tag ensures this file is only included when building with -tags=coverage
The blank import triggers the coverage server's init() function
This file should be at the root of your Go module (where main.go is, or where the main package is)

Step 5: Modify the Dockerfile

Add coverage instrumentation support:

Add build argument (near the top after FROM):

# Build arguments
ARG ENABLE_COVERAGE=false

Modify the build command to conditionally build with coverage tags:

# Build with or without coverage instrumentation
RUN if [ "$ENABLE_COVERAGE" = "true" ]; then \
        echo "Building with coverage instrumentation..."; \
        CGO_ENABLED=0 go build -cover -covermode=atomic -tags=coverage -o <binary-name> .; \
    else \
        echo "Building production binary..."; \
        CGO_ENABLED=0 go build -a -o <binary-name> .; \
    fi

Important:

Replace <binary-name> with the actual binary name
The -tags=coverage flag includes the coverage_init.go file
Build the package (.) rather than individual files
Only instrument binaries that run during e2e tests
Keep other binaries without instrumentation
No need to download external files - coverport is now a Go module dependency

Step 5.5: Validate Dockerfile Changes Locally

IMPORTANT: Before proceeding to pipeline changes, validate the Dockerfile modifications work correctly using podman or docker:

# Build instrumented image
podman build --build-arg ENABLE_COVERAGE=true -t test-instrumented -f Dockerfile .

# Build production image (without coverage)
podman build -t test-production -f Dockerfile .

# Verify both images built successfully
podman images | grep test-

Expected output in instrumented build:

"Building with coverage instrumentation..."

Expected output in production build:

"Building production binary..."

If builds fail:

Stop and fix the Dockerfile before proceeding
See Troubleshooting section for common issues
Ensure coverage_init.go exists in the correct location
Verify Go module dependencies were downloaded (check go.mod and go.sum)
Check that the build tags syntax is correct in coverage_init.go

Why this validation matters:

Catches Dockerfile syntax errors immediately
Verifies coverport Go module integration works
Confirms both production and instrumented builds succeed
Prevents wasting CI/CD pipeline time on broken builds
Validates the conditional build logic works correctly

Step 6: Update Tekton Push Pipeline

Add a task to build an instrumented image in the push pipeline (e.g., .tekton/*-push.yaml):

Find the location after prefetch-dependencies task and add:

- name: build-instrumented-image
  params:
  - name: IMAGE
    value: $(params.output-image).instrumented
  - name: DOCKERFILE
    value: $(params.dockerfile)
  - name: CONTEXT
    value: $(params.path-context)
  - name: HERMETIC
    value: $(params.hermetic)
  - name: PREFETCH_INPUT
    value: $(params.prefetch-input)
  - name: IMAGE_EXPIRES_AFTER
    value: $(params.image-expires-after)
  - name: COMMIT_SHA
    value: $(tasks.clone-repository.results.commit)
  - name: BUILD_ARGS
    value:
    - $(params.build-args[*])
    - ENABLE_COVERAGE=true
  - name: BUILD_ARGS_FILE
    value: $(params.build-args-file)
  - name: SOURCE_ARTIFACT
    value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT)
  - name: CACHI2_ARTIFACT
    value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT)
  runAfter:
  - prefetch-dependencies
  taskRef:
    params:
    - name: name
      value: buildah-oci-ta
    - name: bundle
      value: quay.io/konflux-ci/tekton-catalog/task-buildah-oci-ta:0.7@sha256:b54509f5f695c0c89de4587a403099a26da5cdc3707037edd4b7cf4342b63edd
    - name: kind
      value: task
    resolver: bundles
  when:
  - input: $(tasks.init.results.build)
    operator: in
    values:
    - "true"

IMPORTANT - Key points:

Use buildah-oci-ta (NOT buildah-remote-oci-ta) - this is a regular local build for amd64 testing clusters
This should be a single task, NOT a matrix build (no PLATFORM parameter, no IMAGE_APPEND_PLATFORM)
Image tagged with .instrumented suffix
HERMETIC: $(params.hermetic) - uses the same hermetic setting as the main build (now supports hermetic builds!)
PREFETCH_INPUT: $(params.prefetch-input) - uses the same prefetch settings as the main build
BUILD_ARGS includes ENABLE_COVERAGE=true
Do NOT add a build-instrumented-image-index task - the instrumented image is single-platform only

Step 5: Update E2E Test Pipeline

Make three changes to the e2e test pipeline:

A. Update test-metadata task from v0.3 to v0.4:

- name: test-metadata
  taskRef:
    resolver: git
    params:
      - name: url
        value: https://github.com/konflux-ci/tekton-integration-catalog.git
      - name: revision
        value: main
      - name: pathInRepo
        value: tasks/test-metadata/0.4/test-metadata.yaml

B. Update image references (if applicable):

NOTE: Only modify this if your e2e tests actually run the containerized application.

If your tests build the manager from source (e.g., using make build or go run main.go), you may need to modify the build/run commands to use coverage flags instead, or deploy the instrumented container image
If your tests deploy and run containers, proceed with updating image references

For tests that deploy/run container images, find parameters that reference images and change:

container-repo → instrumented-container-repo
container-tag → instrumented-container-tag
container-image → instrumented-container-image

Example scenarios:

Scenario 1 (uses container): Tests deploy the app to a cluster using the container image → Update image references
Scenario 2 (builds from source): Tests run make build && ./manager inside the pipeline → May not need image reference changes, but need to ensure the running process is instrumented
Scenario 3 (hybrid): Tests build from source but coverage collection expects instrumented container → Coordinate with user on approach

C. Add coverage collection task after e2e tests:

- name: collect-and-upload-coverage
  runAfter:
    - <e2e-test-task-name>  # Replace with actual task name
  params:
    - name: instrumented-images
      value: "$(tasks.test-metadata.results.instrumented-container-repo):$(tasks.test-metadata.results.instrumented-container-tag)"
    - name: cluster-access-secret-name
      value: kfg-$(context.pipelineRun.name)  # Adjust if different
    - name: test-name
      value: e2e-tests
    - name: oci-container
      value: "$(params.oci-container-repo):$(context.pipelineRun.name)"
    - name: codecov-flags
      value: e2e-tests
    - name: credentials-secret-name
      value: "coverport-secrets"  # Or user-specified name
  taskRef:
    resolver: git
    params:
      - name: url
        value: https://github.com/konflux-ci/tekton-integration-catalog.git
      - name: revision
        value: main
      - name: pathInRepo
        value: tasks/coverport-coverage/0.1/coverport-coverage.yaml

Step 8: Update Tekton PR Pipeline (Pull Request Pipeline)

Update the PR pipeline (e.g., .tekton/*-pull-request.yaml) to build with coverage instrumentation:

A. Enable hermetic build and prefetch (if not already enabled):

Add or ensure these parameters exist in the spec.params section:

  - name: hermetic
    value: "true"
  - name: prefetch-input
    value: '{"type": "gomod", "path": "."}'

B. Add ENABLE_COVERAGE=true to BUILD_ARGS:

Find the build-images task (or equivalent) and add ENABLE_COVERAGE=true to its BUILD_ARGS:

- name: build-images
  # ... other params ...
  params:
  # ... other params ...
  - name: BUILD_ARGS
    value:
    - $(params.build-args[*])
    - ENABLE_COVERAGE=true  # Add this line
  # ... rest of the task ...

Key points:

With the Go module approach, hermetic builds are now supported!
Enable hermetic: "true" and prefetch-input for secure, reproducible builds
Add ENABLE_COVERAGE=true to the regular build task in PR pipeline
This enables coverage collection for PR builds which can be used for PR-level testing
No need to create a separate instrumented image task in PR pipeline - just modify the existing build task

Step 7: Update GitHub Actions

Add codecov flags to distinguish unit tests from e2e tests.

In .github/workflows/pr.yaml (or similar), update the codecov upload step:

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v5
  with:
    flags: unit-tests

If there's a separate codecov.yml workflow, add the same flags there with the token:

- name: Codecov
  uses: codecov/codecov-action@v5
  with:
    token: ${{ secrets.CODECOV_TOKEN }}
    flags: unit-tests

Step 8: Document Manual Steps

After making all changes, inform the user they need to create a Kubernetes secret.

IMPORTANT: The secret must be created in the namespace where your build and integration pipelines run. This is typically your tenant namespace (e.g., my-tenant, not a specific repository namespace like rhtap-release-2-tenant). You can identify the correct namespace by checking the namespace field in your .tekton/*-push.yaml file.

Option A - Using kubectl:

# First, create the dockerconfig JSON file
cat > /tmp/dockerconfig.json <<EOF
{"auths":{"quay.io":{"auth":"<base64-encoded-quay-user:token>","email":""}}}
EOF

# Create the secret with both keys in YOUR tenant namespace
kubectl create secret generic coverport-secrets \
  --from-literal=codecov-token=<your-codecov-token> \
  --from-file=oci-storage-dockerconfigjson=/tmp/dockerconfig.json \
  -n <your-tenant-namespace>

# Clean up
rm /tmp/dockerconfig.json

Option B - Using YAML:

apiVersion: v1
kind: Secret
metadata:
  name: coverport-secrets
  namespace: <your-tenant-namespace>  # Replace with your tenant namespace
type: Opaque
stringData:
  codecov-token: <your-codecov-token>
  oci-storage-dockerconfigjson: '{"auths":{"quay.io":{"auth":"<base64-encoded-quay-user:token>","email":""}}}'

Required secret keys:

codecov-token - Your Codecov API token for uploading coverage reports to Codecov
oci-storage-dockerconfigjson - Docker config JSON with Quay.io credentials for pushing coverage test artifacts to an OCI container registry
- This is used by the collect-and-upload-coverage task to store coverage data as OCI artifacts in quay.io
- The coverage collection process extracts coverage data from instrumented containers and pushes it to the OCI registry before uploading to Codecov
- The auth value should be base64-encoded username:token
- To encode: echo -n "quay-username:quay-token" | base64
- You need push access to the quay.io repository specified in the e2e pipeline's oci-container-repo parameter

Step 9: Post-Integration Validation Checklist

Before committing the changes, verify all modifications are correct:

Local validation (already completed in Step 5.5):

[ ] podman build (production) succeeds
[ ] podman build --build-arg ENABLE_COVERAGE=true (instrumented) succeeds
[ ] Instrumented build logs show "Building with coverage instrumentation..."
[ ] Production build logs show "Building production binary..."

Go module setup checklist:

[ ] go.mod has coverport dependency added
[ ] go.sum has coverport checksums
[ ] coverage_init.go exists at the root of the module with correct build tags

File modifications checklist:

[ ] Dockerfile has ENABLE_COVERAGE build arg (removed COVERAGE_SERVER_URL)
[ ] Dockerfile has conditional build logic with -tags=coverage flag
[ ] .tekton/*-push.yaml has build-instrumented-image task after prefetch-dependencies
[ ] .tekton/*-push.yaml instrumented task uses buildah-oci-ta (not buildah-remote-oci-ta)
[ ] .tekton/*-push.yaml instrumented task uses HERMETIC: $(params.hermetic) and PREFETCH_INPUT: $(params.prefetch-input)
[ ] .tekton/*-pull-request.yaml has hermetic: "true" and prefetch-input enabled
[ ] .tekton/*-pull-request.yaml has ENABLE_COVERAGE=true in BUILD_ARGS
[ ] integration-tests/pipelines/*e2e*.yaml uses test-metadata v0.4
[ ] integration-tests/pipelines/*e2e*.yaml has collect-and-upload-coverage task
[ ] integration-tests/pipelines/*e2e*.yaml updated image references (if applicable)
[ ] .github/workflows/pr.y*ml has flags: unit-tests in codecov action
[ ] .github/workflows/codecov.y*ml has flags: unit-tests in codecov action

Documentation provided to user:

[ ] Instructions for creating coverport-secrets Kubernetes secret in their tenant namespace
[ ] Explanation that the namespace should be where their build and integration pipelines run
[ ] Required secret keys: codecov-token and oci-storage-dockerconfigjson
[ ] Explanation of what oci-storage-dockerconfigjson is used for (pushing coverage artifacts to quay.io)
[ ] Instructions for encoding auth credentials
[ ] Note about needing push access to the quay.io repository

Summary to provide user: List all modified files with brief description of changes:

Modified files:
- coverage_init.go: NEW - Coverage initialization with build tags
- go.mod: Added coverport dependency
- go.sum: Added coverport checksums
- Dockerfile: Added coverage instrumentation with build tags
- .tekton/<name>-push.yaml: Added instrumented image build task with hermetic support
- .tekton/<name>-pull-request.yaml: Enabled coverage and hermetic builds for PR builds
- integration-tests/pipelines/<name>-e2e-pipeline.yaml: Added coverage collection
- .github/workflows/pr.yml: Added unit-tests flag
- .github/workflows/codecov.yml: Added unit-tests flag

Validation

After integration is deployed to CI/CD, provide these verification steps to the user:

Check instrumented image build:
- Push a commit to main branch
- Verify the push pipeline creates an image with .instrumented tag
- Check build logs for "Building with coverage instrumentation..." message
Check e2e coverage collection:
- Run e2e tests
- Verify collect-and-upload-coverage task executes successfully
- Check Codecov dashboard for coverage data with e2e-tests flag
Check unit test coverage:
- Create a PR
- Verify unit tests upload coverage with unit-tests flag
- Check Codecov shows both unit and e2e coverage

Troubleshooting

Common issues and solutions:

Build error: "coverage_init.go not found" or "package not imported"

Cause: The coverage_init.go file is missing or in the wrong location
Solution:
- Ensure coverage_init.go exists at the root of your Go module (same directory as main.go)
- Verify the file has the correct //go:build coverage build tag
- Check that the package declaration matches your main package (package main)

Build error: "cannot find package"

Cause: Coverport dependency not properly added to Go modules
Solution:
- Run go get github.com/konflux-ci/coverport/instrumentation/go
- Verify go.mod has the coverport dependency
- Run go mod tidy to clean up dependencies

Instrumented build fails:

Verify coverport Go module dependency is in go.mod and go.sum
Check that coverage_init.go has the correct build tag syntax (//go:build coverage, not // +build coverage)
Ensure the Dockerfile build command includes -tags=coverage
Verify hermetic mode is enabled with proper prefetch configuration
Ensure you're using buildah-oci-ta for instrumented builds in push pipeline, not buildah-remote-oci-ta
Verify there's no matrix build or PLATFORM parameter for the instrumented image task

Hermetic build fails with "cannot download dependencies"

Cause: Go module dependencies not properly prefetched
Solution:
- Ensure hermetic: "true" is set in the pipeline parameters
- Verify prefetch-input is set correctly: {"type": "gomod", "path": "."}
- Check that the prefetch-dependencies task completed successfully
- Review prefetch task logs for any download errors

Coverage data not uploaded:

Verify coverport-secrets exists in your tenant namespace (the namespace where your build and integration pipelines run)
Check codecov-token key exists in the secret
Check oci-storage-dockerconfigjson key exists and is valid (should be a valid Docker config JSON)
Verify you have push access to the quay.io repository specified in the e2e pipeline's oci-container-repo parameter
Review collect-and-upload-coverage task logs for errors related to OCI push or Codecov upload

Coverage data incomplete:

Verify e2e tests are using the instrumented image (check image tag has .instrumented suffix)
Ensure coverage server is properly included in the build (check that -tags=coverage is used)
Check that the correct binaries are instrumented
Verify coverage_init.go exists and has the correct import

E2E tests pass but no coverage data collected:

Verify the e2e tests are actually running the instrumented binary/container
If tests build from source (e.g., make build), the build process must include -cover -tags=coverage flags
Check that GOCOVERDIR environment variable is set in the running container/process
Verify the coverage collection task can access the cluster where instrumented app runs
Review coverage collection task logs for connection or permission errors

Production build includes coverage code:

Cause: Missing or incorrect build tags
Solution:
- Verify coverage_init.go has //go:build coverage at the top
- Ensure production builds do NOT include -tags=coverage
- The coverage code should only be included when ENABLE_COVERAGE=true

Best Practices

Validate early and often - Always run podman/docker builds after Dockerfile changes, before modifying pipelines
Be adaptive - Repository structures vary, adapt the integration to the specific repository
Ask questions - If unsure about something, ask the user for clarification
Show diffs - When modifying files, explain what's changing
Preserve existing logic - Don't break existing functionality
Handle edge cases - Check for existing build args, multiple Dockerfiles, etc.
Provide context - Explain why each change is needed
Use checklists - Go through the post-integration checklist before completing
Test both paths - Ensure both production and instrumented builds work

Reference Implementation

The reference implementation can be found in the release-service repository:

Initial implementation (wget approach): commits 1b2208f..dbf965d
Updated implementation (Go module approach): commit 5ed6752 - "fix: use coverport as go module"

Key files modified in the Go module approach:

coverage_init.go - NEW: Coverage initialization file with build tags
go.mod - Added coverport dependency
go.sum - Added coverport checksums
Dockerfile - Updated to use -tags=coverage instead of downloading files
.tekton/release-service-push.yaml - Updated to support hermetic builds for instrumented images
.tekton/release-service-pull-request.yaml - Enabled hermetic builds and prefetch
integration-tests/pipelines/konflux-e2e-tests-pipeline.yaml - Updated to use test-metadata v0.4 and added coverage collection
.github/workflows/codecov.yml and .github/workflows/pr.yml - Added codecov flags

Key changes in Go module approach:

No more wget to download coverage_server.go during build
Coverport is a proper Go module dependency
Hermetic builds are now supported
Cleaner separation using Go build tags

Note: The release-service repository uses the rhtap-release-2-tenant namespace, which is specific to that repository. When implementing coverport for other repositories, use the appropriate tenant namespace where that repository's build and integration pipelines run.

Examples

Example 1: Single Binary Repository (main.go in root)

For a repository that builds one binary (manager) where main.go is in the root directory:

Step 1: Add Go module dependency

go get github.com/konflux-ci/coverport/instrumentation/go

Step 2: Create coverage_init.go at the root

//go:build coverage

package main

import _ "github.com/konflux-ci/coverport/instrumentation/go"

Step 3: Update Dockerfile

# Before
RUN CGO_ENABLED=0 go build -a -o manager main.go

# After
ARG ENABLE_COVERAGE=false

RUN if [ "$ENABLE_COVERAGE" = "true" ]; then \
        CGO_ENABLED=0 go build -cover -covermode=atomic -tags=coverage -o manager .; \
    else \
        CGO_ENABLED=0 go build -a -o manager .; \
    fi

Example 2: Multiple Binaries

For a repository that builds manager and snapshotgc, where only manager needs coverage:

Steps 1-2: Same as Example 1 (add Go module, create coverage_init.go)