Debugging Pipeline Failures

Overview

Core Principle: Systematic investigation of Konflux CI/CD failures by correlating logs, events, and resource states to identify root causes.

Key Abbreviations:

• PR = PipelineRun
• TR = TaskRun
• SA = ServiceAccount
• PVC = PersistentVolumeClaim

When to Use

Invoke when encountering:

• PipelineRun failures or stuck pipelines
• TaskRun errors with unclear messages
• Build container issues (ImagePullBackOff)
• Resource constraints (OOMKilled, quota exceeded)
• Pipeline timeouts
• Workspace or volume mount failures
• Permission errors

Quick Reference

| Symptom | First Check | Common Cause | |---------|-------------|--------------| | ImagePullBackOff | Pod events, image name | Registry auth, typo, missing image | | TaskRun timeout | Step execution time in logs | Slow operation, network issues | | Pending TaskRun | Resource quotas, node capacity | Quota exceeded, insufficient resources | | Permission denied | ServiceAccount, RBAC | Missing Role/RoleBinding | | Volume mount error | PVC status, workspace config | PVC not bound, wrong access mode | | Exit code 127 | Container logs, command | Command not found, wrong image |

Investigation Phases

Phase 1: Identify Failed Component

PipelineRun Status Check:

kubectl get pipelinerun <pr-name> -n <namespace>
kubectl describe pipelinerun <pr-name> -n <namespace>

Look for:

• Overall status (Succeeded/Failed/Running)
• Conditions and reasons
• Which TaskRun(s) failed
• Duration and timestamps

TaskRun Identification:

kubectl get taskruns -l tekton.dev/pipelineRun=<pr-name> -n <namespace>

Identify failed TaskRuns by status.

Phase 2: Log Analysis

Get TaskRun Pod Logs:

# Find the pod
kubectl get pods -l tekton.dev/taskRun=<tr-name> -n <namespace>

# Get logs from specific step
kubectl logs <pod-name> -c step-<step-name> -n <namespace>

# Get logs from all containers
kubectl logs <pod-name> --all-containers=true -n <namespace>

# For previous failures
kubectl logs <pod-name> -c step-<step-name> --previous -n <namespace>

What to Look For:

• Error messages (search for "error", "failed", "fatal")
• Exit codes
• Last successful operation before failure
• Timeout indicators
• Resource exhaustion messages

Phase 3: Event Correlation

Check Kubernetes Events:

kubectl get events -n <namespace> --sort-by='.lastTimestamp'

# Filter for specific resource
kubectl get events --field-selector involvedObject.name=<pod-name> -n <namespace>

Critical Events:

• FailedScheduling → Resource constraints
• FailedMount → Volume/PVC issues
• ImagePullBackOff → Registry/image problems
• Evicted → Resource pressure

Phase 4: Resource Inspection

PipelineRun Details:

kubectl get pipelinerun <pr-name> -n <namespace> -o yaml

Check:

• Parameters passed correctly
• Workspace configurations
• ServiceAccount specified
• Timeout values

TaskRun Details:

kubectl get taskrun <tr-name> -n <namespace> -o yaml

Examine:

• Step definitions and images
• Resource requests/limits
• Status.steps for individual step states
• Conditions for failure reasons

Pod Inspection:

kubectl describe pod <pod-name> -n <namespace>

Look for:

• Container states and exit codes
• Resource requests vs limits
• Volume mounts
• Node placement

Phase 5: Root Cause Analysis

Correlate Findings:

1. Timeline: When did failure occur?
2. First failure: Which step/component failed first?
3. Error pattern: Consistent or intermittent?
4. Recent changes: New code, config, images?

Distinguish Symptom from Cause:

• ❌ "Build failed" (symptom)
• ✓ "npm install timed out due to registry being unavailable" (root cause)

Common Failure Patterns

1. Image Pull Failures

Symptoms: ImagePullBackOff, ErrImagePull

Investigation:

kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Events"

Check:

• Image name and tag spelling
• Image exists in registry
• ServiceAccount has imagePullSecrets
• Registry is accessible

Common Fixes:

• Correct image name/tag
• Add imagePullSecret to ServiceAccount
• Verify registry credentials
• Check network policies

2. Resource Exhaustion

Symptoms: OOMKilled, Pending pods, quota errors

Investigation:

kubectl describe namespace <namespace> | grep -A5 "Resource Quotas"
kubectl top pods -n <namespace>
kubectl describe node | grep -A5 "Allocated resources"

Common Causes:

• Memory limits too low
• Namespace quota exceeded
• No nodes with available resources

Fixes:

• Increase resource limits in Task
• Adjust namespace quotas
• Optimize memory usage in build

3. Build Script Failures

Symptoms: Non-zero exit code, "command not found"

Investigation:

kubectl logs <pod-name> -c step-build -n <namespace>

Check:

• Script syntax errors
• Missing tools in container image
• Wrong working directory
• Environment variables not set

Fixes:

• Fix script errors
• Use image with required tools
• Set correct workingDir in Task
• Pass required params/env vars

4. Timeout Issues

Symptoms: TaskRun shows timeout in status

Investigation:

kubectl get taskrun <tr-name> -n <namespace> -o jsonpath='{.spec.timeout}'
kubectl get taskrun <tr-name> -n <namespace> -o jsonpath='{.status.startTime}{"\n"}{.status.completionTime}'

Common Causes:

• Timeout value too low
• Slow network operations (downloads)
• Build complexity underestimated
• Process hanging

Fixes:

• Increase timeout in Task/PipelineRun
• Use caching for dependencies
• Optimize build process
• Add progress logging to detect hangs

5. Workspace/Volume Issues

Symptoms: CreateContainerError, volume mount failures

Investigation:

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

Check:

• PVC exists and is Bound
• Workspace name matches between Pipeline and PipelineRun
• AccessMode is correct (RWO vs RWX)
• Storage class exists

Fixes:

• Create or fix PVC
• Correct workspace name references
• Use appropriate access mode
• Verify storage provisioner

6. Permission Errors

Symptoms: "Forbidden", "unauthorized", RBAC errors

Investigation:

kubectl get sa <sa-name> -n <namespace>
kubectl get rolebindings -n <namespace>
kubectl auth can-i create pods --as=system:serviceaccount:<namespace>:<sa-name>

Check:

• ServiceAccount exists
• Role/RoleBinding grants needed permissions
• ClusterRole if cross-namespace access needed

Fixes:

• Create ServiceAccount
• Add RoleBinding for required permissions
• Grant pod creation, secret access, etc.

Common Confusions

✗ Incorrect Approach

"Pipeline failed, let me rerun it immediately"

• No root cause identified
• Will likely fail again
• Wastes resources and time

✓ Correct Approach

"Let me check logs and events to understand why it failed, then fix the root cause"

• Identifies actual problem
• Prevents repeat failures
• Efficient resolution

---

✗ Incorrect Approach

"Build timed out. I'll set timeout to 2 hours"

• May hide real issues
• Delays problem detection

✓ Correct Approach

"Let me check what operation is slow in the logs, then optimize or increase timeout if truly needed"

• Identifies slow operations
• Optimizes where possible
• Sets appropriate timeout

---

✗ Incorrect Approach

"Too many logs to read, I'll just try changing something"

• Random changes
• May make it worse
• Doesn't address root cause

✓ Correct Approach

"I'll search logs for error keywords and check the last successful step before failure"

• Focused investigation
• Finds actual error
• Targeted fix

Troubleshooting Workflow

1. GET PIPELINERUN STATUS
   ↓
2. IDENTIFY FAILED TASKRUN(S)
   ↓
3. CHECK POD LOGS (specific step that failed)
   ↓
4. REVIEW EVENTS (timing correlation)
   ↓
5. INSPECT RESOURCE YAML (config issues)
   ↓
6. CORRELATE FINDINGS → IDENTIFY ROOT CAUSE
   ↓
7. APPLY FIX → VERIFY → DOCUMENT

Decision Tree

Q: Is the PipelineRun stuck in "Running"?

• Yes → Check which TaskRuns are pending or running
  • Pending → Resource constraints (Phase 2: Resource Exhaustion)
  • Running too long → Check logs for progress (Phase 4: Timeouts)
• No → PipelineRun Failed → Continue

Q: Which TaskRun failed first?

• Check status of all TaskRuns to find first failure
• Focus investigation on that TaskRun

Q: What does the pod log show?

• Error message → Address specific error
• No output → Check if pod started (events)
• Exit code 127 → Command not found (wrong image)
• Exit code 137 → OOMKilled (increase memory)
• Other exit code → Script/command failure

Q: Do events show image, volume, or scheduling issues?

• ImagePullBackOff → Phase 1: Image Pull Failures
• FailedMount → Phase 5: Workspace/Volume Issues
• FailedScheduling → Phase 2: Resource Exhaustion

Keywords for Search

Konflux pipeline failure, Tekton debugging, PipelineRun failed, TaskRun errors, build failures, CI/CD troubleshooting, ImagePullBackOff, OOMKilled, kubectl logs, pipeline timeout, workspace errors, RBAC permissions