Deploying Kubernetes Services

This skill provides step-by-step guidance for deploying new services to the Superbloom K3s cluster using GitOps.

Capabilities

• Create Flux manifests for new services (namespace, HelmRelease, secrets)
• Configure bjw-s app-template v4 Helm values correctly
• Set up Authelia forward authentication
• Configure Caddy reverse proxy routing
• Add domains to Cloudflare DDNS
• Set up RBAC for sidecars (e.g., Tailscale)
• Encrypt secrets with SOPS

Architecture

Internet → Cloudflare → Caddy (LoadBalancer) → forward_auth (Authelia) → Service → Pod
                              ↓
                         DDNS updates Cloudflare DNS

Required Files

Create in flux/clusters/superbloom/apps/<app-name>/:

ns.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: <app-name>

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ns.yaml
  - rbac.yaml      # if using Tailscale or K8s API access
  - release.yaml
  - secrets.yaml

release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: <app-name>
  namespace: <app-name>
spec:
  interval: 5m
  chart:
    spec:
      chart: app-template
      version: "4.1.1"
      sourceRef:
        kind: HelmRepository
        name: bjw-s
        namespace: flux-system
  valuesFrom:
    - kind: Secret
      name: <app-name>-secrets
      valuesKey: values.yaml

secrets.yaml (SOPS encrypted)

CRITICAL: In app-template v4, serviceAccount goes inside controllers.<name>.serviceAccount, NOT at top level.

apiVersion: v1
kind: Secret
metadata:
  name: <app-name>-secrets
  namespace: <app-name>
type: Opaque
stringData:
  values.yaml: |
    controllers:
      main:
        serviceAccount:
          name: <app-name>    # MUST be inside controllers.main
        containers:
          main:
            image:
              repository: ghcr.io/saavy1/<app-name>
              tag: latest
            env:
              PORT: "3000"
            probes:
              liveness:
                enabled: true
                custom: true
                spec:
                  httpGet:
                    path: /health
                    port: 3000
              readiness:
                enabled: true
                custom: true
                spec:
                  httpGet:
                    path: /health
                    port: 3000
    service:
      main:
        controller: main
        ports:
          http:
            port: 3000

rbac.yaml (for Tailscale sidecar)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: <app-name>
  namespace: <app-name>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <app-name>
  namespace: <app-name>
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <app-name>
  namespace: <app-name>
subjects:
  - kind: ServiceAccount
    name: <app-name>
    namespace: <app-name>
roleRef:
  kind: Role
  name: <app-name>
  apiGroup: rbac.authorization.k8s.io

Caddy Configuration

Add to flux/clusters/superbloom/infra/caddy/caddyfile.yaml:

<app-name>.saavylab.dev {
    forward_auth authelia.authelia.svc.cluster.local:9091 {
        uri /api/authz/forward-auth
        copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
    }
    reverse_proxy <app-name>.<app-name>.svc.cluster.local:3000
}

DDNS Configuration

Add domain to flux/clusters/superbloom/infra/ddns/secrets.yaml DOMAINS field:

DOMAINS: saavylab.dev,mc.saavylab.dev,auth.saavylab.dev,nexus.saavylab.dev,<app-name>.saavylab.dev

SOPS Encryption

cd flux
sops --encrypt --in-place clusters/superbloom/apps/<app-name>/secrets.yaml

Deployment Commands

git add -A && git commit -m "feat: add <app-name> service" && git push
flux reconcile kustomization flux-system --with-source
kubectl rollout restart deployment ddns -n ddns
kubectl rollout restart deployment caddy -n caddy-system

Debugging Commands

kubectl get pods -n <app-name>
kubectl get endpoints <app-name> -n <app-name>
kubectl get helmrelease -n <app-name>
kubectl describe helmrelease <app-name> -n <app-name>
kubectl logs -n <app-name> -l app.kubernetes.io/name=<app-name> --all-containers

Common Issues

| Issue | Cause | Solution | |-------|-------|----------| | Pod 1/2 Ready | Sidecar crashing | Add RBAC, check TS_KUBE_SECRET | | Empty endpoints | Pod not Ready | All containers must pass probes | | Schema validation error | Wrong values structure | serviceAccount inside controllers.main | | 521 from Cloudflare | DNS not updated | Add domain to DDNS | | ACME rate limit | Failed cert attempts | Wait 1 hour |

Tailscale Sidecar

For services needing Tailscale network access, add to containers:

tailscale:
  image:
    repository: ghcr.io/tailscale/tailscale
    tag: latest
  env:
    TS_AUTHKEY: "tskey-auth-..."
    TS_HOSTNAME: <app-name>
    TS_STATE_DIR: /var/lib/tailscale
    TS_USERSPACE: "true"
    TS_KUBE_SECRET: <app-name>-tailscale
  securityContext:
    runAsUser: 0
    runAsGroup: 0

Requires RBAC for secrets management.