shield_lock
JinDaGe
← Back to skills
extension
Category: Development & EngineeringAPI key requirement unconfirmed

sync-system-bus

Deploy the system-bus-worker to the joelclaw Kubernetes cluster from local machine. Use when syncing changes in packages/system-bus to k8s, especially because the GitHub Actions deploy job targets a non-existent self-hosted runner and cannot complete deploys automatically.

personAuthor: jakexiaohubgithub
open_in_newRepositorydownloadDownload package

Sync System Bus Worker

Deploy system-bus-worker to the local joelclaw k8s cluster (Talos v1.12.4 / k8s v1.35.0).

Important: .github/workflows/system-bus-worker-deploy.yml has a deploy job on self-hosted. That runner does not exist, so deploys must be completed locally.

Quick Deploy

The publish script handles everything — build, auth, push, k8s apply, rollout, verification:

cd ~/Code/joelhooks/joelclaw
k8s/publish-system-bus-worker.sh

Optional: pass a tag (defaults to timestamp):

k8s/publish-system-bus-worker.sh a6de1e0

GHCR Auth Order

publish-system-bus-worker.sh now authenticates in this order:

  1. GHCR_TOKEN env var (if provided)
  2. secrets lease ghcr_pat (agent-secrets)
  3. gh auth token fallback

If your gh auth token lacks read:packages/write:packages, push will 403. Use ghcr_pat.

What the Script Does

  1. Builds ARM64 Docker image (required — Talos/Colima node is aarch64)
  2. Authenticates to GHCR (prefers agent-secrets lease ghcr_pat; falls back to gh auth token) with temp Docker config
  3. Pushes ghcr.io/joelhooks/system-bus-worker:${TAG} and :latest
  4. Updates the image ref in k8s/system-bus-worker.yaml
  5. kubectl apply the manifest
  6. Waits for rollout (--timeout=180s)
  7. Probes the new pod's health endpoint

Post-Deploy Verification

joelclaw refresh                           # Re-register functions with Inngest
joelclaw functions | grep "<new-function>" # Verify new function appears
joelclaw status                            # Full health check
joelclaw runs --count 3                    # Confirm runs are flowing

Restart Safety (ADR-0156)

The worker is stateless between Inngest steps. Each step is a separate HTTP call; Inngest stores step output server-side. This means k8s rolling restarts are safe — Inngest retries the in-flight step against the new pod.

Critical rule: NEVER set retries: 0 on Inngest functions. With retries: 0, a worker restart during step execution kills the run permanently. With retries ≥ 1, Inngest retries and hits the new pod.

Current story-pipeline has retries: 2 specifically to survive the ~1s restart window during deploys.

What happens during deploy

Step executing on old pod → old pod terminates → step fails (SDK unreachable)
→ Inngest retries after backoff → new pod handles retry → step completes

All previously completed steps are memoized. Only the in-flight step reruns.

Long-running steps (codex implement: 5-10 min)

If a deploy kills a codex step mid-execution, the step reruns from scratch on the new pod (5-10 min wasted but not fatal). For time-critical deploys during active loops, check joelclaw loop status first and deploy between stories.

Manual Steps (if script fails)

Build

cd ~/Code/joelhooks/joelclaw
TAG=$(git rev-parse --short HEAD)
IMAGE="ghcr.io/joelhooks/system-bus-worker:${TAG}"
docker build --platform linux/arm64 -t "$IMAGE" -t ghcr.io/joelhooks/system-bus-worker:latest -f packages/system-bus/Dockerfile .

Push

gh auth token | docker login ghcr.io -u $(gh api user -q .login) --password-stdin
docker push "$IMAGE"
docker push ghcr.io/joelhooks/system-bus-worker:latest

Deploy

kubectl -n joelclaw set image deployment/system-bus-worker system-bus-worker="$IMAGE"
kubectl -n joelclaw rollout status deployment/system-bus-worker --timeout=180s

Verify

joelclaw refresh
joelclaw status

Receipt

Verify the rollout and keep the image/digest plus command output as the deploy receipt. Runtime deploy telemetry belongs in canonical OTel; durable behavior/config changes belong in the relevant Brain .svx. The former slog journal is retired.

Talon Rebuild (Adding Secrets / Changing Worker Supervision)

Talon is a Rust binary that supervises the worker process. It leases secrets from agent-secrets and injects them as env vars. When adding new webhook secrets or changing supervision behavior:

# 1. Add secret to agent-secrets
secrets add my_new_secret --value "the-secret-value"

# 2. Update Talon source — add mapping to SECRET_MAPPINGS array
#    File: ~/Code/joelhooks/joelclaw/infra/talon/src/worker.rs
#    ("my_new_secret", "MY_NEW_SECRET_ENV_VAR"),

# 3. Recompile (fast — ~3s incremental)
export PATH="$HOME/.cargo/bin:$PATH"
cd ~/Code/joelhooks/joelclaw/infra/talon
cargo build --release

# 4. Install + re-sign (macOS kills unsigned binaries)
cp target/release/talon ~/.local/bin/talon
codesign -fs - ~/.local/bin/talon

# 5. Restart via launchd
launchctl bootout gui/$(id -u)/com.joel.talon
sleep 1
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.joel.talon.plist
sleep 12

# 6. Verify
curl -s http://localhost:3111/ | jq '.status'
curl -X PUT http://localhost:3111/api/inngest  # Force function sync

Current SECRET_MAPPINGS (worker.rs)

| Secret Name | Env Var | |------------|---------| | claude_oauth_token | CLAUDE_CODE_OAUTH_TOKEN | | todoist_client_secret | TODOIST_CLIENT_SECRET | | todoist_api_token | TODOIST_API_TOKEN | | front_rules_webhook_secret | FRONT_WEBHOOK_SECRET | | front_api_token | FRONT_API_TOKEN | | vercel_webhook_secret | VERCEL_WEBHOOK_SECRET | | joelclaw_webhook_secret | JOELCLAW_WEBHOOK_SECRET | | revalidation_secret | REVALIDATION_SECRET |

Talon Key Paths

| What | Path | |------|------| | Binary | ~/.local/bin/talon | | Source | ~/Code/joelhooks/joelclaw/infra/talon/src/ | | LaunchAgent plist | ~/Library/LaunchAgents/com.joel.talon.plist | | Logs | ~/.local/log/talon.log / talon.err | | ADR | ~/Vault/docs/decisions/0159-talon-worker-manager.md |

Gotcha: codesign -fs - is required

After cargo build, the binary has adhoc linker-signed signature. macOS launchd may SIGKILL:9 it. Re-signing with codesign -fs - fixes this.

Common Gotchas

| Problem | Cause | Fix | |---------|-------|-----| | exec format error in pod | Built for amd64, not arm64 | Rebuild with --platform linux/arm64 | | GHCR push fails with 403 Forbidden on blob HEAD | gh auth token missing package scopes | Use ghcr_pat via agent-secrets or export GHCR_TOKEN with package scope | | docker-credential-desktop error | Docker config has credsStore | Script uses temp config dir — if manual, remove "credsStore": "desktop" | | Function missing after deploy | Not in the index for the worker that should own it | Add it to index.cluster.ts for cluster-owned callbacks or index.host.ts for host-only work; do not blindly add to both or Inngest will create duplicate runs | | Function still missing | Stale Inngest registration | joelclaw refresh then check again | | "Unable to reach SDK URL" | Worker pod not ready | Wait for rollout, then joelclaw refresh | | Runs stuck after deploy | retries: 0 on the function | Set retries: 2 minimum (ADR-0156) | | Stale app registrations | Multiple apps registered | Delete old registrations in Inngest dashboard (:8289) |

Key Paths

| What | Path | |------|------| | Publish script | k8s/publish-system-bus-worker.sh | | Dockerfile | packages/system-bus/Dockerfile | | k8s manifest | k8s/system-bus-worker.yaml | | Host function index | packages/system-bus/src/inngest/functions/index.host.ts | | Cluster function index | packages/system-bus/src/inngest/functions/index.cluster.ts | | Worker entry | packages/system-bus/src/serve.ts | | GH Actions workflow | .github/workflows/system-bus-worker-deploy.yml | | ADR-0156 | ~/Vault/docs/decisions/0156-graceful-worker-restart.md |