worker-image-investigation

Investigate Taskcluster task failures related to worker images. Compare passing vs failing tasks, extract image versions, find running workers, and debug Windows VMs via Azure CLI. Use when debugging CI failures after image upgrades, investigating intermittent test failures, or comparing worker configurations. Triggers on "image investigation", "worker image", "compare tasks", "why is this failing", "image regression", "worker debug".

Worker Image Investigation

Prerequisites

Install uv and taskcluster; add az for VM access. Follow fx-tests.md for XPCShell or Mochitest history.

Usage

SCRIPTS=~/.claude/skills/worker-image-investigation/scripts
export TASKCLUSTER_ROOT_URL=https://firefox-ci-tc.services.mozilla.com
uv run "$SCRIPTS/triage.py" <FAILING_TASK_ID>
uv run "$SCRIPTS/investigate.py" compare <PASSING_TASK_ID> <FAILING_TASK_ID>

Run investigate.py --help for image, SBOM, pool, and VM commands.

Investigation

Treat the triage.py verdict as a hypothesis; read the task log.
Check history with fx-tests or treeherder.
Compare an equivalent pass and failure (same revision, test, platform, configuration). A version difference is not proof.
Check today's equivalent baseline tasks. Shared failures, including Tier 1, do not block; failures against passing equivalents do. Read validation-gate.md before deciding.
Read azure-commands.md only when guest inspection is needed.

Gotchas

Passing retries can hide earlier harness failures; inspect full task logs.
--skip-treeherder drops history and classification evidence.
Missing fx-tests history means unknown, not new.
Some SBOM files need UTF-16LE decoding.
Debug VM names are at most 15 characters; take image and size from the pool config and delete the VM after use.

Related Skills

taskcluster (tasks), taskcluster-worker-lifecycle-logs (unclaimed tasks), papertrail (guest logs), worker-image-build (new build), bugzilla (confirmed regression).