Deploy What Changed: Nx affected + Cloud Run + Workload Identity Federation

A merge to main should ship the services that merge touched — and nothing else. Here is the whole pipeline: keyless auth, one deploy target per app, and the graph work that makes "affected" trustworthy. Stack: Nx monorepo · Python + Node · Cloud Run. Auth: Workload Identity Federation. Secrets in repo: zero. 00. The shape One command, two callers Most broken deploy workflows are broken the same way: the YAML contains a second, drifting copy of the deploy commands. Someone tunes memory limits locally, ships it by hand, and CI keeps deploying the old shape for a month before anyone notices. So the rule that makes everything else work: the deploy command lives in the repo, next to the app it deploys. CI does not know how to deploy anything. It knows how to ask. In an Nx workspace that means a deploy target per app: apps/web/project.json { "targets": { "deploy": { "executor": "nx:run-commands", "dependsOn": ["build"], "options": { "command": "docker build --platform linux/amd64 -f apps/web/Dockerfile -t gcr.io/$PROJECT/web:latest . && docker push gcr.io/$PROJECT/web:latest && gcloud run deploy web --image gcr.io/$PROJECT/web:latest --region us-central1 --allow-unauthenticated" } } } } Now a human types nx deploy web and CI runs nx affected -t deploy. Same command, same flags, same memory limits, no drift. The rest of this post is the two hard parts: proving to Google who GitHub is, and proving to Nx what actually changed. New project? Create the registry first. Container Registry stopped accepting writes on March 18, 2025 — gcr.io hostnames now proxy to Artifact Registry, but only for repositories that already exist. An established project usually has that mirror already; a brand-new one doesn't, and docker push fails with a 404 the first time. Create it once: gcloud artifacts repositories create gcr.io --repository-format=docker --location=us --project=$PROJECT. New projects are better off skipping the mirror entirely and pushing to us-docker.pkg.dev/$PROJECT/gcr.io/web:latest instead. 01. Identity Stop putting service account keys in GitHub The tutorial answer is gcloud iam service-accounts keys create, paste the JSON into a repo secret, done. That key is a permanent credential with deploy rights, sitting in a place many people can read, that nothing rotates and nobody revokes. Workload Identity Federation replaces it. GitHub already signs a short-lived OpenID Connect (OIDC) token for every workflow run, describing the repo, the branch, the workflow. You teach Google to trust that issuer, then narrow the trust to exactly one repository. No key exists, so no key can leak. Before the gcloud commands, here's the whole exchange as a diagram — step through it to see where a token from the wrong repository actually gets rejected: Interactive diagram: the full GitHub to Google token exchange is a step-through diagram on the original post - see it here. In short: GitHub Actions requests an OIDC token, GitHub signs it, Google's Workload Identity Pool verifies the issuer and repository (rejecting any repo that does not match), then exchanges it for short-lived credentials scoped to the deploy service account, and those credentials deploy to Cloud Run. No long-lived key exists at any step. Create the deploy identity # The service account CI will impersonate. It has no key. gcloud iam service-accounts create github-deploy \ --display-name="GitHub Actions deploy" --project=$PROJECT SA=github-deploy@$PROJECT.iam.gserviceaccount.com # Deploy revisions, push images, and act as the runtime SA. for R in roles/run.admin \ roles/artifactregistry.writer \ roles/storage.admin \ roles/iam.serviceAccountUser \ roles/secretmanager.viewer; do gcloud projects add-iam-policy-binding $PROJECT \ --member="serviceAccount:$SA" --role=$R --condition=None done Why serviceAccountUser. Deploying a Cloud Run service that runs as another service account is an impersonation. Without roles/iam.serviceAccountUser the deploy fails at the very last step with a permission error that names the runtime account, not the deployer — an easy twenty minutes to lose. Trust GitHub, and only your repo gcloud iam workload-identity-pools create github \ --location=global --project=$PROJECT gcloud iam workload-identity-pools providers create-oidc github-provider \ --location=global --workload-identity-pool=github --project=$PROJECT \ --issuer-uri="https://token.actions.githubusercontent.com" \ --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository,attribute.repository_owner=assertion.repository_owner" \ --attribute-condition="assertion.repository=='my-org/my-repo'" The attribute condition is the security boundary. Omit --attribute-condition and you have told Google to trust every repository on GitHub. Anyone who can run a public workflow can then mint a token your pool accepts. Newer gcloud versions refuse to create an unconditioned provider for exactly this reason. Keep the condition; make it as narrow as your workflow allows. Then let that repository — and nothing else in the pool — impersonate the deploy account: PN=$(gcloud projects describe $PROJECT --format='value(projectNumber)') gcloud iam service-accounts add-iam-policy-binding $SA --project=$PROJECT \ --role=roles/iam.workloadIdentityUser \ --member="principalSet://iam.googleapis.com/projects/$PN/locations/global/workloadIdentityPools/github/attribute.repository/my-org/my-repo" Three values go into repo secrets. None of them is a credential — they are addresses, and they are useless without a signed token from your repo. echo "projects/$PN/locations/global/workloadIdentityPools/github/providers/github-provider" \ | gh secret set WIF_PROVIDER echo "$SA" | gh secret set WIF_SERVICE_ACCOUNT echo "$PROJECT" | gh secret set GCP_PROJECT Tighten further before this touches production. The condition above is repo-only, which means it trusts any branch and any workflow in my-org/my-repo — not just deploy.yml on main. Two changes close that gap: Restrict to the deploy ref. Add && assertion.ref=='refs/heads/main' to the condition. Now a workflow triggered from a feature branch — even one added by someone with ordinary write access — can't mint a usable token. Bind to the repository ID, not its name. assertion.repository is a name, and names get reused: delete my-org/my-repo and recreate it (or let the org rename), and a new, unrelated repository inherits the trust. Map attribute.repository_id=assertion.repository_id and attribute.repository_owner_id=assertion.repository_owner_id in --attribute-mapping, then condition and bind on those IDs instead — they don't get reassigned when a name does. The roles nobody's tutorial lists The role list above already includes storage.admin at the project level — for a reason we only learned by getting it wrong first. Wiring this up for a real project, we started tighter: storage.admin scoped to just the Cloud Build staging bucket, on the theory that project-wide storage admin is a lot of blast radius for one deploy pipeline. Two failed deploys later, the reason became clear: gcloud builds submit doesn't only read and write objects in that bucket — before it uploads anything, it calls a project-scoped storage.buckets.list to confirm the bucket exists. No bucket-level binding, however permissive, can satisfy a call scoped to the whole project. The advice above is right; don't "improve" it. The second gap isn't in the role list at all. roles/viewer is required to stream build logs back to the CLI, separate from anything Cloud Build- or Storage-specific. Without it, gcloud builds submit exits non-zero and the whole nx deploy chain reports failure — even though the Cloud Build job itself finished and pushed the image successfully. We nearly chased a phantom second bug before running gcloud builds describe --format="value(status)" and seeing SUCCESS on a build the CLI, and therefore CI, had just reported as failed. roles/logging.viewer looks like the fix and isn't: the CLI's own error text checks for "Viewer/Owner of the project" — a primitive-role check, not a fine-grained permission. The lesson generalizes past this one pipeline: an error from a wrapper CLI ("forbidden from accessing bucket") describes where the failure surfaced, not why. When a permission error survives an IAM grant that should have fixed it, stop guessing at roles and get the client's raw HTTP trace instead: gcloud builds submit apps/web --tag $IMAGE --project $PROJECT --verbosity=debug It names the exact API call and scope that got denied — in less time than a second guess costs. 02. The workflow Ask Nx what to ship The whole file, then the four lines in it that matter. .github/workflows/deploy.yml name: Deploy on: push: branches: [main] workflow_dispatch: inputs: projects: description: 'Comma-separated projects (empty = affected only)' required: false default: '' permissions: contents: read actions: read id-token: write # mint the OIDC token — without this, auth fails # One deploy at a time. A second merge waits instead of racing Cloud Run. concurrency: group: deploy-main cancel-in-progress: false jobs: deploy: runs-on: ubuntu-latest env: # The deploy target expands $PROJECT (see apps/web/project.json above). # auth@v2 exports GCP_PROJECT/GOOGLE_CLOUD_PROJECT, not PROJECT — without # this line the image ref silently becomes gcr.io//web:latest. PROJECT: ${{ secrets.GCP_PROJECT }} steps: # Full history: `affected` diffs two commits. - uses: actions/checkout@v4 with: { fetch-depth: 0 } - uses: pnpm/action-setup@v4 with: { version: 10 } - uses: actions/setup-node@v4 with: { node-version: 22, cache: pnpm } - run: pnpm install --frozen-lockfile - uses: google-github-actions/auth@v2 with: workload_identity_provider: ${{ secrets.WIF_PROVIDER }} service_account: ${{ secrets.WIF_SERVICE_ACCOUNT }} - uses: google-github-actions/setup-gcloud@v2 - run: gcloud auth configure-docker gcr.io --quiet # Sets NX_BASE / NX_HEAD. Base = last commit this workflow # deployed successfully, so a red run is retried, not skipped. - uses: nrwl/nx-set-shas@v4 - name: Deploy affected projects if: ${{ inputs.projects == '' }} run: pnpm nx affected -t deploy --parallel=1 - name: Deploy selected projects if: ${{ inputs.projects != '' }} run: pnpm nx run-many -t deploy --parallel=1 --projects=${{ inputs.projects }} The four decisions id-token: write — without it there is no OIDC token to exchange and auth@v2 fails with a message about missing credentials that sounds like a secrets problem. fetch-depth: 0 — affected is a diff. A shallow clone gives it nothing to diff against. nx-set-shas — bases the diff on the last successful run of this workflow. If a deploy fails, the next merge redeploys everything that failed rather than pretending it shipped. --parallel=1 — these tasks are Docker builds pushing to a registry. Running four at once on a two-core runner buys nothing and fills the disk. The workflow_dispatch input is the escape hatch. When you need to force a service out — a secret rotated, a base image patched, no code change to point at — you run the workflow and type the project names, rather than inventing an empty commit. 03. The graph "Affected" is only as good as your dependency graph This is the part every tutorial skips, and it is the part that will bite you. nx affected answers a graph question: which projects depend on the files that changed? If the graph doesn't know about a dependency, the answer is confidently wrong — and a wrong answer here means your fix silently never deployed. That failure is much worse than a workflow that errors, because nothing turns red. Gap one: dependencies your language has but Nx can't see Our five Python services each vendor a shared library. Every Dockerfile does COPY packages/, and every pyproject.toml declares py-shared = { path = "../packages" }. Real dependency, universally understood — by everything except the Nx graph, which reads TypeScript imports. So a change to the shared LLM client marked nothing as affected, and CI would have deployed nothing. Try it yourself below: toggle whether implicitDependencies is declared, and watch what nx show projects --affected returns for the same file change. Interactive toggle: the original post has a live before/after toggle for this - try it here. Before: nx show projects --affected --files=packages/py_shared/llm.py -t deploy returns [] - every service uses this file, and none is marked affected. After adding implicitDependencies to each service's project.json: the same command returns all five services. The fix is one line per consumer, and it is worth a comment, because the next person will read it as redundant and delete it: { "name": "agents-create", // Each image COPYs the shared package, but the python path-dep // is invisible to the Nx graph — declare it or `affected` misses it. "implicitDependencies": ["agents-py-shared"] } Gap two: files that belong to no project at all Our five services share one Dockerfile at apps/agents/Dockerfile, parameterised by build arg. It sits one level above every project root, so Nx cannot attribute it to any project — change it and nothing is affected. The tempting fix is sharedGlobals in nx.json. We tried it; it does not work here. It marked only the TypeScript projects, because the Python projects have no target whose inputs reference default. Rather than contort the config, we made the rule explicit in the workflow, where it is visible: - name: Deploy agent services on shared Dockerfile change run: | # This file builds all 5 images but lives outside every project # root, so Nx cannot attribute it. Deploy the five by hand. if git diff --name-only "$NX_BASE" "$NX_HEAD" | grep -qx 'apps/agents/Dockerfile'; then pnpm nx run-many -t deploy --parallel=1 \ --projects=agents-create,agents-review,agents-resolve,agents-plan,agents-learn fi The general rule. Before you trust affected with production, list the ways your build crosses a language or a directory boundary — vendored packages, generated clients, shared Dockerfiles, root config. For each one, run nx show projects --affected --files= and check the answer against what you know. Every mismatch is a deploy that will quietly not happen. 04. Proof Verify both halves A deploy pipeline has two jobs and most people only test one. Shipping the right thing is the obvious half. Not shipping the wrong thing is the half that saves you at 2am, and it needs its own test. Merge Expected Actual Workflow + 5 project.json files 5 agent services, not web 5 new revisions, web untouched Documentation only Nothing NX No tasks were run The second row is the one to celebrate. A green run that deployed nothing means the pipeline understands your repo rather than rebuilding the world on every commit. Confirm against the platform, not the workflow log — a step can exit 0 having done nothing: for s in create review plan resolve learn web; do gcloud run services describe codemagpie-$s --region us-central1 \ --format='value(status.latestReadyRevisionName, status.conditions[0].lastTransitionTime)' done 05. Consequences What turning this on will find Deploying from CI changes which tests run, and that surfaces things hand-deploying hid. Ours went red on the first attempt, in a service the change never touched. Three tests in the plan agent asserted the issue title reached the model: prompt = mock.messages.create.call_args.kwargs["messages"][0]["content"] assert "Add dark mode" in prompt # green locally, red in CI messages[0] is the system prompt. The assertions had been checking the wrong string ever since a system message was introduced — and a fourth, assert "Additional instructions" not in prompt, had been passing vacuously the whole time. They passed locally because the developer's virtualenv still held a months-old build of the shared package, from before the system message existed. CI installs from source, so CI was right. Nobody had noticed, because nothing had ever marked that project as affected — the same graph gap, showing up as a stale test instead of a missed deploy.