xiaoxia-saas/docs/RELEASE-AUTOMATION-RETROSPECTIVE-2026-06-22.md

v0.1.9 Release Automation Retrospective

Date: 2026-06-22 Branch: develop Production release: v0.1.9

Result

v0.1.9 is the first verified end-to-end automatic production release for 小虾 SaaS:

1. Gitea tag push starts production release workflow.
2. Dedicated runtime-builder-01 builds API/Worker runtime images.
3. Dedicated builder builds Web apps/web/dist in a Node Docker container.
4. Dedicated builder uploads both artifacts to production.
5. Dedicated builder drives production deployment over SSH.
6. Production host only extracts artifacts, loads runtime images, runs migrations, restarts containers, and health checks.
7. Public Web/API smoke tests pass.

Verified Production State

After v0.1.9:

• xiaoxia-api-production runs xiaoxia-saas-api:v0.1.9 and is healthy.
• xiaoxia-worker-production runs xiaoxia-saas-worker:v0.1.9.
• xiaoxia-web-production is healthy and serves prebuilt Web dist.
• /var/lib/xiaoxia-saas-production/runtime-images-v0.1.9.tar is present, about 325 MB.
• /var/lib/xiaoxia-saas-production/release-v0.1.9.tar.gz is present, about 883 KB.

Public smoke results:

python scripts\smoke_public_auth_flow.py
public_auth_flow=ok

python scripts\smoke_public_upload_flow.py
public_upload_flow=ok

public project detail smoke
public_project_get=ok

What Repeated

The repeated tags were not feature releases; they were release-chain verification tags.

• v0.1.6: runtime-builder checkout/token fix worked, but Gitea skipped downstream production deploy after cross-runner needs.
• v0.1.7: production deploy moved to runtime-builder over SSH; build/upload succeeded, but deploy failed because deploy-production.sh defaulted to /host paths while SSH uses host paths directly.
• v0.1.8: SSH deploy used HOST_PREFIX=, but release artifact packaging used broad --exclude=dist, which removed apps/web/dist and correctly failed the Web artifact gate.
• v0.1.9: artifact packaging changed to --exclude=./dist, preserving apps/web/dist; automatic release succeeded.

Usable rollback tags:

• v0.1.5: previously verified stable production release.
• v0.1.9: current verified automatic production release.

Do not use v0.1.6, v0.1.7, or v0.1.8 as rollback targets unless intentionally reproducing release-chain failures.

Root Causes Fixed

1. GITHUB_TOKEN was not implicitly available in act_runner shell scripts.

   • Fix: checkout steps explicitly bind GITHUB_TOKEN: ${{ github.token }}.

2. Gitea needs with production deploy on a different runner path skipped the downstream deploy job.

   • Fix: production deploy also runs on runtime-builder, then SSHes to production.

3. Production deploy script defaults to /host for Docker-in-Docker CI execution.

   • Fix: SSH deploy passes HOST_PREFIX= so the script uses normal host paths.

4. Release source artifact accidentally included build output tarballs.

   • Fix: exclude root ./dist from release tar.

5. A broad --exclude=dist also excluded Web dist.

   • Fix: use --exclude=./dist, not --exclude=dist.

Current Release Contract

For production v* tag releases:

• Production host must not build API or Worker images.
• Production host must not build frontend assets.
• runtime-builder-01 is the build and deployment driver.
• runtime-images-<tag>.tar must exist before production deploy.
• release-<tag>.tar.gz must include apps/web/dist/index.html.
• Production deploy must fail if either artifact is missing.
• Production smoke is required before saying release is done.

Follow-Ups

• Add an explicit dry-run workflow or staging release rehearsal to reduce tag-based trial and error.
• Keep v0.1.6 to v0.1.8 documented as failed release-chain probes.
• Consider adding an API version endpoint that reports the Docker image tag; current /health version text can lag image tags.
• Review temporary credential posture for builder SSH key and old registration tokens.