perf(ci): 流水线提速 - 路径过滤/持久builder缓存/依赖缓存卷/部署job迁移ci-l2
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2m20s
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 3m28s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 3m16s
AI Code Review / AI Code Review (pull_request) Successful in 1m59s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 2m13s
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m59s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 2m8s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 4m45s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 3m8s
CI/CD Pipeline / Validate - Code Quality (pull_request) Successful in 9m25s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 4m16s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 16m45s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 28s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2m20s
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 3m28s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 3m16s
AI Code Review / AI Code Review (pull_request) Successful in 1m59s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 2m13s
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m59s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 2m8s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 4m45s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 3m8s
CI/CD Pipeline / Validate - Code Quality (pull_request) Successful in 9m25s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 4m16s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 16m45s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 28s
1. build-staging 路径过滤:纯前端改动只构建Web,纯后端只构建API/Worker; 未重建镜像通过 retag-staging-skipped job 将分支tag retag为新SHA推送,Watchtower链路不变 2. deploy-staging、acr-cleanup runs-on 从 runtime-builder 改为 ci-l2,释放构建槽 3. buildx 改为宿主机持久 builder(ci-builder-persist)+buildkit命名卷缓存, job结束不再prune,Worker镜像同样接入buildx缓存 4. validate/unit-test/frontend-test 的 pip/npm 依赖通过宿主机命名卷持久化缓存 (runner config 挂载 ci-pip-cache/xiaoxia-npm-cache/act-toolcache) 5. 不影响生产构建/发版/canary 相关 job
This commit is contained in:
@@ -696,14 +696,14 @@ jobs:
|
||||
done
|
||||
echo
|
||||
echo "${{ matrix.service_display }} PR build verified: ${IMAGE_TAG}"
|
||||
- name: Cleanup buildx builder
|
||||
- name: Builder cache note
|
||||
if: always()
|
||||
shell: sh
|
||||
run: |
|
||||
BUILDER_NAME="ci-pr-builder-${GITHUB_RUN_ID:-local}"
|
||||
docker buildx rm "$BUILDER_NAME" 2>/dev/null || true
|
||||
docker buildx prune -f 2>/dev/null || true
|
||||
echo "Builder cleanup done"
|
||||
# 持久 builder (ci-builder-persist) 跨 job 共享,不删除不 prune;
|
||||
# 残留容器/卷由宿主机 /usr/local/bin/ci-docker-cleanup.sh 兜底清理
|
||||
docker buildx ls | head -5
|
||||
echo "Persistent builder kept warm for next job"
|
||||
- name: Job duration summary
|
||||
if: always()
|
||||
shell: sh
|
||||
@@ -729,11 +729,49 @@ jobs:
|
||||
[ -f /tmp/ci_job_start_time ] && START_TIME=$(cat /tmp/ci_job_start_time)
|
||||
python3 scripts/ci/ci_trace_report.py --service xiaoxia-saas-ci --status $STATUS --start-time "$START_TIME" || true
|
||||
|
||||
check-push-paths:
|
||||
name: Check push changed paths
|
||||
runs-on: ci-l2
|
||||
if: github.event_name == 'push' && (github.ref_name == 'main' || github.ref_name == 'develop')
|
||||
outputs:
|
||||
skip_backend: ${{ steps.check.outputs.skip_backend }}
|
||||
skip_frontend: ${{ steps.check.outputs.skip_frontend }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
curl -sfH "Authorization: token $GITHUB_TOKEN" -o /tmp/_ci_checkout.sh "${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/raw/scripts/ci/step_checkout.sh?ref=${GITHUB_SHA}" && bash /tmp/_ci_checkout.sh
|
||||
- name: Check changed paths
|
||||
id: check
|
||||
shell: bash
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
bash scripts/ci/ci_push_paths.sh
|
||||
- name: Report CI trace
|
||||
if: always()
|
||||
shell: sh
|
||||
env:
|
||||
AGENTLOOP_LICENSE_KEY: ${{ secrets.AGENTLOOP_LICENSE_KEY }}
|
||||
run: |
|
||||
STATUS="ok"
|
||||
[ ${{ job.status }} = "success" ] || STATUS="error"
|
||||
START_TIME=""
|
||||
[ -f /tmp/ci_job_start_time ] && START_TIME=$(cat /tmp/ci_job_start_time)
|
||||
python3 scripts/ci/ci_trace_report.py --service xiaoxia-saas-ci --status $STATUS --start-time "$START_TIME" || true
|
||||
|
||||
build-staging:
|
||||
name: Build Staging ${{ matrix.service_display }} Image
|
||||
runs-on: runtime-builder
|
||||
timeout-minutes: ${{ matrix.timeout }}
|
||||
if: github.event_name == 'push' && (github.ref_name == 'main' || github.ref_name == 'develop')
|
||||
needs: check-push-paths
|
||||
if: |
|
||||
github.event_name == 'push' && (github.ref_name == 'main' || github.ref_name == 'develop') && (
|
||||
(matrix.service == 'web' && needs.check-push-paths.outputs.skip_frontend != 'true') ||
|
||||
(matrix.service != 'web' && needs.check-push-paths.outputs.skip_backend != 'true')
|
||||
)
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -797,23 +835,16 @@ jobs:
|
||||
echo "Cache mode: read-only"
|
||||
fi
|
||||
|
||||
- name: Setup buildx builder
|
||||
if: matrix.service != 'worker'
|
||||
- name: Ensure persistent buildx builder
|
||||
shell: sh
|
||||
run: |
|
||||
set -eu
|
||||
if ! docker buildx inspect ci-builder-${GITHUB_RUN_ID}-${GITHUB_JOB}-${{ matrix.cache_name }} > /dev/null 2>&1; then
|
||||
docker buildx create --use --name ci-builder-${GITHUB_RUN_ID}-${GITHUB_JOB}-${{ matrix.cache_name }} --driver docker-container
|
||||
echo "Created ci-builder (docker-container driver)"
|
||||
else
|
||||
docker buildx use ci-builder-${GITHUB_RUN_ID}-${GITHUB_JOB}-${{ matrix.cache_name }}
|
||||
echo "Using existing ci-builder"
|
||||
fi
|
||||
docker buildx inspect --bootstrap
|
||||
run: bash scripts/ci/ensure_persistent_builder.sh
|
||||
|
||||
- name: Pre-build worker base image (fallback if not exist)
|
||||
- name: Pre-pull worker base image (fallback build if not exist)
|
||||
if: matrix.service == 'worker'
|
||||
shell: sh
|
||||
env:
|
||||
ACR_USERNAME: ${{ secrets.ACR_USERNAME }}
|
||||
ACR_PASSWORD: ${{ secrets.ACR_PASSWORD }}
|
||||
run: |
|
||||
set -eu
|
||||
REGISTRY="xiaoxia-registry.cn-hangzhou.cr.aliyuncs.com/xiaoxiakeji"
|
||||
@@ -821,11 +852,13 @@ jobs:
|
||||
|
||||
echo "检查 Worker 基础镜像..."
|
||||
if docker pull "$BASE_IMAGE" 2>/dev/null; then
|
||||
echo "✅ 基础镜像已存在"
|
||||
echo "✅ 基础镜像已存在(buildkit 可直接命中)"
|
||||
else
|
||||
echo "⚠️ 基础镜像不存在,本地构建(fallback)..."
|
||||
docker build -f infra/docker/worker-base.Dockerfile -t "$BASE_IMAGE" .
|
||||
echo "✅ Worker 基础镜像本地构建完成"
|
||||
echo "⚠️ 基础镜像不存在,用持久 builder 构建并推送(fallback)..."
|
||||
docker buildx build --builder ci-builder-persist \
|
||||
-f infra/docker/worker-base.Dockerfile \
|
||||
-t "$BASE_IMAGE" --push .
|
||||
echo "✅ Worker 基础镜像构建推送完成"
|
||||
fi
|
||||
|
||||
- name: Build and push ${{ matrix.service_display }} image
|
||||
@@ -833,48 +866,34 @@ jobs:
|
||||
run: |
|
||||
set -eu
|
||||
REGISTRY="xiaoxia-registry.cn-hangzhou.cr.aliyuncs.com/xiaoxiakeji"
|
||||
IMAGE_TAG="${REGISTRY}/${{ matrix.image_name }}:${GITHUB_SHA}"
|
||||
IMAGE_FULL="${REGISTRY}/${{ matrix.image_name }}"
|
||||
IMAGE_TAG="${IMAGE_FULL}:${GITHUB_SHA}"
|
||||
# 同时推分支 tag,作为未重建镜像 retag 的稳定来源
|
||||
BRANCH_TAG="${IMAGE_FULL}:${GITHUB_REF_NAME}"
|
||||
CACHE_REF="${REGISTRY}/${{ matrix.cache_name }}:${GITHUB_REF_NAME}"
|
||||
|
||||
if [ "${{ matrix.service }}" = "worker" ]; then
|
||||
# Worker: plain docker build(基础镜像已预装全部依赖,无需 buildx)
|
||||
echo "=== Worker: plain docker build ==="
|
||||
docker build -f ${{ matrix.dockerfile }} -t "${IMAGE_TAG}" --build-arg APP_VERSION="${GITHUB_SHA}" .
|
||||
docker push "${IMAGE_TAG}"
|
||||
echo "✅ Worker image pushed: ${IMAGE_TAG}"
|
||||
else
|
||||
# API/Web: buildx with registry cache
|
||||
CACHE_REF="${REGISTRY}/${{ matrix.cache_name }}:${GITHUB_REF_NAME}"
|
||||
EXTRA_BUILD_ARGS="APP_VERSION=\"${GITHUB_SHA}\""
|
||||
if [ "${{ matrix.service }}" = "web" ]; then
|
||||
EXTRA_BUILD_ARGS="$EXTRA_BUILD_ARGS NGINX_CONF=infra/docker/nginx-staging.conf"
|
||||
fi
|
||||
|
||||
NO_CACHE_FLAG=""
|
||||
for i in 1 2 3; do
|
||||
echo "=== Docker build 尝试 $i/3 ==="
|
||||
if bash scripts/ci/docker_build_push.sh $NO_CACHE_FLAG ${{ matrix.dockerfile }} "${IMAGE_TAG}" "${CACHE_REF}" $EXTRA_BUILD_ARGS; then
|
||||
echo "✅ Docker build 成功"
|
||||
break
|
||||
fi
|
||||
echo "❌ Docker build 失败(尝试 $i/3)"
|
||||
[ $i -eq 3 ] && exit 1
|
||||
sleep 10
|
||||
if [ $i -eq 2 ]; then
|
||||
NO_CACHE_FLAG="--no-cache"
|
||||
echo "下次重试将使用 --no-cache"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "${{ matrix.service_display }} image pushed: ${IMAGE_TAG}"
|
||||
EXTRA_BUILD_ARGS="APP_VERSION=\"${GITHUB_SHA}\""
|
||||
if [ "${{ matrix.service }}" = "web" ]; then
|
||||
EXTRA_BUILD_ARGS="$EXTRA_BUILD_ARGS NGINX_CONF=infra/docker/nginx-staging.conf"
|
||||
fi
|
||||
- name: Cleanup buildx builder
|
||||
if: matrix.service != 'worker' && always()
|
||||
shell: sh
|
||||
run: |
|
||||
docker buildx rm ci-builder-${GITHUB_RUN_ID}-${GITHUB_JOB}-${{ matrix.cache_name }} 2>/dev/null || true
|
||||
docker buildx rm ci-builder 2>/dev/null || true
|
||||
docker buildx prune -f 2>/dev/null || true
|
||||
echo "Builder cleanup done"
|
||||
|
||||
NO_CACHE_FLAG=""
|
||||
for i in 1 2 3; do
|
||||
echo "=== Docker build 尝试 $i/3 (${{ matrix.service_display }}) ==="
|
||||
if EXTRA_TAGS="$BRANCH_TAG" bash scripts/ci/docker_build_push.sh $NO_CACHE_FLAG ${{ matrix.dockerfile }} "${IMAGE_TAG}" "${CACHE_REF}" $EXTRA_BUILD_ARGS; then
|
||||
echo "✅ Docker build 成功"
|
||||
break
|
||||
fi
|
||||
echo "❌ Docker build 失败(尝试 $i/3)"
|
||||
[ $i -eq 3 ] && exit 1
|
||||
sleep 10
|
||||
if [ $i -eq 2 ]; then
|
||||
NO_CACHE_FLAG="--no-cache"
|
||||
echo "下次重试将使用 --no-cache"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "${{ matrix.service_display }} image pushed: ${IMAGE_TAG} (+ ${BRANCH_TAG})"
|
||||
|
||||
- name: Job duration summary
|
||||
if: always()
|
||||
@@ -902,15 +921,97 @@ jobs:
|
||||
[ -f /tmp/ci_job_start_time ] && START_TIME=$(cat /tmp/ci_job_start_time)
|
||||
python3 scripts/ci/ci_trace_report.py --service xiaoxia-saas-ci --status $STATUS --start-time "$START_TIME" || true
|
||||
|
||||
retag-staging-skipped:
|
||||
name: Retag skipped Staging ${{ matrix.service_display }} Image
|
||||
runs-on: runtime-builder
|
||||
timeout-minutes: 10
|
||||
needs:
|
||||
- check-push-paths
|
||||
- build-staging
|
||||
if: |
|
||||
github.event_name == 'push' && (github.ref_name == 'main' || github.ref_name == 'develop') && (
|
||||
(matrix.service == 'web' && needs.check-push-paths.outputs.skip_frontend == 'true') ||
|
||||
(matrix.service != 'web' && needs.check-push-paths.outputs.skip_backend == 'true')
|
||||
)
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- service: api
|
||||
service_display: API
|
||||
image_name: xiaoxia-saas-api
|
||||
- service: worker
|
||||
service_display: Worker
|
||||
image_name: xiaoxia-saas-worker
|
||||
- service: web
|
||||
service_display: Web
|
||||
image_name: xiaoxia-saas-web
|
||||
steps:
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
curl -sfH "Authorization: token $GITHUB_TOKEN" -o /tmp/_ci_checkout.sh "${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/raw/scripts/ci/step_checkout.sh?ref=${GITHUB_SHA}" && bash /tmp/_ci_checkout.sh
|
||||
- name: Docker login to Registry
|
||||
shell: sh
|
||||
env:
|
||||
ACR_USERNAME: ${{ secrets.ACR_USERNAME }}
|
||||
ACR_PASSWORD: ${{ secrets.ACR_PASSWORD }}
|
||||
GITEA_REGISTRY_USER: xiaoxia
|
||||
GITEA_REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
||||
run: |
|
||||
set -eu
|
||||
for i in 1 2 3; do
|
||||
if printf '%s' "${ACR_PASSWORD}" | docker login xiaoxia-registry.cn-hangzhou.cr.aliyuncs.com -u "${ACR_USERNAME}" --password-stdin \
|
||||
&& docker login git.xiaoxiajianji.com -u "${GITEA_REGISTRY_USER}" -p "${GITEA_REGISTRY_TOKEN}"; then
|
||||
echo "✅ Docker login successful"
|
||||
break
|
||||
fi
|
||||
echo "Docker login 失败($i/3),5s 后重试..."
|
||||
sleep 5
|
||||
done
|
||||
- name: Retag previous branch image to new SHA
|
||||
shell: sh
|
||||
run: |
|
||||
set -eu
|
||||
REGISTRY="xiaoxia-registry.cn-hangzhou.cr.aliyuncs.com/xiaoxiakeji"
|
||||
bash scripts/ci/retag_skipped_image.sh \
|
||||
"${REGISTRY}/${{ matrix.image_name }}" \
|
||||
"${GITHUB_SHA}" \
|
||||
"${GITHUB_REF_NAME}"
|
||||
- name: Notify on failure
|
||||
continue-on-error: true
|
||||
if: failure()
|
||||
shell: sh
|
||||
env:
|
||||
CI_NOTIFY_WEBHOOK: ${{ secrets.CI_NOTIFY_WEBHOOK }}
|
||||
run: |
|
||||
set +e
|
||||
NOTIFY_MODE=failure JOB_NAME="Retag Staging ${{ matrix.service_display }} Image" python3 scripts/ci_notify.py
|
||||
- name: Report CI trace
|
||||
if: always()
|
||||
shell: sh
|
||||
env:
|
||||
AGENTLOOP_LICENSE_KEY: ${{ secrets.AGENTLOOP_LICENSE_KEY }}
|
||||
run: |
|
||||
STATUS="ok"
|
||||
[ ${{ job.status }} = "success" ] || STATUS="error"
|
||||
START_TIME=""
|
||||
[ -f /tmp/ci_job_start_time ] && START_TIME=$(cat /tmp/ci_job_start_time)
|
||||
python3 scripts/ci/ci_trace_report.py --service xiaoxia-saas-ci --status $STATUS --start-time "$START_TIME" || true
|
||||
|
||||
deploy-staging:
|
||||
name: Deploy Staging (Watchtower auto-deploy)
|
||||
runs-on: runtime-builder
|
||||
runs-on: ci-l2
|
||||
timeout-minutes: 15
|
||||
concurrency:
|
||||
group: deploy-staging-${{ gitea.ref }}
|
||||
cancel-in-progress: false
|
||||
needs:
|
||||
- check-push-paths
|
||||
- build-staging
|
||||
- retag-staging-skipped
|
||||
if: github.event_name == 'push' && (github.ref_name == 'main' || github.ref_name == 'develop')
|
||||
steps:
|
||||
- name: Checkout code
|
||||
@@ -1491,7 +1592,7 @@ jobs:
|
||||
|
||||
acr-cleanup:
|
||||
name: ACR Image Cleanup
|
||||
runs-on: runtime-builder
|
||||
runs-on: ci-l2
|
||||
timeout-minutes: 10
|
||||
needs:
|
||||
- deploy-staging
|
||||
|
||||
Executable
+77
@@ -0,0 +1,77 @@
|
||||
#!/bin/bash
|
||||
# CI 公共步骤:检测 push(develop/main) 事件的改动范围
|
||||
# 输出 skip_backend / skip_frontend(复用 PR check 的语义)
|
||||
# - 纯前端改动(仅 apps/web/): skip_backend=true
|
||||
# - 纯后端改动(不含 apps/web/): skip_frontend=true
|
||||
# - 全栈 / 无法判断: 两者都 false(走全量,安全兜底)
|
||||
# 需要环境变量: GITHUB_TOKEN, GITHUB_API_URL, GITHUB_REPOSITORY, GITHUB_SHA
|
||||
set -eu
|
||||
|
||||
OUTPUT="${GITHUB_OUTPUT:-/dev/stdout}"
|
||||
|
||||
before="${GITHUB_EVENT_BEFORE:-}"
|
||||
after="${GITHUB_SHA:-}"
|
||||
repo="${GITHUB_REPOSITORY:-}"
|
||||
base="${GITHUB_API_URL:-}"
|
||||
|
||||
# Gitea Actions 中 push 事件的前一个 SHA 在 event payload 的 before 字段
|
||||
if [ -z "$before" ] && [ -n "${GITHUB_EVENT_PATH:-}" ] && [ -f "$GITHUB_EVENT_PATH" ]; then
|
||||
before=$(python3 -c "
|
||||
import json,sys
|
||||
try:
|
||||
d=json.load(open('${GITHUB_EVENT_PATH}'))
|
||||
print(d.get('before','') or '')
|
||||
except Exception:
|
||||
print('')
|
||||
")
|
||||
fi
|
||||
|
||||
echo "改动范围检测: before=${before:-<empty>} after=${after}"
|
||||
|
||||
FILES=""
|
||||
if [ -n "$before" ] && [ "$before" != "0000000000000000000000000000000000000000" ]; then
|
||||
# compare API: {base}...{head}
|
||||
API_URL="${base}/repos/${repo}/compare/${before}...${after}?per_page=300"
|
||||
for attempt in 1 2 3; do
|
||||
FILES=$(curl -s --max-time 30 -H "Authorization: token ${GITHUB_TOKEN}" "$API_URL" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
try:
|
||||
d=json.load(sys.stdin)
|
||||
for f in d.get('files', []):
|
||||
print(f.get('filename',''))
|
||||
except Exception:
|
||||
pass
|
||||
")
|
||||
[ -n "$FILES" ] && break
|
||||
echo "compare API 无返回,重试 $attempt/3..."
|
||||
sleep 3
|
||||
done
|
||||
fi
|
||||
|
||||
if [ -z "$FILES" ]; then
|
||||
echo "⚠️ 无法获取改动文件列表(新分支/API异常),保守起见走全量构建"
|
||||
echo "skip_backend=false" >> "$OUTPUT"
|
||||
echo "skip_frontend=false" >> "$OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TOTAL=$(printf '%s\n' "$FILES" | grep -c . || true)
|
||||
FRONTEND_COUNT=$(printf '%s\n' "$FILES" | grep -c '^apps/web/' || true)
|
||||
BACKEND_COUNT=$(python3 -c "print($TOTAL - $FRONTEND_COUNT)")
|
||||
|
||||
echo "变更文件: ${TOTAL} 个 (前端: ${FRONTEND_COUNT}, 后端/公共: ${BACKEND_COUNT})"
|
||||
|
||||
if [ "$BACKEND_COUNT" = "0" ] && [ "$FRONTEND_COUNT" -gt "0" ]; then
|
||||
echo "skip_backend=true" >> "$OUTPUT"
|
||||
echo "skip_frontend=false" >> "$OUTPUT"
|
||||
echo "✅ 纯前端改动,跳过后端镜像构建"
|
||||
elif [ "$FRONTEND_COUNT" = "0" ] && [ "$BACKEND_COUNT" -gt "0" ]; then
|
||||
echo "skip_backend=false" >> "$OUTPUT"
|
||||
echo "skip_frontend=true" >> "$OUTPUT"
|
||||
echo "🔧 纯后端改动,跳过 Web 镜像构建"
|
||||
else
|
||||
echo "skip_backend=false" >> "$OUTPUT"
|
||||
echo "skip_frontend=false" >> "$OUTPUT"
|
||||
echo "🔧 包含全栈/公共变更,三个镜像全部构建"
|
||||
fi
|
||||
@@ -1,6 +1,7 @@
|
||||
#!/bin/bash
|
||||
# PR构建专用:只构建不输出,验证Dockerfile能否正常构建
|
||||
# 无本地缓存(12个runner不共享,反而添乱),只用ACR远程缓存
|
||||
# 缓存:复用宿主机持久 builder (ci-builder-persist) 的层缓存 + ACR registry 缓存兜底
|
||||
# 无状态:build-only 不推送,job 结束无需清理(builder 为共享持久资源)
|
||||
set -eu
|
||||
|
||||
NO_CACHE_FLAG=""
|
||||
@@ -18,17 +19,21 @@ for arg in "$@"; do
|
||||
BUILD_ARGS="$BUILD_ARGS --build-arg $arg"
|
||||
done
|
||||
|
||||
BUILDER_NAME="ci-pr-builder-${GITHUB_RUN_ID:-local}"
|
||||
BUILDER_NAME="ci-builder-persist"
|
||||
if ! docker buildx inspect "$BUILDER_NAME" > /dev/null 2>&1; then
|
||||
docker buildx create --use --name "$BUILDER_NAME" --driver docker-container
|
||||
else
|
||||
docker buildx use "$BUILDER_NAME"
|
||||
echo "持久 builder 不存在,创建中..."
|
||||
docker buildx create --name "$BUILDER_NAME" --driver docker-container \
|
||||
--driver-opt network=host \
|
||||
--buildkitd-flags "--allow-insecure-entitlement network.host" \
|
||||
--platform linux/amd64
|
||||
fi
|
||||
docker buildx inspect --bootstrap
|
||||
docker buildx use "$BUILDER_NAME"
|
||||
docker buildx inspect "$BUILDER_NAME" --bootstrap
|
||||
|
||||
echo "=== PR Build: build only, no output, remote cache only ==="
|
||||
echo "=== PR Build: build only, no push (persistent builder cache) ==="
|
||||
echo "Dockerfile: ${DOCKERFILE}"
|
||||
echo "Image tag: ${IMAGE_TAG}"
|
||||
echo "Image tag: ${IMAGE_TAG}"
|
||||
echo "Builder: ${BUILDER_NAME}"
|
||||
echo ""
|
||||
|
||||
docker buildx build \
|
||||
|
||||
@@ -1,10 +1,19 @@
|
||||
#!/bin/bash
|
||||
# 通用Docker镜像构建+推送脚本(local cache为主 + registry cache共享)
|
||||
# 通用Docker镜像构建+推送脚本
|
||||
# 缓存策略(2026-08 起):
|
||||
# - buildx 使用宿主机持久 builder (ci-builder-persist),层缓存保存在
|
||||
# buildkit 容器/命名卷中,跨 job 共享、job 结束不清理
|
||||
# - registry cache 仅作为冷启动兜底读取
|
||||
# - 额外 tag(如分支 tag :develop)通过 EXTRA_TAGS 环境变量传入,随构建一并推送
|
||||
# 用法: docker_build_push.sh [--no-cache] <Dockerfile> <image_tag> <cache_ref> [build_arg...]
|
||||
# 环境变量:
|
||||
# EXTRA_TAGS 空格分隔的额外 tag(完整 image:tag 引用),可选
|
||||
set -eu
|
||||
|
||||
# 单次 build 超时时间(秒),防止 docker buildx build 无限挂起
|
||||
BUILD_TIMEOUT=1500
|
||||
# 持久 builder 名(宿主机级,所有 CI job 共享;由 ensure_persistent_builder.sh 维护)
|
||||
BUILDER_NAME="ci-builder-persist"
|
||||
|
||||
NO_CACHE_FLAG=""
|
||||
if [ "$1" = "--no-cache" ]; then
|
||||
@@ -22,23 +31,27 @@ for arg in "$@"; do
|
||||
BUILD_ARGS="$BUILD_ARGS --build-arg $arg"
|
||||
done
|
||||
|
||||
if ! docker buildx inspect ci-builder > /dev/null 2>&1; then
|
||||
docker buildx create --use --name ci-builder --driver docker-container
|
||||
echo "Created ci-builder"
|
||||
else
|
||||
docker buildx use ci-builder
|
||||
echo "Using existing ci-builder"
|
||||
# 确保持久 builder 存在并使用(幂等)
|
||||
if ! docker buildx inspect "$BUILDER_NAME" > /dev/null 2>&1; then
|
||||
echo "持久 builder 不存在,创建中..."
|
||||
docker buildx create --name "$BUILDER_NAME" --driver docker-container \
|
||||
--driver-opt network=host \
|
||||
--buildkitd-flags "--allow-insecure-entitlement network.host" \
|
||||
--platform linux/amd64
|
||||
fi
|
||||
docker buildx inspect --bootstrap
|
||||
docker buildx use "$BUILDER_NAME"
|
||||
docker buildx inspect "$BUILDER_NAME" --bootstrap
|
||||
|
||||
# 从cache_ref中提取缓存名称(如 api-cache:develop -> api-cache-develop)
|
||||
CACHE_NAME=$(echo "$CACHE_REF" | tr '/' '_' | tr ':' '-')
|
||||
LOCAL_CACHE_DIR="/tmp/buildx-cache/${CACHE_NAME}"
|
||||
# 组装额外 tag 参数
|
||||
EXTRA_TAG_FLAGS=""
|
||||
EXTRA_TAG_LIST=""
|
||||
if [ -n "${EXTRA_TAGS:-}" ]; then
|
||||
for t in $EXTRA_TAGS; do
|
||||
EXTRA_TAG_FLAGS="$EXTRA_TAG_FLAGS -t $t"
|
||||
EXTRA_TAG_LIST="$EXTRA_TAG_LIST $t"
|
||||
done
|
||||
fi
|
||||
|
||||
mkdir -p "$LOCAL_CACHE_DIR"
|
||||
|
||||
# 缓存源:local优先(带自动修复),registry兜底读写
|
||||
# 本地缓存损坏时自动清理后重试,避免snapshot not found导致构建全挂
|
||||
build_with_cache_retry() {
|
||||
local attempt=1
|
||||
local max_attempts=2
|
||||
@@ -49,12 +62,10 @@ build_with_cache_retry() {
|
||||
build_output=$(timeout ${BUILD_TIMEOUT} docker buildx build \
|
||||
$NO_CACHE_FLAG \
|
||||
$BUILD_ARGS \
|
||||
--cache-from "type=local,src=${LOCAL_CACHE_DIR}" \
|
||||
--cache-from "type=registry,ref=${CACHE_REF}" \
|
||||
--cache-to "type=local,dest=${LOCAL_CACHE_DIR},mode=max" \
|
||||
--cache-to "type=registry,ref=${CACHE_REF},mode=max,ignore-error=true" \
|
||||
-f "${DOCKERFILE}" \
|
||||
-t "${IMAGE_TAG}" \
|
||||
$EXTRA_TAG_FLAGS \
|
||||
--push \
|
||||
. 2>&1)
|
||||
exit_code=$?
|
||||
@@ -69,48 +80,39 @@ build_with_cache_retry() {
|
||||
echo "$build_output" | tail -20
|
||||
return $exit_code
|
||||
fi
|
||||
# 检测到缓存损坏类错误,清掉本地缓存重试
|
||||
if echo "$build_output" | grep -qE "parent snapshot.*not found|snapshot.*does not exist|cache.*corrupt|failed to compute cache key"; then
|
||||
# 检测到缓存/快照损坏类错误,重建 builder 后重试
|
||||
if echo "$build_output" | grep -qE "parent snapshot.*not found|snapshot.*does not exist|cache.*corrupt|failed to compute cache key|no such file or directory.*cache"; then
|
||||
echo "$build_output"
|
||||
echo ""
|
||||
echo "⚠️ Local cache appears corrupted, cleaning up and retrying (attempt $attempt/$max_attempts)..."
|
||||
rm -rf "${LOCAL_CACHE_DIR}"
|
||||
mkdir -p "${LOCAL_CACHE_DIR}"
|
||||
# 清理buildx builder的内部snapshot状态
|
||||
docker buildx prune -f -a > /dev/null 2>&1 || true
|
||||
echo "⚠️ builder 缓存异常,重建持久 builder 后重试 (attempt $attempt/$max_attempts)..."
|
||||
docker buildx rm "$BUILDER_NAME" >/dev/null 2>&1 || true
|
||||
docker buildx create --name "$BUILDER_NAME" --driver docker-container \
|
||||
--driver-opt network=host \
|
||||
--buildkitd-flags "--allow-insecure-entitlement network.host" \
|
||||
--platform linux/amd64
|
||||
docker buildx use "$BUILDER_NAME"
|
||||
docker buildx inspect "$BUILDER_NAME" --bootstrap
|
||||
attempt=$((attempt + 1))
|
||||
else
|
||||
# 非缓存类错误,直接输出并返回
|
||||
echo "$build_output"
|
||||
return $exit_code
|
||||
fi
|
||||
done
|
||||
# 重试完还是失败,不用本地缓存最后试一次(只从registry读)
|
||||
echo "⚠️ All cached attempts failed, building without local cache..."
|
||||
timeout ${BUILD_TIMEOUT} docker buildx build \
|
||||
$NO_CACHE_FLAG \
|
||||
$BUILD_ARGS \
|
||||
--cache-from "type=registry,ref=${CACHE_REF}" \
|
||||
--cache-to "type=local,dest=${LOCAL_CACHE_DIR},mode=max" \
|
||||
--cache-to "type=registry,ref=${CACHE_REF},mode=max,ignore-error=true" \
|
||||
-f "${DOCKERFILE}" \
|
||||
-t "${IMAGE_TAG}" \
|
||||
--push \
|
||||
.
|
||||
return 1
|
||||
}
|
||||
|
||||
echo "=== Step 1: Build & push image (local cache + registry cache, with auto-repair) ==="
|
||||
echo "Local cache: ${LOCAL_CACHE_DIR}"
|
||||
echo "Registry cache: ${CACHE_REF}"
|
||||
echo "Build timeout: ${BUILD_TIMEOUT}s"
|
||||
echo "=== Build & push image (persistent builder cache) ==="
|
||||
echo "Builder: ${BUILDER_NAME} (persistent)"
|
||||
echo "Registry cache(from): ${CACHE_REF}"
|
||||
echo "Image tag: ${IMAGE_TAG}"
|
||||
[ -n "$EXTRA_TAG_LIST" ] && echo "Extra tags: ${EXTRA_TAG_LIST}"
|
||||
echo "Timeout: ${BUILD_TIMEOUT}s"
|
||||
echo ""
|
||||
|
||||
build_with_cache_retry
|
||||
|
||||
echo ""
|
||||
echo "Image pushed: ${IMAGE_TAG}"
|
||||
echo "Local cache updated"
|
||||
echo "Registry cache updated (if supported)"
|
||||
|
||||
[ -n "$EXTRA_TAG_LIST" ] && echo "Also pushed: ${EXTRA_TAG_LIST}"
|
||||
echo ""
|
||||
echo "Build completed: ${IMAGE_TAG}"
|
||||
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/bin/bash
|
||||
# CI 公共步骤:确保宿主机持久 buildx builder 存在(DooD 模式下所有 job 共享)
|
||||
# - builder 名固定: ci-builder-persist
|
||||
# - docker-container driver, host 网络
|
||||
# - 层缓存保存在 buildkit 容器及其 _state 命名卷中,job 结束不清理
|
||||
# - 宿主机 ci-docker-cleanup.sh 已豁免该 builder
|
||||
# 用法: bash scripts/ci/ensure_persistent_builder.sh
|
||||
set -eu
|
||||
|
||||
BUILDER="ci-builder-persist"
|
||||
|
||||
if ! docker buildx inspect "$BUILDER" >/dev/null 2>&1; then
|
||||
echo "=== 创建持久 buildx builder: $BUILDER ==="
|
||||
docker buildx create --name "$BUILDER" --driver docker-container \
|
||||
--driver-opt network=host \
|
||||
--buildkitd-flags "--allow-insecure-entitlement network.host" \
|
||||
--platform linux/amd64
|
||||
else
|
||||
echo "=== 复用持久 buildx builder: $BUILDER ==="
|
||||
fi
|
||||
|
||||
docker buildx use "$BUILDER"
|
||||
docker buildx inspect "$BUILDER" --bootstrap
|
||||
echo "✅ builder ready"
|
||||
docker buildx ls | head -5
|
||||
Executable
+47
@@ -0,0 +1,47 @@
|
||||
#!/bin/bash
|
||||
# CI 步骤:未重建的镜像,把 registry 上一个分支 tag 复制为新 SHA tag
|
||||
# 保证 deploy-staging 的 Watchtower 链路三个镜像都有新 SHA 可拉
|
||||
# 用法: bash scripts/ci/retag_skipped_image.sh <image_full_name> <new_sha> <branch>
|
||||
# 例: bash scripts/ci/retag_skipped_image.sh xiaoxia-registry.cn-hangzhou.cr.aliyuncs.com/xiaoxiakeji/xiaoxia-saas-web <sha> develop
|
||||
set -eu
|
||||
|
||||
IMAGE="$1"
|
||||
NEW_TAG="$2"
|
||||
BRANCH="${3:-develop}"
|
||||
|
||||
NEW_REF="${IMAGE}:${NEW_TAG}"
|
||||
|
||||
echo "=== 复用已有镜像(本次未重建): $IMAGE ==="
|
||||
echo "目标 tag: $NEW_TAG"
|
||||
|
||||
# 源 tag 候选(按优先级)
|
||||
CANDIDATES=()
|
||||
# 1. 分支 tag(构建 job 每次成功都会推)
|
||||
CANDIDATES+=("$BRANCH")
|
||||
# 2. 本 push 的前一个 commit SHA(compare 事件)
|
||||
if [ -n "${GITHUB_EVENT_BEFORE:-}" ] && [ "${GITHUB_EVENT_BEFORE}" != "0000000000000000000000000000000000000000" ]; then
|
||||
CANDIDATES+=("${GITHUB_EVENT_BEFORE}")
|
||||
fi
|
||||
# 3. registry 上最新的 sha 形式 tag(通过 ACR tags API 兜底,不需要额外认证则跳过)
|
||||
|
||||
SRC_TAG=""
|
||||
for cand in "${CANDIDATES[@]}"; do
|
||||
echo "尝试拉取 ${IMAGE}:${cand} ..."
|
||||
if docker pull "${IMAGE}:${cand}" >/dev/null 2>&1; then
|
||||
SRC_TAG="$cand"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$SRC_TAG" ]; then
|
||||
echo "❌ 找不到可复用的源镜像(已尝试: ${CANDIDATES[*]})"
|
||||
echo " 请检查该镜像是否曾成功构建推送,或临时改用全量构建。"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ 源镜像: ${IMAGE}:${SRC_TAG}"
|
||||
docker tag "${IMAGE}:${SRC_TAG}" "${NEW_REF}"
|
||||
|
||||
# 推新 SHA tag;分支 tag 若指向的就是源 digest 则无需重复,失败可忽略
|
||||
docker push "${NEW_REF}"
|
||||
echo "✅ retag 推送完成: ${NEW_REF} (from ${SRC_TAG})"
|
||||
Reference in New Issue
Block a user