fix(ci): 修复Staging部署和CI通知缺失脚本
CI/CD Pipeline / Check if frontend-only change (push) Has been skipped
CI/CD Pipeline / PR Build Web Image (push) Has been skipped
CI/CD Pipeline / PR Build API Image (push) Has been skipped
CI/CD Pipeline / PR Build Worker Image (push) Has been skipped
CI/CD Pipeline / Frontend Lint (push) Successful in 1m14s
CI/CD Pipeline / Validate - Type Check (mypy) (push) Successful in 1m16s
CI/CD Pipeline / Build Production Worker Image (push) Successful in 1m54s
CI/CD Pipeline / Validate - Migration (alembic) (push) Successful in 1m57s
CI/CD Pipeline / Frontend Unit Tests (push) Failing after 1m56s
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 2m8s
CI/CD Pipeline / Build Production Web Image (push) Successful in 2m50s
CI/CD Pipeline / Build Staging Web Image (push) Successful in 2m53s
CI/CD Pipeline / Validate - Code Quality (push) Successful in 5m2s
CI/CD Pipeline / Unit Tests (push) Successful in 5m2s
CI/CD Pipeline / Integration Tests (push) Successful in 2m30s
CI/CD Pipeline / CI Gate (push) Has been skipped
CI/CD Pipeline / Build Production API Image (push) Successful in 14m30s
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Build Staging API Image (push) Successful in 14m37s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Failing after 4m13s
CI/CD Pipeline / Staging E2E Tests (push) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (push) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (push) Has been skipped
CI/CD Pipeline / Canary Release to Production (push) Has been skipped
CI/CD Pipeline / Check if frontend-only change (push) Has been skipped
CI/CD Pipeline / PR Build Web Image (push) Has been skipped
CI/CD Pipeline / PR Build API Image (push) Has been skipped
CI/CD Pipeline / PR Build Worker Image (push) Has been skipped
CI/CD Pipeline / Frontend Lint (push) Successful in 1m14s
CI/CD Pipeline / Validate - Type Check (mypy) (push) Successful in 1m16s
CI/CD Pipeline / Build Production Worker Image (push) Successful in 1m54s
CI/CD Pipeline / Validate - Migration (alembic) (push) Successful in 1m57s
CI/CD Pipeline / Frontend Unit Tests (push) Failing after 1m56s
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 2m8s
CI/CD Pipeline / Build Production Web Image (push) Successful in 2m50s
CI/CD Pipeline / Build Staging Web Image (push) Successful in 2m53s
CI/CD Pipeline / Validate - Code Quality (push) Successful in 5m2s
CI/CD Pipeline / Unit Tests (push) Successful in 5m2s
CI/CD Pipeline / Integration Tests (push) Successful in 2m30s
CI/CD Pipeline / CI Gate (push) Has been skipped
CI/CD Pipeline / Build Production API Image (push) Successful in 14m30s
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Build Staging API Image (push) Successful in 14m37s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Failing after 4m13s
CI/CD Pipeline / Staging E2E Tests (push) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (push) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (push) Has been skipped
CI/CD Pipeline / Canary Release to Production (push) Has been skipped
- Bug4(P0): 新增scripts/ci_staging_deploy.sh - Staging环境部署脚本(pull镜像+migration+重启) - Bug4(P0): 新增scripts/ci_staging_healthcheck.sh - Staging健康检查脚本(API/Worker/Web) - Bug5(P1): 新增scripts/ci_notify.py - 飞书通知脚本(start/success/failure模式)
This commit is contained in:
Executable
+135
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
CI 通知脚本 - 发送飞书消息通知
|
||||
用法:
|
||||
NOTIFY_MODE=start JOB_NAME="Build API" python3 scripts/ci_notify.py
|
||||
NOTIFY_MODE=success JOB_NAME="Deploy Staging" python3 scripts/ci_notify.py
|
||||
NOTIFY_MODE=failure JOB_NAME="Unit Tests" python3 scripts/ci_notify.py
|
||||
|
||||
环境变量:
|
||||
NOTIFY_MODE - 通知类型: start/success/failure
|
||||
JOB_NAME - Job名称
|
||||
CI_NOTIFY_WEBHOOK - 飞书Webhook地址
|
||||
GITHUB_SHA - Commit SHA (可选)
|
||||
GITHUB_REF_NAME - 分支名 (可选)
|
||||
GITHUB_RUN_ID - Run ID (可选)
|
||||
GITHUB_REPOSITORY - 仓库名 (可选)
|
||||
GITHUB_SERVER_URL - Gitea地址 (可选)
|
||||
|
||||
设计原则:
|
||||
通知失败永远不阻断主流程(永远返回0)
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
def get_env(name, default=""):
|
||||
return os.environ.get(name, default)
|
||||
|
||||
|
||||
def send_feishu_notify(webhook_url, title, content, color="blue"):
|
||||
"""发送飞书通知(简单卡片格式)"""
|
||||
if not webhook_url:
|
||||
print("[INFO] 未配置 CI_NOTIFY_WEBHOOK,跳过通知")
|
||||
return True
|
||||
|
||||
# 状态颜色映射
|
||||
color_map = {
|
||||
"green": "green",
|
||||
"red": "red",
|
||||
"blue": "blue",
|
||||
"yellow": "yellow",
|
||||
}
|
||||
header_color = color_map.get(color, "blue")
|
||||
|
||||
# 构造卡片
|
||||
card = {
|
||||
"config": {"wide_screen_mode": True},
|
||||
"header": {
|
||||
"title": {"tag": "plain_text", "content": title},
|
||||
"template": header_color,
|
||||
},
|
||||
"elements": [
|
||||
{"tag": "div", "text": {"tag": "lark_md", "content": content}},
|
||||
],
|
||||
}
|
||||
|
||||
payload = {"msg_type": "interactive", "card": card}
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
|
||||
req = urllib.request.Request(
|
||||
webhook_url,
|
||||
data=data,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=10) as resp:
|
||||
resp_body = resp.read().decode("utf-8")
|
||||
result = json.loads(resp_body)
|
||||
if result.get("code", 0) != 0:
|
||||
print(f"[WARN] 飞书通知返回错误: {result.get('msg', resp_body)}", file=sys.stderr)
|
||||
return False
|
||||
print("[INFO] 飞书通知发送成功")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"[WARN] 飞书通知发送失败: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
|
||||
def build_message():
|
||||
"""根据环境变量构造通知消息"""
|
||||
notify_mode = get_env("NOTIFY_MODE", "info").lower()
|
||||
job_name = get_env("JOB_NAME", "未知Job")
|
||||
branch = get_env("GITHUB_REF_NAME", "未知分支")
|
||||
sha = get_env("GITHUB_SHA", "")[:8]
|
||||
run_id = get_env("GITHUB_RUN_ID", "")
|
||||
repo = get_env("GITHUB_REPOSITORY", "")
|
||||
server_url = get_env("GITHUB_SERVER_URL", "https://git.xiaoxiajianji.com")
|
||||
|
||||
# 状态映射
|
||||
status_map = {
|
||||
"start": ("🔔 CI 任务开始", "blue", "开始执行"),
|
||||
"success": ("✅ CI 任务成功", "green", "执行成功"),
|
||||
"failure": ("❌ CI 任务失败", "red", "执行失败"),
|
||||
"info": ("ℹ️ CI 通知", "blue", "通知"),
|
||||
}
|
||||
title, color, status_text = status_map.get(notify_mode, status_map["info"])
|
||||
|
||||
# 构造内容
|
||||
content_lines = [
|
||||
f"**任务**: {job_name}",
|
||||
f"**状态**: {status_text}",
|
||||
f"**分支**: {branch}",
|
||||
]
|
||||
if sha:
|
||||
content_lines.append(f"**Commit**: `{sha}`")
|
||||
if run_id and repo and server_url:
|
||||
run_url = f"{server_url}/{repo}/actions/runs/{run_id}"
|
||||
content_lines.append(f"**详情**: [点击查看]({run_url})")
|
||||
|
||||
content_lines.append(f"**时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
|
||||
|
||||
content = "\n".join(content_lines)
|
||||
return title, content, color
|
||||
|
||||
|
||||
def main():
|
||||
webhook = get_env("CI_NOTIFY_WEBHOOK", "")
|
||||
title, content, color = build_message()
|
||||
|
||||
print(f"[CI Notify] 模式: {get_env('NOTIFY_MODE')}")
|
||||
print(f"[CI Notify] 任务: {get_env('JOB_NAME')}")
|
||||
|
||||
send_feishu_notify(webhook, title, content, color)
|
||||
|
||||
# 永远返回0,不阻断主流程
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+173
@@ -0,0 +1,173 @@
|
||||
#!/bin/sh
|
||||
# ============================================
|
||||
# CI Staging 部署脚本 - 通过 SSH 在 Staging 服务器执行
|
||||
# 用法:IMAGE_TAG=<sha> ACR_USERNAME=<user> ACR_PASSWORD=<pass> sh ci_staging_deploy.sh
|
||||
# ============================================
|
||||
|
||||
set -eu
|
||||
|
||||
IMAGE_TAG="${IMAGE_TAG:-}"
|
||||
ACR_REGISTRY="${ACR_REGISTRY:-xiaoxia-registry.cn-hangzhou.cr.aliyuncs.com/xiaoxiakeji}"
|
||||
ACR_USERNAME="${ACR_USERNAME:-}"
|
||||
ACR_PASSWORD="${ACR_PASSWORD:-}"
|
||||
|
||||
ENV_FILE="${ENV_FILE:-/var/lib/xiaoxia-saas-staging/.env}"
|
||||
GENERATED_DIR="${GENERATED_DIR:-/var/lib/xiaoxia-saas-staging/generated}"
|
||||
STAGING_NETWORK="${STAGING_NETWORK:-xiaoxia-net-staging}"
|
||||
|
||||
if [ -z "$IMAGE_TAG" ]; then
|
||||
echo "ERROR: IMAGE_TAG is required"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "============================================"
|
||||
echo " Staging 部署 - CI触发"
|
||||
echo " Version: $IMAGE_TAG"
|
||||
echo "============================================"
|
||||
echo ""
|
||||
|
||||
# ---- 登录 ACR ----
|
||||
if [ -n "$ACR_PASSWORD" ] && [ -n "$ACR_USERNAME" ]; then
|
||||
echo "登录 ACR..."
|
||||
printf '%s' "$ACR_PASSWORD" | docker login "$(echo "$ACR_REGISTRY" | cut -d/ -f1)" -u "$ACR_USERNAME" --password-stdin 2>/dev/null || {
|
||||
echo "WARN: docker login failed, will try to pull anyway"
|
||||
}
|
||||
fi
|
||||
|
||||
# ---- Pull 3个镜像 ----
|
||||
API_IMAGE="${ACR_REGISTRY}/xiaoxia-saas-api:${IMAGE_TAG}"
|
||||
WORKER_IMAGE="${ACR_REGISTRY}/xiaoxia-saas-worker:${IMAGE_TAG}"
|
||||
WEB_IMAGE="${ACR_REGISTRY}/xiaoxia-saas-web:${IMAGE_TAG}"
|
||||
|
||||
echo ""
|
||||
echo "拉取 API 镜像..."
|
||||
docker pull "$API_IMAGE"
|
||||
echo "拉取 Worker 镜像..."
|
||||
docker pull "$WORKER_IMAGE"
|
||||
echo "拉取 Web 镜像..."
|
||||
docker pull "$WEB_IMAGE"
|
||||
echo ""
|
||||
echo "所有镜像拉取完成"
|
||||
|
||||
# ---- 确保基础设施在运行 ----
|
||||
echo ""
|
||||
echo "检查基础设施容器..."
|
||||
for c in xiaoxia-postgres-staging xiaoxia-redis-staging; do
|
||||
if ! docker inspect "$c" >/dev/null 2>&1; then
|
||||
echo "ERROR: 基础设施容器不存在: $c"
|
||||
exit 1
|
||||
fi
|
||||
state=$(docker inspect -f '{{.State.Status}}' "$c")
|
||||
if [ "$state" != "running" ]; then
|
||||
echo "ERROR: 基础设施容器未运行: $c ($state)"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "基础设施正常"
|
||||
|
||||
# ---- 确保 staging 网络存在 ----
|
||||
docker network create "$STAGING_NETWORK" 2>/dev/null || true
|
||||
|
||||
mkdir -p "$GENERATED_DIR"
|
||||
|
||||
# ---- 数据库 Migration ----
|
||||
echo ""
|
||||
echo "执行数据库迁移..."
|
||||
docker run --rm --env-file "$ENV_FILE" --network "$STAGING_NETWORK" "$API_IMAGE" sh -c "cd /app && /opt/venv/bin/alembic upgrade head"
|
||||
echo "数据库迁移完成"
|
||||
|
||||
# ---- 停止旧容器 ----
|
||||
echo ""
|
||||
echo "停止旧容器..."
|
||||
docker rm -f xiaoxia-api-staging 2>/dev/null || true
|
||||
docker rm -f xiaoxia-worker-staging 2>/dev/null || true
|
||||
docker rm -f xiaoxia-web-staging 2>/dev/null || true
|
||||
|
||||
# ---- 启动 API ----
|
||||
echo "启动 API 容器..."
|
||||
docker run -d \
|
||||
--name xiaoxia-api-staging \
|
||||
--env-file "$ENV_FILE" \
|
||||
--network "$STAGING_NETWORK" \
|
||||
-p 127.0.0.1:8000:8000 \
|
||||
-e APP_ENV=staging \
|
||||
-e APP_VERSION="$IMAGE_TAG" \
|
||||
-e GENERATED_FILES_DIR=/app/generated \
|
||||
-e GENERATED_FILES_URL_PREFIX=/generated-files \
|
||||
-v "$GENERATED_DIR:/app/generated" \
|
||||
--restart unless-stopped \
|
||||
--health-cmd "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=5)\"" \
|
||||
--health-interval 30s \
|
||||
--health-timeout 10s \
|
||||
--health-retries 3 \
|
||||
--health-start-period 40s \
|
||||
"$API_IMAGE"
|
||||
|
||||
# ---- 启动 Worker ----
|
||||
echo "启动 Worker 容器..."
|
||||
docker run -d \
|
||||
--name xiaoxia-worker-staging \
|
||||
--env-file "$ENV_FILE" \
|
||||
--network "$STAGING_NETWORK" \
|
||||
-e APP_ENV=staging \
|
||||
-e APP_VERSION="$IMAGE_TAG" \
|
||||
-e WORKER_CONCURRENCY=1 \
|
||||
-e WORKER_MAX_TASKS_PER_CHILD=100 \
|
||||
-e GENERATED_FILES_DIR=/app/generated \
|
||||
-e GENERATED_FILES_URL_PREFIX=/generated-files \
|
||||
-v "$GENERATED_DIR:/app/generated" \
|
||||
--restart unless-stopped \
|
||||
--health-cmd "sh -c \"grep -q celery /proc/1/cmdline || exit 1\"" \
|
||||
--health-interval 30s \
|
||||
--health-timeout 10s \
|
||||
--health-retries 3 \
|
||||
--health-start-period 30s \
|
||||
"$WORKER_IMAGE"
|
||||
|
||||
# ---- 启动 Web ----
|
||||
echo "启动 Web 容器..."
|
||||
docker run -d \
|
||||
--name xiaoxia-web-staging \
|
||||
--network "$STAGING_NETWORK" \
|
||||
-p 127.0.0.1:3001:80 \
|
||||
--restart unless-stopped \
|
||||
--health-cmd "wget --spider -q http://127.0.0.1:80" \
|
||||
--health-interval 30s \
|
||||
--health-timeout 5s \
|
||||
--health-retries 3 \
|
||||
"$WEB_IMAGE"
|
||||
|
||||
# ---- 等待 API 健康 ----
|
||||
echo ""
|
||||
echo "等待 API 健康检查..."
|
||||
i=0
|
||||
while [ "$i" -lt 30 ]; do
|
||||
if curl -sf --max-time 5 http://127.0.0.1:8000/health >/dev/null 2>&1; then
|
||||
echo "API 健康检查通过"
|
||||
break
|
||||
fi
|
||||
i=$((i + 1))
|
||||
echo " 等待中... ($i/30)"
|
||||
sleep 2
|
||||
done
|
||||
|
||||
if [ "$i" -ge 30 ]; then
|
||||
echo "ERROR: API 在 60s 内未通过健康检查"
|
||||
echo ""
|
||||
echo "=== API 最近日志 ==="
|
||||
docker logs --tail 50 xiaoxia-api-staging
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ---- 清理旧镜像 ----
|
||||
docker image prune -af --filter "until=72h" 2>/dev/null || true
|
||||
|
||||
echo ""
|
||||
echo "============================================"
|
||||
echo " ✅ Staging 部署完成"
|
||||
echo "============================================"
|
||||
echo "API: http://127.0.0.1:8000"
|
||||
echo "Web: http://127.0.0.1:3001"
|
||||
echo "Version: $IMAGE_TAG"
|
||||
echo ""
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}" | grep staging
|
||||
Executable
+110
@@ -0,0 +1,110 @@
|
||||
#!/bin/bash
|
||||
# ============================================
|
||||
# CI Staging 健康检查脚本 - 在 CI Runner 上执行
|
||||
# 通过 SSH 连接 Staging 服务器,检查 API/Worker/Web 健康状态
|
||||
# ============================================
|
||||
|
||||
set -eu
|
||||
|
||||
STAGING_SSH_HOST="${STAGING_SSH_HOST:-47.98.113.167}"
|
||||
STAGING_SSH_USER="${STAGING_SSH_USER:-root}"
|
||||
STAGING_SSH_PORT="${STAGING_SSH_PORT:-22222}"
|
||||
STAGING_SSH_KEY="${STAGING_SSH_KEY:-}"
|
||||
|
||||
HEALTH_TIMEOUT="${HEALTH_TIMEOUT:-120}"
|
||||
HEALTH_INTERVAL="${HEALTH_INTERVAL:-5}"
|
||||
|
||||
echo "============================================"
|
||||
echo " Staging 健康检查"
|
||||
echo " Host: $STAGING_SSH_HOST:$STAGING_SSH_PORT"
|
||||
echo "============================================"
|
||||
echo ""
|
||||
|
||||
# ---- 准备 SSH key ----
|
||||
mkdir -p ~/.ssh
|
||||
key_path=""
|
||||
if [ -f /root/.ssh/xiaoxia_runtime_builder ]; then
|
||||
key_path="/root/.ssh/xiaoxia_runtime_builder"
|
||||
elif [ -f "$HOME/.ssh/xiaoxia_runtime_builder" ]; then
|
||||
key_path="$HOME/.ssh/xiaoxia_runtime_builder"
|
||||
elif [ -n "${STAGING_SSH_KEY:-}" ]; then
|
||||
key_path="$HOME/.ssh/staging_deploy_key"
|
||||
printf '%s\n' "$STAGING_SSH_KEY" > "$key_path"
|
||||
chmod 600 "$key_path"
|
||||
else
|
||||
echo "ERROR: 没有可用的 SSH key"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ssh-keyscan -p "$STAGING_SSH_PORT" -H "$STAGING_SSH_HOST" >> ~/.ssh/known_hosts 2>/dev/null || true
|
||||
|
||||
SSH_CMD="ssh -p $STAGING_SSH_PORT -i $key_path -o StrictHostKeyChecking=no ${STAGING_SSH_USER}@${STAGING_SSH_HOST}"
|
||||
|
||||
# ---- 检查函数 ----
|
||||
check_api() {
|
||||
$SSH_CMD "curl -sf --max-time 5 http://127.0.0.1:8000/health" 2>/dev/null >/dev/null
|
||||
}
|
||||
|
||||
check_web() {
|
||||
$SSH_CMD "curl -sf --max-time 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:3001/" 2>/dev/null | grep -q "200"
|
||||
}
|
||||
|
||||
check_worker() {
|
||||
$SSH_CMD "docker inspect --format='{{.State.Status}}' xiaoxia-worker-staging 2>/dev/null | grep -q running" 2>/dev/null
|
||||
}
|
||||
|
||||
# ---- 循环等待健康 ----
|
||||
echo "等待 Staging 环境健康 (超时: ${HEALTH_TIMEOUT}s)..."
|
||||
elapsed=0
|
||||
api_ok=0
|
||||
web_ok=0
|
||||
worker_ok=0
|
||||
|
||||
while [ "$elapsed" -lt "$HEALTH_TIMEOUT" ]; do
|
||||
# 检查 API
|
||||
if [ "$api_ok" -eq 0 ] && check_api; then
|
||||
echo " ✅ API 健康"
|
||||
api_ok=1
|
||||
fi
|
||||
|
||||
# 检查 Web
|
||||
if [ "$web_ok" -eq 0 ] && check_web; then
|
||||
echo " ✅ Web 健康"
|
||||
web_ok=1
|
||||
fi
|
||||
|
||||
# 检查 Worker
|
||||
if [ "$worker_ok" -eq 0 ] && check_worker; then
|
||||
echo " ✅ Worker 运行中"
|
||||
worker_ok=1
|
||||
fi
|
||||
|
||||
# 全部通过
|
||||
if [ "$api_ok" -eq 1 ] && [ "$web_ok" -eq 1 ] && [ "$worker_ok" -eq 1 ]; then
|
||||
echo ""
|
||||
echo "============================================"
|
||||
echo " ✅ 全部健康检查通过"
|
||||
echo "============================================"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
sleep "$HEALTH_INTERVAL"
|
||||
elapsed=$((elapsed + HEALTH_INTERVAL))
|
||||
echo " 等待中... (${elapsed}s/${HEALTH_TIMEOUT}s)"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "============================================"
|
||||
echo " ❌ 健康检查超时"
|
||||
echo "============================================"
|
||||
echo ""
|
||||
echo "未通过的检查项:"
|
||||
[ "$api_ok" -eq 0 ] && echo " - API (http://127.0.0.1:8000/health)"
|
||||
[ "$web_ok" -eq 0 ] && echo " - Web (http://127.0.0.1:3001/)"
|
||||
[ "$worker_ok" -eq 0 ] && echo " - Worker 容器"
|
||||
|
||||
echo ""
|
||||
echo "=== API 最近日志 ==="
|
||||
$SSH_CMD "docker logs --tail 30 xiaoxia-api-staging" 2>/dev/null || echo "无法获取日志"
|
||||
|
||||
exit 1
|
||||
Reference in New Issue
Block a user