Compare commits

..

1 Commits

Author SHA1 Message Date
Coze Agent 7a06ad52de fix(generate): 修复进度条卡56%不刷新P1 bug
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m15s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 1m27s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Successful in 1m32s
CI/CD Pipeline / PR Build Web Image (pull_request) Successful in 1m57s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 3m22s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 4m11s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 4m40s
AI Code Review / AI Code Review (pull_request) Successful in 6m42s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 10m44s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 14m36s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 1s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
根因分析:
1. 后端任务状态新增/历史脏值(pending/queued/unknown等)未在前端
   normalizeTaskStatus映射中,fallback分支既不更新progress也不终止
   轮询,导致视觉停在上一次progress(约56%)
2. 旧版轮询网络/5xx错误连续10次即终止,浏览器tab切后台被节流或
   短暂网络抖动后永久停止轮询,用户看到的是停滞的旧progress
3. 浏览器后台tab的setTimeout被节流,切回前台时不会主动拉取最新状态,
   必须等下一个定时器才能看到结果
4. 生成完成后需手动点击「下一步」,用户误以为仍在渲染

修复:
- 新增normalizeTaskStatus兼容pending/queued/scheduled/processing/
  rendering/unknown/complete/done等状态,pending阶段显示≤5%进度,
  unknown状态保持上次进度继续轮询
- 网络/5xx错误改为指数退避无限重试(最大15s间隔),连续5次错误弹
  一次提示;4xx才终止(任务不存在/未授权)
- visibilitychange监听:页面从后台切回立即触发一次poll,秒级看到最新状态
- unmount时useEffect cleanup兜底清理所有定时器和控制器
- 进度停滞探测(15s无变化自动缩短到3s再查),防后端最后progress回调丢失
- awaiting_cover阶段results接口write-after-read抖动:空结果补1.5s后再查
- 终态前强制onProgress(100),避免UI停在95%
- 生成完成(单视频finalVideo就绪/批量全部完成且至少一个成功)后延迟800ms
  自动跳转到Step5封面选择页,无需手动点下一步

验证:tsc/eslint/prettier/vite build全通过
2026-09-27 23:12:30 +08:00
13 changed files with 546 additions and 605 deletions
+2 -21
View File
@@ -79,33 +79,14 @@ CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/1
# ==================== Worker 配置(#2073 队列分流) ====================
#
# 容器内跑三个独立进程:beat(只发定时任务)+ generation worker(实时高优)
# + transcode worker(后台批量/清理)。三个进程的并发与开关独立配置。
# ==================== Worker 配置 ====================
# Worker 进程名称
WORKER_NAME=xiaoxia-saas-worker
# 总并发参考(兼容旧变量):
# - 若 GENERATION_CONCURRENCY 与 TRANSCODE_CONCURRENCY 都未显式设置,
# entrypoint 会按此总数对半分配(gen=ceil(total/2), trans=剩余,各至少 1);
# - 任一个 *_CONCURRENCY 显式设置后,按显式值生效,忽略此变量对应部分。
# Worker 并发数(同时执行的任务数)
WORKER_CONCURRENCY=4
# Generation worker 并发数(用户实时任务:视频生成/TTS/音色克隆/lipsync/数字人)
# 实时链路对延迟敏感,建议 2C 以上机器设为 2;高负载场景可加到 4。
GENERATION_CONCURRENCY=2
# Transcode worker 并发数(后台批量:素材入库转码/AI 分类打标/质量评分/查重/批量下载)
# 后台任务可排队,独立伸缩;素材入库量大时可加到 4。
TRANSCODE_CONCURRENCY=2
# 是否在本容器启动 celery beat 进程(默认 1)。
# 默认 beat 与 worker 同容器部署;若要独立 beat 容器部署,worker 容器设为 0、
# beat 容器单独跑 `celery -A worker_app.celery_app beat` 并设 BEAT_ENABLED=1。
BEAT_ENABLED=1
# 每个子进程最多处理多少任务后重启(防止内存泄漏)
WORKER_MAX_TASKS_PER_CHILD=1000
+1 -9
View File
@@ -1302,16 +1302,8 @@ jobs:
"${staging_user}@${staging_host}:/var/lib/xiaoxia-saas-staging/configs/douyin_cookies.txt"
echo "✅ Douyin cookies uploaded"
# 上传 infra/docker 配置到服务器(compose 单一事实来源)
echo "Uploading infra/docker configs to staging server..."
ssh -p "$staging_port" -i "$key_path" -o StrictHostKeyChecking=no "${staging_user}@${staging_host}" \
"mkdir -p /var/lib/xiaoxia-saas-staging/infra/docker"
scp -P "$staging_port" -i "$key_path" -o StrictHostKeyChecking=no infra/docker/compose.yml \
"${staging_user}@${staging_host}:/var/lib/xiaoxia-saas-staging/infra/docker/compose.yml"
echo "✅ infra/docker/compose.yml uploaded"
# 通过环境变量传递凭证,避免命令行引号转义问题
cat scripts/ci_staging_deploy.sh | ssh -p "$staging_port" -i "$key_path" -o StrictHostKeyChecking=no "${staging_user}@${staging_host}" "IMAGE_TAG=${GITHUB_SHA} ACR_USERNAME=${ACR_USERNAME} ACR_PASSWORD=${ACR_PASSWORD} COMPOSE_SYNC=0 sh"
cat scripts/ci_staging_deploy.sh | ssh -p "$staging_port" -i "$key_path" -o StrictHostKeyChecking=no "${staging_user}@${staging_host}" "IMAGE_TAG=${GITHUB_SHA} ACR_USERNAME=${ACR_USERNAME} ACR_PASSWORD=${ACR_PASSWORD} sh"
# 清理 CI runner 上的渲染文件
rm -f .env.rendered
@@ -494,6 +494,49 @@ const GeneratePage: React.FC = () => {
navigate,
])
/* ── 生成完成后自动跳转封面选择页(修复进度条卡 56% 看不到结果的体验问题) ──
单视频:生成完成且有 finalVideo 时延迟 800ms 自动跳 step5
批量:所有任务完成(无 running 项)且至少有一个成功时延迟 800ms 跳转
失败/用户手动点重试时不触发(progress 回到 0 或 generating=false 且无 finalVideo)
*/
useEffect(() => {
if (currentStep !== 4) return
if (generating) return
if (generateError) return
if (!generated) return
let shouldJump = false
if (isBatch) {
const allDone =
batchTasks.length > 0 &&
batchTasks.every(
(t) => t.status === "completed" || t.status === "awaiting_cover" || t.status === "failed",
)
const anySuccess = batchTasks.some(
(t) => t.status === "completed" || t.status === "awaiting_cover",
)
shouldJump = allDone && anySuccess
} else {
shouldJump = !!(finalVideo || currentTaskId)
}
if (!shouldJump) return
const t = setTimeout(() => {
if (currentStep === 4) setCurrentStep(5)
}, 800)
return () => clearTimeout(t)
}, [
currentStep,
generating,
generateError,
generated,
isBatch,
batchTasks,
finalVideo,
currentTaskId,
setCurrentStep,
])
/* ── 布局 class ── */
const layoutClassName = "xx-generate-layout full-width"
@@ -1,4 +1,4 @@
import { useRef, useCallback, useState } from "react"
import { useRef, useCallback, useState, useEffect } from "react"
import { message } from "antd"
import axios from "axios"
import { getGenerationTask, retryTask as retryGenerationTaskApi } from "@/api/tasks/tasks"
@@ -25,19 +25,80 @@ interface UseGenerationPollingOptions {
onBatchTaskUpdate?: (taskId: string, patch: Partial<BatchTaskState>) => void
}
/** 最大连续错误次数(仅对可重试错误),超过后终止轮询 */
const MAX_RETRYABLE_ERRORS = 10
/** 获取结果的最大重试次数 */
/** 获取结果的最大重试次数(5xx/网络错误时;4xx 直接放弃) */
const MAX_RESULTS_RETRIES = 3
/** 轮询基础间隔(毫秒) */
const BASE_POLL_INTERVAL_MS = 2000
/** 初始启动延迟 */
const INITIAL_DELAY_MS = 1000
/** 网络/5xx 错误时的退避基础间隔(指数退避,最大 15s) */
const NETWORK_ERROR_BASE_MS = 2000
const NETWORK_ERROR_MAX_MS = 15000
/** 连续网络错误达到阈值弹一次提示(避免 toast 风暴) */
const NETWORK_ERROR_NOTICE_THRESHOLD = 5
/** 进度长时间无变化时,缩短轮询间隔主动探测,防止后端最后一次 progress 回调丢失 */
const STALL_PROBE_MS = 15000
/**
* 生成状态轮询 Hook(v4 — 批量任务独立状态 + 单任务重试)
* 把后端返回的任务状态字符串归一化到前端处理的状态。
* 兼容历史脏数据/后端新增状态(pending/queued/scheduled 等),避免出现"未知状态→既不更新进度也不终止"的卡死。
*/
function normalizeTaskStatus(
raw: string | undefined,
): "completed" | "awaiting_cover" | "failed" | "cancelled" | "running" | "pending" | "unknown" {
if (!raw) return "unknown"
const s = String(raw).trim().toLowerCase()
if (
s === "completed" ||
s === "success" ||
s === "done" ||
s === "finished" ||
s === "complete"
) {
return "completed"
}
if (
s === "awaiting_cover" ||
s === "waiting_cover" ||
s === "pending_cover" ||
s === "video_ready" ||
s === "rendered"
) {
return "awaiting_cover"
}
if (s === "failed" || s === "error" || s === "err" || s === "fail") return "failed"
if (s === "cancelled" || s === "canceled") return "cancelled"
if (
s === "running" ||
s === "processing" ||
s === "in_progress" ||
s === "rendering" ||
s === "process"
) {
return "running"
}
if (s === "pending" || s === "queued" || s === "scheduled") return "pending"
return "unknown"
}
/**
* 生成状态轮询 Hook(v5 — 修复进度卡 56% 的 P1 bug)
*
* startPolling(taskId) 轮询单个任务;
* startPollingBatch(tasks) 并行轮询 N 个任务:
* - 每个任务独立进度/状态/失败,通过 onBatchTaskUpdate 实时回传
* - 全部成功才 onComplete(聚合视频按变体顺序);任一失败不影响其他任务继续
* - retryTask(taskId) 单独重试失败任务(重新轮询,后端任务仍在跑则直接接续)
* v5 修复点(2026-09-27):
* 1. pending/queued/unknown 状态兼容:旧版只处理 completed/awaiting_cover/failed/cancelled/running,
* 任务刚入队(pending)或后端新增状态时会走到 fallback 分支,但 progress 可能停在上一次值,
* 视觉上"卡"在 56%。新版本 pending 显示 ≤5%,unknown 保持上次进度继续轮询,不阻断。
* 2. 网络/5xx 错误无限重试+指数退避:旧版连续 10 次错误就终止轮询。新版本只在 4xx(任务不存在/
* 未授权)时终止,其他错误指数退避重试,连续错误达到阈值弹一次"网络不稳定"提示。
* 3. 页面可见性恢复主动 poll:浏览器后台 tab 会被节流 setTimeout,用户切回标签立即触发一次 poll,
* 无需等下一个定时器,感知"秒跳"到完成状态。
* 4. 进度停滞探测:若 STALL_PROBE_MS 内进度无变化但任务仍 running,缩短到 3s 主动再查一次,
* 防止后端最后一次 progress/status 更新丢失。
* 5. unmount 时清理所有定时器:useEffect cleanup 兜底,避免组件卸载后残留定时器触发 setState 警告。
* 6. 终态前强制推 100%:completed/awaiting_cover 分支先 onProgress(100) 再 onComplete,避免 UI 停在 95%。
* 7. 终态 results 接口抖动兜底:awaiting_cover 时 results 可能因 write-after-read 返回空,
* 补一次 1.5s 后的重试再判定失败。
*/
export function useGenerationPolling({
onProgress,
@@ -49,21 +110,93 @@ export function useGenerationPolling({
const cancelledRef = useRef(false)
/** 批量任务上下文:taskId → 变体序号 */
const batchContextRef = useRef<Map<string, number>>(new Map())
/** 当前单任务轮询的 taskId(供 visibilitychange 快速调度) */
const activeTaskRef = useRef<string | null>(null)
/** 当前批量轮询的 tasks(供 visibilitychange 全量触发一次) */
const activeBatchRef = useRef<{ taskId: string; variantIndex: number }[]>([])
/** 批量下每个 task 独立的调度控制对象 */
const batchControllersRef = useRef<
Map<
string,
{
done: boolean
schedule: (delay: number) => void
}
>
>(new Map())
/** 单任务 controller(供 visibilitychange 使用) */
const singleTaskScheduleRef = useRef<{ done: boolean; schedule: (delay: number) => void } | null>(
null,
)
const [, forceTick] = useState(0)
const clearTimer = useCallback(() => {
cancelledRef.current = true
const _clearAllTimers = useCallback(() => {
progressTimer.current.forEach((t) => clearTimeout(t))
progressTimer.current = []
}, [])
/** 任务完成后拉取结果列表,带重试 */
const clearTimer = useCallback(() => {
cancelledRef.current = true
_clearAllTimers()
batchControllersRef.current.clear()
singleTaskScheduleRef.current = null
activeTaskRef.current = null
activeBatchRef.current = []
}, [_clearAllTimers])
/** unmount 兜底清理 */
useEffect(() => {
const controllers = batchControllersRef
const singleCtrl = singleTaskScheduleRef
return () => {
cancelledRef.current = true
_clearAllTimers()
controllers.current.clear()
singleCtrl.current = null
}
}, [_clearAllTimers])
/** 页面从后台切回时,立即触发一次 poll(避免浏览器后台 tab 节流导致的延迟) */
useEffect(() => {
const handleVisibility = () => {
if (document.visibilityState !== "visible") return
if (cancelledRef.current) return
// 单任务
if (
activeTaskRef.current &&
singleTaskScheduleRef.current &&
!singleTaskScheduleRef.current.done
) {
const t = setTimeout(() => {
singleTaskScheduleRef.current?.schedule(0)
}, 0)
progressTimer.current.push(t)
}
// 批量
activeBatchRef.current.forEach(({ taskId }) => {
const ctrl = batchControllersRef.current.get(taskId)
if (ctrl && !ctrl.done) {
const t = setTimeout(() => ctrl.schedule(0), 0)
progressTimer.current.push(t)
}
})
}
document.addEventListener("visibilitychange", handleVisibility)
return () => document.removeEventListener("visibilitychange", handleVisibility)
}, [])
/** 任务完成后拉取结果列表,带重试(网络/5xx 重试,4xx 直接放弃) */
const fetchResultsWithRetry = useCallback(
async (taskId: string, attempt = 0): Promise<unknown[] | null> => {
try {
return await getGenerationTaskResults(taskId)
} catch (err) {
if (cancelledRef.current) return null
const status = axios.isAxiosError(err) ? err.response?.status : undefined
if (status && status >= 400 && status < 500) {
console.warn(`[获取结果4xx,放弃重试] taskId=${taskId} status=${status}`)
return null
}
console.error(`[获取生成结果失败] 第 ${attempt + 1} 次`, err)
if (attempt < MAX_RESULTS_RETRIES - 1) {
await new Promise((resolve) => setTimeout(resolve, 1000 * (attempt + 1)))
@@ -87,8 +220,6 @@ export function useGenerationPolling({
/**
* 轮询单个任务。
* - isBatch=true:状态变化通过 onBatchTaskUpdate 回传,不触发整体 onProgress/onComplete
* - resolve(videos) 成功;reject(Error) 失败
*/
const pollSingleTask = useCallback(
(
@@ -101,8 +232,54 @@ export function useGenerationPolling({
},
): Promise<unknown[]> => {
return new Promise((resolve, reject) => {
let consecutiveErrors = 0
let done = false
let consecutiveErrors = 0
let lastProgress = 0
let lastProgressAt = Date.now()
let hasNoticedNetworkError = false
const finish = (videos: unknown[], taskStatus: "completed" | "awaiting_cover") => {
if (done) return
done = true
callbacks?.onTaskProgress?.(100)
if (!callbacks && runId === 0) {
onProgress(100)
}
callbacks?.onTaskCompleted?.(videos, taskStatus)
resolve(videos)
}
const fail = (msg: string) => {
if (done) return
done = true
callbacks?.onTaskFailed?.(msg)
reject(new Error(msg))
}
const scheduleNext = (delay: number) => {
if (cancelledRef.current || done) return
const timeSinceProgress = Date.now() - lastProgressAt
// 停滞探测:长时间进度不变且是正常轮询间隔,缩短到 3s 主动再查一次
const actualDelay =
delay === BASE_POLL_INTERVAL_MS && timeSinceProgress > STALL_PROBE_MS ? 3000 : delay
const timer = setTimeout(poll, actualDelay)
progressTimer.current.push(timer)
}
// 暴露 schedule 给 visibilitychange 使用
const scheduleApi = {
get done() {
return done
},
schedule: (d: number) => {
if (!done) scheduleNext(d)
},
}
if (!callbacks && runId === 0) {
singleTaskScheduleRef.current = scheduleApi
} else {
batchControllersRef.current.set(taskId, scheduleApi)
}
const poll = async () => {
if (cancelledRef.current || done) return
@@ -111,65 +288,75 @@ export function useGenerationPolling({
if (cancelledRef.current || done) return
consecutiveErrors = 0
if (task.status === "completed" || task.status === "awaiting_cover") {
done = true
const videos = await fetchResultsWithRetry(taskId)
const status = normalizeTaskStatus(task.status)
if (status === "completed" || status === "awaiting_cover") {
let videos = await fetchResultsWithRetry(taskId)
if (cancelledRef.current) return
if (videos === null) {
const msg = "视频已生成,但获取结果列表失败,请稍后在任务列表查看"
callbacks?.onTaskFailed?.(msg)
reject(new Error(msg))
// results 为空时补一次短延迟重试(write-after-read 抖动)
if ((!videos || videos.length === 0) && status === "awaiting_cover") {
await new Promise((r) => setTimeout(r, 1500))
if (cancelledRef.current || done) return
videos = await fetchResultsWithRetry(taskId)
}
if (videos === null || videos.length === 0) {
const msg = "视频已生成,但获取结果列表失败,请刷新页面或稍后在任务列表查看"
fail(msg)
return
}
callbacks?.onTaskCompleted?.(videos, task.status as "completed" | "awaiting_cover")
resolve(videos)
finish(videos, status as "completed" | "awaiting_cover")
return
}
if (task.status === "failed" || task.status === "cancelled") {
done = true
if (status === "failed" || status === "cancelled") {
const rawMsg =
task.error_info?.error_message ||
task.error_message ||
(task.status === "cancelled" ? "任务已取消" : "视频生成失败,请联系管理员或重试")
(status === "cancelled" ? "任务已取消" : "视频生成失败,请联系管理员或重试")
const msg = safeExtractError(rawMsg)
callbacks?.onTaskFailed?.(msg)
reject(new Error(msg))
fail(msg)
return
}
const pct = Math.max(0, Math.min(99, Math.round(Number(task.progress) || 0)))
// running / pending / unknown:继续轮询
const rawPct = Number(task.progress) || 0
const pct =
status === "pending"
? Math.max(0, Math.min(5, Math.round(rawPct)))
: Math.max(0, Math.min(99, Math.round(rawPct)))
if (pct !== lastProgress) {
lastProgress = pct
lastProgressAt = Date.now()
}
callbacks?.onTaskProgress?.(pct)
if (!callbacks && runId === 0) {
onProgress(pct)
}
const timer = setTimeout(poll, 2000)
progressTimer.current.push(timer)
scheduleNext(BASE_POLL_INTERVAL_MS)
} catch (pollErr) {
if (cancelledRef.current || done) return
console.error("[轮询出错] taskId:", taskId, pollErr)
const status = axios.isAxiosError(pollErr) ? pollErr.response?.status : undefined
if (status && status >= 400 && status < 500) {
done = true
const msg = extractErrorMessage(pollErr, status)
callbacks?.onTaskFailed?.(msg)
reject(new Error(msg))
fail(msg)
return
}
// 网络/5xx:指数退避重试,永不终止
consecutiveErrors += 1
if (consecutiveErrors >= MAX_RETRYABLE_ERRORS) {
done = true
const msg = "任务状态查询连续失败,请稍后在任务列表查看结果"
callbacks?.onTaskFailed?.(msg)
reject(new Error(msg))
return
if (consecutiveErrors >= NETWORK_ERROR_NOTICE_THRESHOLD && !hasNoticedNetworkError) {
hasNoticedNetworkError = true
message.warning("网络不稳定,正在继续查询任务状态…")
}
const timer = setTimeout(poll, 3000)
progressTimer.current.push(timer)
const backoff = Math.min(
NETWORK_ERROR_BASE_MS * Math.pow(1.5, Math.min(consecutiveErrors - 1, 6)),
NETWORK_ERROR_MAX_MS,
)
scheduleNext(backoff)
}
}
const timer = setTimeout(poll, 1500)
const timer = setTimeout(poll, INITIAL_DELAY_MS)
progressTimer.current.push(timer)
})
},
@@ -180,38 +367,49 @@ export function useGenerationPolling({
const startPolling = useCallback(
(taskId: string) => {
cancelledRef.current = false
_clearAllTimers()
batchContextRef.current.clear()
batchControllersRef.current.clear()
activeTaskRef.current = taskId
activeBatchRef.current = []
singleTaskScheduleRef.current = null
pollSingleTask(taskId, 0)
.then((videos) => {
activeTaskRef.current = null
singleTaskScheduleRef.current = null
if (cancelledRef.current) return
onProgress(100)
onComplete(videos)
message.success("视频生成完成!")
})
.catch((err: Error) => {
activeTaskRef.current = null
singleTaskScheduleRef.current = null
if (cancelledRef.current) return
console.error("[生成失败] taskId:", taskId, err.message)
onFailed(err.message)
message.error(err.message)
})
},
[pollSingleTask, onProgress, onComplete, onFailed],
[pollSingleTask, onProgress, onComplete, onFailed, _clearAllTimers],
)
/**
* 批量多任务轮询:
* - 每个任务独立进度/状态回传 onBatchTaskUpdate
* * 全部完成后按变体顺序聚合视频 onComplete
* - 部分失败:整体不 onFailed(第5步逐卡片展示失败+重试按钮);全部失败才 onFailed
* 批量多任务轮询
*/
const startPollingBatch = useCallback(
(tasks: { taskId: string; variantIndex: number }[]) => {
cancelledRef.current = false
_clearAllTimers()
batchControllersRef.current.clear()
singleTaskScheduleRef.current = null
const runId = Date.now()
const progressMap = new Map<string, number>()
const resultMap = new Map<string, unknown[]>()
const failureMap = new Map<string, string>()
batchContextRef.current = new Map(tasks.map((t) => [t.taskId, t.variantIndex]))
activeBatchRef.current = [...tasks]
activeTaskRef.current = null
const reportAggregateProgress = () => {
if (cancelledRef.current) return
@@ -228,7 +426,6 @@ export function useGenerationPolling({
onComplete(ordered)
message.success(`全部 ${tasks.length} 个视频生成完成!`)
} else if (resultMap.size > 0) {
// 部分失败:成功的视频聚合进成片列表(可进封面),失败卡片带重试按钮
onProgress(100)
const ordered = tasks
.filter((t) => resultMap.has(t.taskId))
@@ -273,14 +470,14 @@ export function useGenerationPolling({
checkAllSettled()
},
}).catch(() => {
// 失败已在 onTaskFailed 处理,这里吞掉 Promise rejection
// 失败已在 onTaskFailed 处理
})
})
},
[pollSingleTask, onProgress, onComplete, onFailed, onBatchTaskUpdate],
[pollSingleTask, onProgress, onComplete, onFailed, onBatchTaskUpdate, _clearAllTimers],
)
/** 单独重试失败任务(第5步卡片「重试此视频」):先调后端重试接口,再轮询 */
/** 单独重试失败任务(第5步卡片「重试此视频」) */
const retryTask = useCallback(
async (taskId: string) => {
if (cancelledRef.current) cancelledRef.current = false
@@ -289,9 +486,9 @@ export function useGenerationPolling({
try {
await retryGenerationTaskApi(taskId)
} catch (err) {
// 后端不支持重试或任务不可重试:直接重新轮询(任务可能已被自动恢复)
console.warn("[重试任务接口调用失败,改为直接轮询]", err)
}
batchControllersRef.current.delete(taskId)
pollSingleTask(taskId, Date.now(), {
onTaskProgress: (pct) => onBatchTaskUpdate?.(taskId, { status: "running", progress: pct }),
onTaskCompleted: (videos, taskStatus) => {
View File
@@ -1,112 +0,0 @@
"""一次性脚本:对历史 quality_score 缺失的视频素材重新打分。
背景(#2073):镜像 97ad0ae2 时期 calculate_quality_score / classify_from_analysis
返回 str 而非 AssetClassification 枚举,导致 calculate_asset_quality 连续报
"'str' object has no attribute 'value'",大量视频素材的 quality_score 卡在 NULL。
镜像 8abdeb95 已修复枚举 bug,但历史失败记录不会自动重跑。本脚本扫描全表,
把 quality_score IS NULL 的视频素材重新投递到 worker.calculate_asset_quality 任务。
使用方式(在 worker 容器内执行):
cd /app/apps/worker
# 干跑,只打印会重跑多少条,不发任务
python -m scripts.backfill_asset_quality --dry-run
# 正式执行
python -m scripts.backfill_asset_quality
# 只重跑最近 N 天的
python -m scripts.backfill_asset_quality --since-days 30
# 限流:每投递一批 sleep 几秒,避免瞬间打爆 transcode 队列
python -m scripts.backfill_asset_quality --batch-size 50 --sleep 2
也可以直接在 staging 机器上 exec 进容器:
docker exec -e PYTHONPATH=/app:/app/apps/api:/app/packages xiaoxia-worker-staging \
python -m scripts.backfill_asset_quality --dry-run
"""
from __future__ import annotations
import argparse
# 保证可以以 python -m scripts.xxx 在容器 /app/apps/worker 下执行
# 也兼容在 repo 根目录下执行(注入路径)
import os
import sys
import time
from datetime import UTC, datetime, timedelta
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
_WORKER_DIR = os.path.dirname(_SCRIPT_DIR) # apps/worker
_APPS_DIR = os.path.dirname(_WORKER_DIR) # apps
_REPO_ROOT = os.path.dirname(_APPS_DIR) # repo root
for p in (_REPO_ROOT, os.path.join(_REPO_ROOT, "apps", "api"), _REPO_ROOT):
if p not in sys.path:
sys.path.insert(0, p)
def main() -> int:
parser = argparse.ArgumentParser(description="补打历史视频素材 quality_score")
parser.add_argument("--dry-run", action="store_true", help="只统计数量,不投递任务")
parser.add_argument("--since-days", type=int, default=0, help="只处理最近 N 天上传的素材(0=全部)")
parser.add_argument("--batch-size", type=int, default=50, help="每批投递数量,默认 50")
parser.add_argument("--sleep", type=float, default=1.0, help="批次之间 sleep 秒数,默认 1s")
parser.add_argument("--queue", type=str, default="transcode", help="投递队列(默认 transcode)")
args = parser.parse_args()
# 延迟 import,避免在 dry-run 时依赖完整 DB 环境
from worker_app.celery_app import celery_app
from worker_app.db import SessionLocal
from packages.adapters.sqlalchemy_impl.models import AssetModel
db = SessionLocal()
try:
q = db.query(AssetModel).filter(
AssetModel.file_type == "video",
AssetModel.quality_score.is_(None),
)
if args.since_days > 0:
cutoff = datetime.now(UTC) - timedelta(days=args.since_days)
q = q.filter(AssetModel.created_at >= cutoff)
# 先 count 打印
total = q.count()
print(
f"[backfill] 待重跑 quality_score 的视频素材: {total} 条"
f"{' (dry-run,不投递)' if args.dry_run else ''}"
f"{' (最近 ' + str(args.since_days) + ' 天)' if args.since_days > 0 else ''}",
flush=True,
)
if total == 0 or args.dry_run:
return 0
# 分批投递
submitted = 0
batch = 0
offset = 0
while True:
assets = q.order_by(AssetModel.created_at.desc()).offset(offset).limit(args.batch_size).all()
if not assets:
break
batch += 1
for a in assets:
try:
celery_app.send_task(
"worker.calculate_asset_quality",
args=[a.id],
queue=args.queue,
)
submitted += 1
except Exception as e: # noqa: BLE001
print(f"[backfill] 投递失败 asset_id={a.id}: {e}", flush=True)
print(f"[backfill] batch {batch}: 已累计投递 {submitted}/{total}", flush=True)
offset += len(assets)
if args.sleep > 0 and offset < total:
time.sleep(args.sleep)
print(f"[backfill] 完成,共投递 {submitted} 条任务到 {args.queue} 队列", flush=True)
return 0
finally:
db.close()
if __name__ == "__main__":
sys.exit(main())
@@ -18,7 +18,6 @@ from worker_app.celery_app import celery_app
from worker_app.db import SessionLocal
from packages.adapters.sqlalchemy_impl.asset_repository import SQLAlchemyAssetRepository
from packages.domain.classification import ClassificationStatus
from packages.shared.storage import get_shared_storage_service
logger = get_task_logger(__name__)
@@ -97,7 +96,7 @@ def calculate_asset_quality_task(self, asset_id: str) -> dict:
confidence = 1.0
existing_meta["classification"] = classification
existing_meta["classification_confidence"] = confidence
asset.classification_status = ClassificationStatus.COMPLETED
asset.classification_status = "completed"
asset.metadata = existing_meta
logger.info(
"[quality_score] asset=%s 自动分类完成: category=%s confidence=%.2f",
@@ -111,8 +110,6 @@ def calculate_asset_quality_task(self, asset_id: str) -> dict:
asset_id,
cls_err,
)
# 分类失败显式标记 FAILED,避免停留在 PENDING 被反复重试
asset.classification_status = ClassificationStatus.FAILED
asset_repo.update(asset)
db.commit()
+72 -44
View File
@@ -10,25 +10,20 @@
# API_IMAGE - API 镜像名称 (默认: xiaoxia-saas-api:dev)
# WORKER_IMAGE - Worker 镜像名称 (默认: xiaoxia-saas-worker:dev)
# WEB_IMAGE - Web 镜像名称 (默认: xiaoxia-saas-web:dev)
# WEB_DOCKERFILE - Web Dockerfile 路径
# WEB_NGINX_CONF - Nginx 配置文件路径
# API_PORT - API 端口映射 (staging: 8000, production: 8001)
# WEB_PORT - Web 端口映射 (staging: 3001, production: 3002)
# GENERATED_FILES_HOST_DIR - 生成文件的主机目录
# GENERATION_CONCURRENCY - Generation worker 并发(用户实时任务,默认 2)
# TRANSCODE_CONCURRENCY - Transcode worker 并发(后台/转码/AI,默认 2)
# WORKER_CONCURRENCY - Worker 并发数 (默认: 4)
# WORKER_MAX_TASKS_PER_CHILD - Worker 每个子进程最大任务数 (默认: 100)
# BEAT_ENABLED - 容器内启动 celery beat(默认 1;独立 beat 容器部署设为 0)
# WORKER_CONCURRENCY - 兼容旧变量:未显式设置上面两个并发时按此总数分配
#
# 重要:
# 重要:
# - 生产环境不要挂载 web-dist volume,否则会导致 403
# - 确保环境隔离网络已创建: docker network create xiaoxia-net-${ENV}
# - ENV=staging -> xiaoxia-net-staging
# - ENV=production -> xiaoxia-net-production
# - ENV=staging → xiaoxia-net-staging
# - ENV=production → xiaoxia-net-production
#
# #2073 队列分流:worker 容器内跑三个独立进程——beat(只发定时任务)、
# generation worker(只消费 generation 队列,实时高优)、transcode worker(消费
# transcode + celery 队列,后台任务)。beat 不再嵌入 generation worker,
# 不占实时任务槽位;TRANSCODE_CONCURRENCY 独立伸缩,不再依赖 WORKER_CONCURRENCY 差值。
# ===========================================
# 日志轮转配置(所有服务共享)
@@ -45,39 +40,53 @@ services:
# =========================================
api:
image: ${API_IMAGE:-xiaoxia-saas-api:dev}
# 不在生产环境构建镜像,使用预构建的镜像
# build:
# context: ../..
# dockerfile: infra/docker/api.Dockerfile
container_name: xiaoxia-api-${ENV:-staging}
restart: unless-stopped
stop_grace_period: 30s
stop_signal: SIGTERM
# 环境变量文件(包含数据库密码等敏感信息)
env_file:
- ../../.env
environment:
APP_ENV: ${APP_ENV:-staging}
GENERATED_FILES_DIR: /app/generated
GENERATED_FILES_URL_PREFIX: /generated-files
PUBLIC_API_BASE_URL: ${PUBLIC_API_BASE_URL:-https://api.xiaoxiajianji.com}
# 端口映射
# Staging: 8000 -> 8000
# Production: 8001 -> 8000
ports:
- "127.0.0.1:${API_PORT:-8000}:8000"
# 共享生成文件目录 + 抖音 cookies 等运行时配置
volumes:
- generated-files:/app/generated
- ../../deploy/configs:/app/configs:ro
networks:
- xiaoxia-net
# 健康检查配置
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=5)"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
logging: *default-logging
# =========================================
# 资源限制建议(生产环境建议启用)
# =========================================
deploy:
resources:
limits:
@@ -88,48 +97,39 @@ services:
memory: 512M
# =========================================
# Worker 服务(#2073 队列分流:beat + generation + transcode 同容器三进程)
# Worker 服务(Celery 任务队列)
# =========================================
# 三个进程独立启动,任一退出则容器整体退出由 docker restart 拉起;
# 各自的并发与资源占用通过环境变量控制:
# - generation:GENERATION_CONCURRENCY(默认 2),消费 generation 队列
# - transcode: TRANSCODE_CONCURRENCY(默认 2),消费 transcode,celery 队列
# - beat: 不消费任务,只发定时任务到 celery 默认队列
worker:
image: ${WORKER_IMAGE:-xiaoxia-saas-worker:dev}
container_name: xiaoxia-worker-${ENV:-staging}
restart: unless-stopped
# 长任务(ingest HEVC 转码最长 30min、生成硬超时 11min)给足优雅关闭窗口
stop_grace_period: 300s
stop_signal: SIGTERM
env_file:
- ../../.env
environment:
APP_ENV: ${APP_ENV:-staging}
# 兼容旧变量:若两个 *_CONCURRENCY 均未显式设置,entrypoint 会按此总数分配
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-4}
WORKER_MAX_TASKS_PER_CHILD: ${WORKER_MAX_TASKS_PER_CHILD:-100}
# #2073 队列独立伸缩:generation 默认 2,transcode 默认 2(不再差值计算)
# #1714 队列隔离:generation 队列独占 worker(默认并发 2),其余并发给转码
GENERATION_CONCURRENCY: ${GENERATION_CONCURRENCY:-2}
TRANSCODE_CONCURRENCY: ${TRANSCODE_CONCURRENCY:-2}
# beat 默认在本容器启动;独立 beat 容器部署时设为 0
BEAT_ENABLED: ${BEAT_ENABLED:-1}
GENERATED_FILES_DIR: /app/generated
GENERATED_FILES_URL_PREFIX: /generated-files
PUBLIC_API_BASE_URL: ${PUBLIC_API_BASE_URL:-https://api.xiaoxiajianji.com}
volumes:
- generated-files:/app/generated
networks:
- xiaoxia-net
# 健康检查:至少有一个 celery worker 进程在跑(beat 本身不作为存活依据)
# 健康检查配置
# 注:容器内无 pgrep/ps,扫描 /proc 所有进程的 cmdline 查找 celery 进程
healthcheck:
test: ["CMD-SHELL", "grep -q 'celery.*worker' /proc/[0-9]*/cmdline 2>/dev/null || exit 1"]
test: ["CMD-SHELL", "grep -lq celery /proc/[0-9]*/cmdline 2>/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 3
@@ -137,8 +137,12 @@ services:
logging: *default-logging
# 资源限制:容器总资源 = gen + trans + beat,按 2+2 并发场景建议 4C8G;
# 后续如需独立扩容/重启,可拆为 worker-generation / worker-transcode / worker-beat 三个 service。
# =========================================
# 资源限制建议(生产环境建议启用)
# =========================================
# 注意: Worker 需要处理视频,建议分配更多资源
# #1714 队列隔离后容器内运行 generation + transcode 两个 worker 进程,
# 总并发 = WORKER_CONCURRENCY(默认 4),4C8G 以上确保视频渲染不 OOM
deploy:
resources:
limits:
@@ -153,21 +157,35 @@ services:
# =========================================
web:
image: ${WEB_IMAGE:-xiaoxia-saas-web:dev}
# 不在生产环境构建镜像,使用 web-artifact.Dockerfile
# build:
# context: ../..
# dockerfile: ${WEB_DOCKERFILE:-infra/docker/web.Dockerfile}
# args:
# (NGINX_CONF no longer needed - all configs baked into image)
container_name: xiaoxia-web-${ENV:-staging}
restart: unless-stopped
# 端口映射
# Staging: 3001 -> 80
# Production: 3002 -> 80 (通过 Nginx 反向代理)
ports:
- "127.0.0.1:${WEB_PORT:-3001}:80"
networks:
- xiaoxia-net
# =========================================
# Nginx 配置运行时覆盖(双保险:entrypoint 也按 APP_ENV 选择配置)
# 确保容器使用正确环境的 nginx 配置,即使镜像构建时使用了默认配置
# 注意: 只覆盖 /etc/nginx/conf.d/default.conf,不挂载 /usr/share/nginx/html
# =========================================
environment:
- APP_ENV=${ENV:-staging}
volumes:
- ./nginx-${ENV:-staging}.conf:/etc/nginx/conf.d/default.conf:ro
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://127.0.0.1:80"]
interval: 30s
@@ -176,6 +194,9 @@ services:
logging: *default-logging
# =========================================
# 资源限制建议
# =========================================
deploy:
resources:
limits:
@@ -191,6 +212,9 @@ volumes:
driver_opts:
type: none
o: bind
# 重要: 确保主机目录存在且有正确权限
# Staging: /var/lib/xiaoxia-saas-staging/generated
# Production: /var/lib/xiaoxia-saas-production/generated
device: ${GENERATED_FILES_HOST_DIR:?GENERATED_FILES_HOST_DIR must be set in .env}
# ===========================================
@@ -199,4 +223,8 @@ volumes:
networks:
xiaoxia-net:
external: true
# 网络名根据 ENV 变量区分,实现 staging/production 环境隔离
# staging: xiaoxia-net-staging
# production: xiaoxia-net-production
name: xiaoxia-net-${ENV:-staging}
+29 -67
View File
@@ -1,77 +1,48 @@
#!/bin/bash
# Worker 启动脚本 — #1714 + #2073 队列分流
# Worker 启动脚本 — #1714 队列隔离
#
# 容器内启动三个独立进程(任一退出则整体退出由 docker restart 拉起):
# 1. beat:celery beat 调度器,不消费任何任务,只发定时任务到 celery 默认队列
# 2. generation-worker:独占消费 generation 队列(用户实时任务,高优先级)
# 3. transcode-worker:消费 transcode + celery 默认队列(后台/清理任务)
# 部署约束:worker 容器单实例(replicas=1),容器内启动两个 celery 进程:
# 1. generation-worker:独占消费 generation 队列(用户视频生成,高优先级),
# 内嵌 celery beat(-B),定时清理任务只在一个进程里跑,避免重复执行;
# 2. transcode-worker:消费 transcode + celery 默认队列(素材转码/分类/查重/
# 配音/下载等后台任务)。
# 转码队列积压时,generation 队列仍有独立 worker 立即领取视频生成任务。
#
# 环境变量:
# WORKER_CONCURRENCY 总并发槽参考(默认 4);生成 worker 并发默认 2,
# 可用 GENERATION_CONCURRENCY 覆盖
# GENERATION_CONCURRENCY generation worker 并发(默认 2)
# TRANSCODE_CONCURRENCY transcode worker 并发(默认 2)
# TRANSCODE_CONCURRENCY transcode worker 并发(默认 = WORKER_CONCURRENCY - 2,最小 1)
# WORKER_MAX_TASKS_PER_CHILD 每个子进程最大任务数(默认 100)
# WORKER_CONCURRENCY 兼容旧变量:若未显式设置 GENERATION_CONCURRENCY /
# TRANSCODE_CONCURRENCY,则按比例分配(gen=ceil(total*1/2),
# trans=剩余,各至少 1);已显式设置时忽略此变量。
# BEAT_ENABLED 是否在本容器内启动 beat 进程(默认 1);
# 若独立 beat 容器部署设为 0。
set -e
CONCURRENCY="${WORKER_CONCURRENCY:-4}"
MAX_TASKS="${WORKER_MAX_TASKS_PER_CHILD:-100}"
# ── 并发计算:显式 env 优先;否则从 WORKER_CONCURRENCY 按比例推导 ──
if [ -n "$GENERATION_CONCURRENCY" ]; then
GEN_CONCURRENCY="$GENERATION_CONCURRENCY"
else
TOTAL="${WORKER_CONCURRENCY:-4}"
GEN_CONCURRENCY=$(( (TOTAL + 1) / 2 ))
if [ "$GEN_CONCURRENCY" -lt 1 ]; then GEN_CONCURRENCY=1; fi
fi
if [ -n "$TRANSCODE_CONCURRENCY" ]; then
TRANS_CONCURRENCY="$TRANSCODE_CONCURRENCY"
else
if [ -n "$WORKER_CONCURRENCY" ] && [ -z "$GENERATION_CONCURRENCY" ]; then
# 两个都没显式设置,按 WORKER_CONCURRENCY 分配剩余
TOTAL="$WORKER_CONCURRENCY"
TRANS_CONCURRENCY=$(( TOTAL - GEN_CONCURRENCY ))
if [ "$TRANS_CONCURRENCY" -lt 1 ]; then TRANS_CONCURRENCY=1; fi
else
# 默认 2(#2073:独立伸缩,不再依赖 WORKER_CONCURRENCY 差值)
TRANS_CONCURRENCY=2
GEN_CONCURRENCY="${GENERATION_CONCURRENCY:-2}"
if [ -z "$TRANSCODE_CONCURRENCY" ]; then
TRANS_CONCURRENCY=$((CONCURRENCY - GEN_CONCURRENCY))
if [ "$TRANS_CONCURRENCY" -lt 1 ]; then
TRANS_CONCURRENCY=1
fi
else
TRANS_CONCURRENCY="$TRANSCODE_CONCURRENCY"
fi
BEAT_ENABLED="${BEAT_ENABLED:-1}"
PIDS=()
# ── 1. Beat 调度器(独立进程,不消费任务)──
if [ "$BEAT_ENABLED" = "1" ] || [ "$BEAT_ENABLED" = "true" ]; then
echo "Starting beat scheduler (schedule file=/tmp/celerybeat-schedule)"
celery \
-A worker_app.celery_app \
beat \
--loglevel=info \
-s /tmp/celerybeat-schedule &
PIDS+=($!)
fi
# ── 2. Generation worker(实时高优队列)──
echo "Starting generation worker (queue=generation, concurrency=$GEN_CONCURRENCY)"
echo "Starting generation worker (queue=generation, concurrency=$GEN_CONCURRENCY, beat embedded)"
celery \
-A worker_app.celery_app \
worker \
--loglevel=info \
"-B" \
-s /tmp/celerybeat-schedule \
-Q generation \
"--concurrency=${GEN_CONCURRENCY}" \
"--max-tasks-per-child=${MAX_TASKS}" \
-n generation@%h &
PIDS+=($!)
GEN_PID=${PIDS[1]:-${PIDS[0]}}
GEN_PID=$!
# ── 3. Transcode worker(后台 + 清理队列)──
echo "Starting transcode worker (queues=transcode,celery, concurrency=$TRANS_CONCURRENCY)"
celery \
-A worker_app.celery_app \
@@ -81,22 +52,13 @@ celery \
"--concurrency=${TRANS_CONCURRENCY}" \
"--max-tasks-per-child=${MAX_TASKS}" \
-n transcode@%h &
PIDS+=($!)
TRANS_PID=${PIDS[2]:-${PIDS[1]}}
TRANS_PID=$!
# 任一进程退出则终止其他进程,让容器整体重启
cleanup() {
echo "Shutting down all celery processes..."
for pid in "${PIDS[@]}"; do
kill -TERM "$pid" 2>/dev/null || true
done
}
trap cleanup TERM INT
# 任一进程退出则终止另一个,让容器整体重启(restart: unless-stopped)
trap 'echo "Shutting down workers..."; kill -TERM $GEN_PID $TRANS_PID 2>/dev/null || true' TERM INT
# wait -n 等待任意一个子进程退出(bash 4.3+)
# 容器镜像基础为 python:3.11-slim,bash 版本满足
wait -n "${PIDS[@]}"
wait -n $GEN_PID $TRANS_PID
EXIT_CODE=$?
echo "One celery process exited (code=$EXIT_CODE), stopping the rest..."
cleanup
exit "$EXIT_CODE"
echo "One worker exited (code=$EXIT_CODE), stopping the other..."
kill -TERM $GEN_PID $TRANS_PID 2>/dev/null || true
exit $EXIT_CODE
@@ -128,12 +128,8 @@ class SQLAlchemyAssetRepository:
height=asset.height,
fps=asset.fps,
codec=asset.codec,
status=(asset.status.value if hasattr(asset.status, "value") else str(asset.status)),
classification_status=(
asset.classification_status.value
if hasattr(asset.classification_status, "value")
else str(asset.classification_status)
),
status=asset.status.value,
classification_status=asset.classification_status.value,
classification_result=(json.dumps(asset.metadata) if asset.metadata else None),
quality_score=asset.quality_score,
uploaded_by_user_id=asset.uploaded_by_user_id or "system",
@@ -146,7 +142,7 @@ class SQLAlchemyAssetRepository:
self.session.flush()
self._sync_asset_tags(asset.id, asset.tag_ids)
# Issue #1776: 自动维护素材库计数(同事务内原子更新)
if asset.library_id and (getattr(asset.status, "value", str(asset.status)) != "deleted"):
if asset.library_id and asset.status.value != "deleted":
from sqlalchemy import func
self.session.query(AssetLibraryModel).filter(AssetLibraryModel.id == asset.library_id).update(
@@ -172,12 +168,8 @@ class SQLAlchemyAssetRepository:
model.height = asset.height
model.fps = asset.fps
model.codec = asset.codec
model.status = asset.status.value if hasattr(asset.status, "value") else str(asset.status)
model.classification_status = (
asset.classification_status.value
if hasattr(asset.classification_status, "value")
else str(asset.classification_status)
)
model.status = asset.status.value
model.classification_status = asset.classification_status.value
model.classification_result = json.dumps(asset.metadata) if asset.metadata else None
model.quality_score = asset.quality_score
model.uploaded_by_user_id = asset.uploaded_by_user_id or model.uploaded_by_user_id
+15 -45
View File
@@ -1,14 +1,14 @@
"""Celery 队列定义与路由配置(API / Worker 共享)。
#1714 + #2073 队列分流:用户同步等待的实时任务路由到 `generation` 高优队列,
由专用 generation worker 独占消费;素材入库/转码/AI 分析/查重等后台批量任务路由
到 `transcode` 队列;beat 定时清理等轻量维护任务走默认 `celery` 队列。
transcode / celery 队列积压时,generation 队列仍能被立即领取,不阻塞用户实时链路。
#1714 队列隔离:用户等待的视频生成任务路由到高优先级 `generation` 队列,
由专用 worker 进程独占消费;素材入库/转码等后台批量任务路由到 `transcode`
队列;其余杂项任务走默认 `celery` 队列。转码队列积压时,视频生成任务
仍能被 generation worker 立即领取执行,不会排队。
队列说明:
- generation: 用户同步等待的实时任务(视频生成、TTS、音色克隆、lipsync、AI 数字人、人声/背景提取)
- transcode: 后台批量/异步任务(素材入库转码、AI 分类打标、质量评分、原子切片、查重、批量下载/缩略图)
- celery: beat 定时巡检/清理等轻量维护任务(极短、低优、不占业务槽)
- generation: 用户提交的视频生成/预览渲染(延迟敏感,资源消耗大)
- transcode: 素材入库(HEVC 转码)、AI 分类、素材查重(批量、可排队)
- celery(默认): 配音、语音、下载缩略图、定时清理等杂项
"""
from __future__ import annotations
@@ -20,9 +20,8 @@ QUEUE_GENERATION = "generation"
QUEUE_TRANSCODE = "transcode"
QUEUE_DEFAULT = "celery"
# 三个消费组各自消费的队列列表(顺序即优先级:高优队列排在前面)
WORKER_QUEUES_GENERATION = (QUEUE_GENERATION,)
WORKER_QUEUES_TRANSCODE = (QUEUE_TRANSCODE, QUEUE_DEFAULT)
# Worker 消费的队列列表(顺序即优先级:高优队列排在前面)
WORKER_QUEUES = (QUEUE_GENERATION, QUEUE_TRANSCODE, QUEUE_DEFAULT)
# 队列声明:持久化队列,broker 重启不丢消息
task_queues = (
@@ -32,45 +31,15 @@ task_queues = (
)
# ── 任务路由表:task name → 队列 ──
# 键支持 celery 标准通配符。所有生产端(API send_task / worker 内 send_task)
# 未显式指定 queue 时按此表路由;漏配会走默认队列 celery,被 transcode worker 消费。
# 新增实时任务务必在此表显式路由到 generation,避免落到后台队列排队。
# 键支持 celery 标准通配符。
task_routes = {
# ── 高优先级:用户同步等待的实时链路 ──
# 视频生成(主链路)
# 高优先级:用户等待的视频生成
"worker.generate_video": {"queue": QUEUE_GENERATION},
# TTS 合成 / 片段合成(配音页、视频生成配乐/TTS 链路)
"worker.process_tts_synthesis": {"queue": QUEUE_GENERATION},
"worker.process_tts_segment_synthesis": {"queue": QUEUE_GENERATION},
# 音色克隆(用户主动上传样本等待克隆完成)
"worker.process_voice_clone": {"queue": QUEUE_GENERATION},
# 人声/背景提取(音色克隆前置步骤,用户同步等待)
"worker.extract_voice": {"queue": QUEUE_GENERATION},
"worker.extract_background": {"queue": QUEUE_GENERATION},
# AI 数字人渲染(用户主动触发,等待成片)
"ai_avatar_render.execute": {"queue": QUEUE_GENERATION},
# GPU MuseTalk 口型同步(用户等成片,链路子任务全部走 generation 避免跨队列阻塞)
"lipsync_gpu_process_async": {"queue": QUEUE_GENERATION},
"lipsync_tts.synthesize_and_submit": {"queue": QUEUE_GENERATION},
"lipsync_tts.poll_mediakit_status": {"queue": QUEUE_GENERATION},
"lipsync_tts.persist_output_video": {"queue": QUEUE_GENERATION},
# ── 后台批量:素材入库/转码 + AI 分析/打标 + 查重,积压不影响生成 ──
# 后台批量:素材入库/转码 + AI 分类 + 素材查重,积压不影响生成
"worker.ingest_asset": {"queue": QUEUE_TRANSCODE},
"worker.classify_asset": {"queue": QUEUE_TRANSCODE},
"worker.calculate_asset_quality": {"queue": QUEUE_TRANSCODE},
"worker.generate_atom_clips": {"queue": QUEUE_TRANSCODE},
"worker.tag_atom_clip": {"queue": QUEUE_TRANSCODE},
"worker.backfill_atom_clip_tags": {"queue": QUEUE_TRANSCODE},
"worker.process_duplication_check": {"queue": QUEUE_TRANSCODE},
"worker.check_duplicate": {"queue": QUEUE_TRANSCODE},
"worker.batch_download_videos": {"queue": QUEUE_TRANSCODE},
"worker.batch_generate_thumbnails": {"queue": QUEUE_TRANSCODE},
# ── beat 定时清理/巡检任务走默认 celery 队列(由 transcode worker 消费)──
# 未在此表显式列出的 cleanup 任务会落到默认队列 celery,不占 generation 槽位。
"worker.cleanup_stale_pending_tasks": {"queue": QUEUE_DEFAULT},
"worker.cleanup_stale_running_tasks": {"queue": QUEUE_DEFAULT},
"worker.cleanup_stale_ingest_jobs": {"queue": QUEUE_DEFAULT},
"worker.cleanup_stale_voice_clones": {"queue": QUEUE_DEFAULT},
}
# 生成任务的预取数:渲染是长任务,预取 1 避免任务被某个 worker 占住不调度
@@ -78,10 +47,11 @@ GENERATION_WORKER_PREFETCH_MULTIPLIER = 1
def apply_queue_settings(app) -> None:
"""把队列分流配置应用到 Celery app(API 生产端与 Worker 消费端都要调用)。
"""把队列隔离配置应用到 Celery app(API 生产端与 Worker 消费端都要调用)。
配置 task_queues / task_routes / task_default_queue。生产端靠 task_routes
把消息投递到对应队列;消费端靠启动参数 -Q 控制自己消费哪些队列(entrypoint)。
把消息投递到对应队列;消费端靠 task_queues 声明自己消费哪些队列
(实际消费集由启动参数 -Q 控制)。
"""
app.conf.task_queues = task_queues
app.conf.task_routes = task_routes
+11 -97
View File
@@ -7,21 +7,13 @@
3. 生成 relay 一次性 key,构造两个带 token 的 URL:
- put_url:给 P4000 回传结果,走 relay_base_url(Tailscale host:8092)
- get/del_url:worker 自己下载+清理用,走 relay_internal_base_url(Docker DNS 直连 API)
4. 【冷启动防护】距上次成功通信 >60s 时,先 GET /health 预热 Tailscale 链路(短超时快速失败)
5. POST P4000 /api/render/sync:inputs={"in.mp4": "<mezzanine-get-url>"}, output_url="<put_url>"
4. POST P4000 /api/render/sync:inputs={"in.mp4": "<mezzanine-get-url>"}, output_url="<put_url>"
ffmpeg_args: -i in.mp4 [-vf <vf>] -c:v h264_nvenc ... -an/-c:a aac -f mp4 pipe:1
- 首字节用短超时(默认20s),避免链路卡死空等上百秒;首字节到达后放宽到 ffmpeg_timeout+60s
6. P4000 从 relay GET mezzanine → h264_nvenc 编码 → PUT 最终 mp4 到 put_url
7. 本客户端通过 get_url(Docker 内网)下载最终文件到 output_path,然后 DELETE 清理
8. 删除 relay 上的 mezzanine 临时文件(以及 OSS fallback 的 key)
5. P4000 从 relay GET mezzanine → h264_nvenc 编码 → PUT 最终 mp4 到 put_url
6. 本客户端通过 get_url(Docker 内网)下载最终文件到 output_path,然后 DELETE 清理
7. 删除 relay 上的 mezzanine 临时文件(以及 OSS fallback 的 key)
任何环节失败抛 GpuEncodeError,调用方应 fallback 到 CPU libx264。
冷启动/链路卡顿背景(2026-09-27 实测):P4000 与 staging 之间走 Tailscale,长时间空闲
(>7h)后首次请求曾出现 150s 延迟才真正开始下载 mezzanine,期间 ffmpeg 尚未启动、GPU 空闲。
根因在服务端/网络层(可能是 Tailscale DERP 打洞或 httpx 连接池重建),本客户端通过
pre_warm + 首字节短超时做兜底:预热打通链路 + 20s 内收不到首字节就快速失败让 CPU fallback,
不再让用户等满 150s+。
"""
from __future__ import annotations
@@ -71,13 +63,6 @@ class GpuEncoderClient:
mezzanine_transport: str = "relay",
sync_timeout: int = 300,
health_timeout: float = 3.0,
# 提交编码任务前先发一次 /health 预热 Tailscale 链路,避免长时间空闲后首次请求
# 因 DERP 打洞/NAT 映射过期/Tailscale 连接重建而阻塞上百秒。
pre_warm: bool = True,
# POST 首次响应超时:P4000 已收到请求后应该在数秒内开始下载 inputs;
# 如果超过这个值还没收到任何响应字节,说明链路/服务卡住,快速失败让调用方 fallback CPU。
# 注意:ffmpeg 编码本身靠 body.timeout 控制(300s),不应该被这个超时影响。
post_first_byte_timeout: float = 20.0,
vcodec: str = "h264_nvenc",
preset: str = "p4",
crf: int = 23,
@@ -95,16 +80,12 @@ class GpuEncoderClient:
self.mezzanine_transport = mezzanine_transport.lower() # "relay" | "oss"
self.sync_timeout = sync_timeout
self.health_timeout = health_timeout
self.pre_warm = pre_warm
self.post_first_byte_timeout = post_first_byte_timeout
self.vcodec = vcodec
self.preset = preset
self.crf = crf
self.bitrate = bitrate
self._relay_secret = relay_secret
self.oss_tmp_prefix = oss_tmp_prefix.rstrip("/") + "/" if oss_tmp_prefix else "tmp/gpu-mezzanine/"
# 上次与 P4000 成功通信的时间戳(用于判断是否需要 pre_warm 预热)
self._last_ok_ts: float = 0.0
RELAY_PATH_PREFIX = "/api/v1/internal/gpu-relay"
@@ -147,16 +128,13 @@ class GpuEncoderClient:
except (urllib.error.URLError, socket.timeout, TimeoutError, json.JSONDecodeError, ConnectionError) as e:
return GpuHealth(healthy=False, error=f"health probe failed: {e}")
try:
h = GpuHealth(
return GpuHealth(
healthy=data.get("status") == "healthy",
worker=str(data.get("worker", "")),
gpu_name=(data.get("gpu") or {}).get("name", ""),
nvenc_h264=bool((data.get("nvenc") or {}).get("h264_nvenc")),
nvenc_hevc=bool((data.get("nvenc") or {}).get("hevc_nvenc")),
)
if h.healthy:
self._last_ok_ts = time.time()
return h
except Exception as e: # noqa: BLE001
return GpuHealth(healthy=False, error=f"malformed health response: {e}")
@@ -249,8 +227,7 @@ class GpuEncoderClient:
ffmpeg_args.append("-an")
ffmpeg_args.extend(["-f", "mp4", "pipe:1"])
# 4. pre-warm then call P4000 sync render
self._warm_up_if_needed()
# 4. call P4000 sync render
body = {
"inputs": {"in.mp4": input_url},
"ffmpeg_args": ffmpeg_args,
@@ -258,7 +235,6 @@ class GpuEncoderClient:
"timeout": int(timeout),
}
job = self._post_sync(body)
self._last_ok_ts = time.time()
logger.info(
"[gpu-encoder] P4000 done: job_id=%s rc=%s size=%s dur=%ss transport=%s",
job.get("job_id"),
@@ -323,38 +299,9 @@ class GpuEncoderClient:
raise GpuEncodeError("GPU_ENCODE_RELAY_SECRET not set")
return secret
def _warm_up_if_needed(self) -> None:
"""POST 前预热:如果距上次成功通信超过 idle 阈值,先打 /health 打通 Tailscale 链路。
背景:Tailscale 在长时间空闲(几小时)后,到对端的直连 NAT 映射可能过期,
首次请求会走 DERP 中继打洞;极少数情况下打洞/重连会卡住上百秒(曾观测到 150s 延迟)。
预热请求本身走短超时快速失败,不会阻塞主流程;预热成功后再发 POST。
"""
if not self.pre_warm:
return
idle = time.time() - self._last_ok_ts
# 空闲超过 60s 才预热(正常流水线里相邻任务间隔通常 <10s,没必要每次都打)
if idle < 60:
return
url = f"{self.endpoint}/health"
t0 = time.time()
try:
with urllib.request.urlopen(url, timeout=min(self.health_timeout, 3.0)) as resp:
resp.read()
self._last_ok_ts = time.time()
logger.debug("[gpu-encoder] pre-warm ok: took=%.2fs idle=%.0fs", time.time() - t0, idle)
except (urllib.error.URLError, socket.timeout, TimeoutError, ConnectionError, OSError) as e:
# 预热失败不致命——主 POST 会带自己的超时,再失败就抛 GpuEncodeError 让调用方 fallback
logger.warning("[gpu-encoder] pre-warm probe failed (will try POST anyway): %s", e)
def _post_sync(self, body: dict[str, Any]) -> dict[str, Any]:
url = f"{self.endpoint}/api/render/sync"
ffmpeg_timeout = body.get("timeout", self.sync_timeout)
# 连接 + 首字节用短超时(防链路卡死数百秒);首字节到达后给 ffmpeg 留足编码+上传时间
# Python urllib 的 timeout 是整个请求总超时,所以用"两段式":
# 阶段1:先 read(1) 拿首字节,用短超时;
# 阶段2:再 read() 读完整 body,用 ffmpeg_timeout+60。
connect_timeout = min(max(self.post_first_byte_timeout, 5.0), 30.0)
req_timeout = body.get("timeout", self.sync_timeout) + 60
payload = json.dumps(body).encode("utf-8")
req = urllib.request.Request(
url,
@@ -363,45 +310,14 @@ class GpuEncoderClient:
method="POST",
)
t0 = time.time()
first_byte_ok = False
resp = None
try:
resp = urllib.request.urlopen(req, timeout=connect_timeout)
# 读首字节 —— 如果 P4000/链路卡死,这里会在 connect_timeout 内抛超时
first_chunk = resp.read(1)
first_byte_ok = True
logger.debug(
"[gpu-encoder] P4000 first byte in %.2fs (connect_timeout=%.1fs)",
time.time() - t0,
connect_timeout,
)
# 剩余用长超时(给底层socket放宽时限;如果是mock/不支持,则跳过)
try:
resp.fp._sock.settimeout(ffmpeg_timeout + 60)
except (AttributeError, OSError):
pass
rest = resp.read()
raw = (first_chunk + rest).decode("utf-8")
resp.close()
resp = None
with urllib.request.urlopen(req, timeout=req_timeout) as resp:
raw = resp.read().decode("utf-8")
except urllib.error.HTTPError as e:
detail = e.read().decode("utf-8", errors="replace")[:1000]
raise GpuEncodeError(f"P4000 HTTP {e.code}: {detail}") from e
except (urllib.error.URLError, socket.timeout, TimeoutError, ConnectionError, OSError) as e:
waited = time.time() - t0
hint = "first-byte" if not first_byte_ok else "ffmpeg/upload"
# 统一以 "connection error" 开头,便于上层 fallback 逻辑用关键词识别;
# 末尾再附带具体错误(timed out / refused ...)供排障
raise GpuEncodeError(
f"P4000 {hint} connection error after {waited:.1f}s "
f"(connect_timeout={connect_timeout:.0f}s, ffmpeg_timeout={ffmpeg_timeout}s): {e}"
) from e
finally:
if resp is not None:
try:
resp.close()
except Exception:
pass
except (urllib.error.URLError, socket.timeout, TimeoutError, ConnectionError) as e:
raise GpuEncodeError(f"P4000 connection error: {e}") from e
try:
result = json.loads(raw)
except json.JSONDecodeError as e:
@@ -535,8 +451,6 @@ def _build_client_from_settings() -> Optional[GpuEncoderClient]:
mezzanine_transport=getattr(settings, "gpu_encode_mezzanine_transport", "relay") or "relay",
sync_timeout=getattr(settings, "gpu_encode_sync_timeout", 300),
health_timeout=getattr(settings, "gpu_encode_health_timeout", 3.0),
pre_warm=getattr(settings, "gpu_encode_pre_warm", True),
post_first_byte_timeout=getattr(settings, "gpu_encode_post_first_byte_timeout", 20.0),
vcodec=getattr(settings, "gpu_encode_vcodec", "h264_nvenc"),
preset=getattr(settings, "gpu_encode_preset", "p4"),
crf=getattr(settings, "gpu_encode_crf", 23),
+115 -138
View File
@@ -1,7 +1,6 @@
#!/bin/sh
# ===========================================
# Staging 部署脚本(SSH 模式,并行优化版)
# worker 已收敛到 infra/docker/compose.yml 单一事实来源;api/web 暂保留 docker run。
# ===========================================
set -eu
@@ -49,11 +48,6 @@ GENERATED_DIR="${GENERATED_DIR:-/var/lib/xiaoxia-saas-staging/generated}"
LEGACY_ASSETS_DIR="${LEGACY_ASSETS_DIR:-/var/lib/xiaoxia-saas-staging/legacy-assets}"
NGINX_CONF_FILE="${NGINX_CONF_FILE:-/var/lib/xiaoxia-saas-staging/nginx-staging.conf}"
COOKIES_FILE="${COOKIES_FILE:-/var/lib/xiaoxia-saas-staging/configs/douyin_cookies.txt}"
INFRA_DOCKER_DIR="${INFRA_DOCKER_DIR:-/var/lib/xiaoxia-saas-staging/infra/docker}"
COMPOSE_PROJECT="${COMPOSE_PROJECT:-xiaoxia-staging}"
COMPOSE_ENV_VALUE="${COMPOSE_ENV_VALUE:-staging}"
# COMPOSE_SYNC: CI workflow 已通过 scp 把 infra/docker/compose.yml 上传到服务器时设为 0 跳过同步
COMPOSE_SYNC="${COMPOSE_SYNC:-1}"
SKIP_MIGRATION="${SKIP_MIGRATION:-false}"
SKIP_ROLLBACK="${SKIP_ROLLBACK:-false}"
@@ -63,6 +57,8 @@ if [ -z "$IMAGE_TAG" ]; then
exit 1
fi
# .env 文件由 CI 从模板 + Secrets 渲染后通过 SCP 上传到服务器
# 如果文件不存在,说明 CI 渲染步骤失败或未执行
if [ ! -f "$ENV_FILE" ]; then
echo "ERROR: $ENV_FILE 不存在。CI 应先在 render_env 步骤渲染并上传此文件"
exit 1
@@ -71,7 +67,7 @@ echo "✅ .env file found: $ENV_FILE ($(wc -l < "$ENV_FILE") lines)"
mkdir -p "$GENERATED_DIR"
mkdir -p "$LEGACY_ASSETS_DIR"
mkdir -p "$(dirname "$COOKIES_FILE")"
mkdir -p "$INFRA_DOCKER_DIR"
# 抖音 cookies 文件:CI workflow 已通过 scp 上传;如果不存在(非 CI 环境)则创建占位
if [ ! -f "$COOKIES_FILE" ] || [ "$(wc -c < "$COOKIES_FILE" 2>/dev/null || echo 0)" -lt 200 ]; then
printf '# Netscape HTTP Cookie File\n# 抖音 cookies 占位(CI 应通过 scp 上传真实 cookies)\n' > "$COOKIES_FILE"
echo "WARNING: Douyin cookies not found or too small at $COOKIES_FILE (extraction will 503)"
@@ -80,6 +76,7 @@ else
fi
# ── 写入 Staging Nginx 配置 ──
# 运行时覆盖 nginx 配置,确保 upstream 指向正确的 staging 网络
echo "Writing staging nginx config..."
cat > "$NGINX_CONF_FILE" << 'NGINX_EOF'
server {
@@ -125,26 +122,8 @@ server {
NGINX_EOF
echo "✅ Nginx config written: $NGINX_CONF_FILE"
# ── 确认 infra/docker/compose.yml 存在 ──
# CI workflow 在执行本脚本前已通过 scp 把 infra/docker/compose.yml 上传到 $INFRA_DOCKER_DIR
# (workflow 里做:scp infra/docker/compose.yml <host>:$INFRA_DOCKER_DIR/compose.yml)。
# 这里只做存在性检查 + nginx 软链;不再 curl 私有仓库(SSH 环境无 Gitea token)。
COMPOSE_FILE_PATH="$INFRA_DOCKER_DIR/compose.yml"
if [ ! -f "$COMPOSE_FILE_PATH" ]; then
echo "ERROR: $COMPOSE_FILE_PATH 不存在。CI workflow 应先 scp infra/docker/compose.yml 到服务器"
exit 1
fi
echo "✅ compose.yml ready: $COMPOSE_FILE_PATH ($(wc -l < "$COMPOSE_FILE_PATH") lines)"
ln -sf "$NGINX_CONF_FILE" "$INFRA_DOCKER_DIR/nginx-${COMPOSE_ENV_VALUE}.conf" 2>/dev/null || true
# 封装 docker compose 调用:统一 --env-file(compose 默认只读取 project 目录下的 .env,
# 我们的 .env 在 $INFRA_DOCKER_DIR/../../.env,必须显式传入才能读到 GENERATED_FILES_HOST_DIR 等变量)
compose() {
(cd "$INFRA_DOCKER_DIR" && docker compose --env-file "$ENV_FILE" -p "$COMPOSE_PROJECT" "$@")
}
echo "==========================================="
echo " Staging 部署 - $IMAGE_TAG"
echo " Staging 部署 - $IMAGE_TAG (并行优化版)"
echo "==========================================="
echo "Recording current image versions for rollback..."
@@ -165,7 +144,6 @@ for c in xiaoxia-api-staging xiaoxia-worker-staging xiaoxia-web-staging; do
fi
done
# ── 回滚函数 ──
rollback() {
echo ""
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
@@ -179,37 +157,9 @@ rollback() {
fi
echo "Stopping new containers..."
docker rm -f xiaoxia-api-staging xiaoxia-web-staging 2>/dev/null || true
if [ -n "$PREV_WORKER_IMAGE" ]; then
echo "Rolling back Worker to: $PREV_WORKER_IMAGE (via compose)"
compose up -d --no-deps worker 2>&1 || echo "WARN: compose rollback failed, fallback to docker run"
# 镜像通过 env 注入:compose 默认读 WORKER_IMAGE(未设则用 :dev),这里用临时 env 覆盖
if ! docker inspect xiaoxia-worker-staging >/dev/null 2>&1; then
echo "Fallback: docker run previous worker image"
docker run -d \
--name xiaoxia-worker-staging \
--env-file "$ENV_FILE" \
--network "xiaoxia-net-${COMPOSE_ENV_VALUE}" \
-e APP_ENV="$COMPOSE_ENV_VALUE" \
-e APP_VERSION="$(echo "$PREV_WORKER_IMAGE" | grep -oE '[^:]+$')" \
-e WORKER_MAX_TASKS_PER_CHILD=100 \
-e GENERATION_CONCURRENCY="${GENERATION_CONCURRENCY:-2}" \
-e TRANSCODE_CONCURRENCY="${TRANSCODE_CONCURRENCY:-2}" \
-e BEAT_ENABLED=1 \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-e PUBLIC_API_BASE_URL=https://staging-api.xiaoxiajianji.com \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--health-cmd "grep -q 'celery.*worker' /proc/[0-9]*/cmdline 2>/dev/null || exit 1" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 40s \
--log-driver json-file --log-opt max-size=50m --log-opt max-file=3 \
"$PREV_WORKER_IMAGE" || true
fi
fi
docker rm -f xiaoxia-api-staging 2>/dev/null || true
docker rm -f xiaoxia-worker-staging 2>/dev/null || true
docker rm -f xiaoxia-web-staging 2>/dev/null || true
LOG_OPTS="--log-driver json-file --log-opt max-size=50m --log-opt max-file=3"
@@ -218,10 +168,10 @@ rollback() {
docker run -d \
--name xiaoxia-api-staging \
--env-file "$ENV_FILE" \
--network "xiaoxia-net-${COMPOSE_ENV_VALUE}" \
--network xiaoxia-net-staging \
-p 127.0.0.1:8000:8000 \
-e APP_ENV="$COMPOSE_ENV_VALUE" \
-e APP_VERSION="$(echo "$PREV_API_IMAGE" | grep -oE '[^:]+$')" \
-e APP_ENV=staging \
-e APP_VERSION="$(echo $PREV_API_IMAGE | grep -oE '[^:]+$')" \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-e PUBLIC_API_BASE_URL=https://staging-api.xiaoxiajianji.com \
@@ -233,7 +183,31 @@ rollback() {
--health-retries 3 \
--health-start-period 40s \
$LOG_OPTS \
"$PREV_API_IMAGE" || true
"$PREV_API_IMAGE" &
fi
if [ -n "$PREV_WORKER_IMAGE" ]; then
echo "Rolling back Worker to: $PREV_WORKER_IMAGE"
docker run -d \
--name xiaoxia-worker-staging \
--env-file "$ENV_FILE" \
--network xiaoxia-net-staging \
-e APP_ENV=staging \
-e APP_VERSION="$(echo $PREV_WORKER_IMAGE | grep -oE '[^:]+$')" \
-e WORKER_CONCURRENCY=1 \
-e WORKER_MAX_TASKS_PER_CHILD=100 \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-e PUBLIC_API_BASE_URL=https://staging-api.xiaoxiajianji.com \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--health-cmd "grep -lq celery /proc/[0-9]*/cmdline 2>/dev/null || exit 1" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 30s \
$LOG_OPTS \
"$PREV_WORKER_IMAGE" &
fi
if [ -n "$PREV_WEB_IMAGE" ]; then
@@ -244,7 +218,7 @@ rollback() {
fi
docker run -d \
--name xiaoxia-web-staging \
--network "xiaoxia-net-${COMPOSE_ENV_VALUE}" \
--network xiaoxia-net-staging \
-p 127.0.0.1:3001:80 \
--restart unless-stopped \
$LEGACY_VOLUME \
@@ -254,25 +228,27 @@ rollback() {
--health-timeout 5s \
--health-retries 3 \
$LOG_OPTS \
"$PREV_WEB_IMAGE" || true
"$PREV_WEB_IMAGE" &
fi
sleep 3
wait
echo "Waiting for rolled-back API to become healthy..."
i=0
while [ "$i" -lt 40 ]; do
if curl -sf --max-time 5 http://127.0.0.1:8000/health >/dev/null 2>&1; then
echo "Rolled-back API is healthy!"
break
if [ -n "$PREV_API_IMAGE" ]; then
echo "Waiting for rolled-back API to become healthy..."
i=0
while [ "$i" -lt 40 ]; do
if curl -sf --max-time 5 http://127.0.0.1:8000/health >/dev/null 2>&1; then
echo "Rolled-back API is healthy!"
break
fi
i=$((i + 1))
echo " Waiting... ($i/40)"
sleep 3
done
if [ "$i" -ge 40 ]; then
echo "WARN: Rolled-back API did not become healthy within 120s"
docker logs --tail 30 xiaoxia-api-staging
fi
i=$((i + 1))
echo " Waiting... ($i/40)"
sleep 3
done
if [ "$i" -ge 40 ]; then
echo "WARN: Rolled-back API did not become healthy within 120s"
docker logs --tail 30 xiaoxia-api-staging 2>/dev/null || true
fi
echo ""
@@ -284,7 +260,7 @@ rollback() {
echo "Previous Web: ${PREV_WEB_IMAGE:-none}"
echo ""
echo "部署失败,已自动回滚到上一版本"
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Image}}" | grep staging || true
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Image}}" | grep staging
exit 1
}
@@ -296,6 +272,10 @@ if [ -n "$REGISTRY_TOKEN" ]; then
retry_docker_login
fi
# ---- 并行 Pull 三个镜像 ----
# 注意:这里必须使用 IMAGE_TAG(commit SHA)做确定性部署,不要改成 :dev。
# :dev 是 floating tag,可能被并发构建覆盖,导致部署版本不可重现、回滚混乱。
# Watchtower 可监听 :dev 做非关键路径的自动同步;正式部署/回滚一律锚定 SHA。
REGISTRY_API="${REGISTRY}/xiaoxia-saas-api:${IMAGE_TAG}"
REGISTRY_WORKER="${REGISTRY}/xiaoxia-saas-worker:${IMAGE_TAG}"
REGISTRY_WEB="${REGISTRY}/xiaoxia-saas-web:${IMAGE_TAG}"
@@ -324,6 +304,7 @@ for svc in api worker web; do
elif grep -qE "Digest:|Status: Downloaded" "$PULL_LOG_DIR/$svc.log" 2>/dev/null; then
echo " OK $svc"
else
# 检查docker pull返回值不直接,用镜像是否存在来判断
img_var="REGISTRY_$(echo $svc | tr '[:lower:]' '[:upper:]')"
img_val=$(eval echo "\$$img_var")
if docker image inspect "$img_val" >/dev/null 2>&1; then
@@ -346,7 +327,7 @@ fi
echo "All images pulled."
# ====== 镜像内容校验 ======
# ====== 镜像内容校验(CI 加固 - 防止静默部署损坏/过期镜像) ======
echo ""
echo "=========================================="
echo " 镜像内容校验"
@@ -355,6 +336,7 @@ echo "=========================================="
VERIFY_FAILED=0
DEPLOY_MANIFEST="${GENERATED_DIR}/deploy-manifest.json"
# 读取上次部署的 manifest(用于对比)
PREV_MANIFEST=""
if [ -f "$DEPLOY_MANIFEST" ]; then
PREV_MANIFEST=$(cat "$DEPLOY_MANIFEST")
@@ -366,12 +348,14 @@ for svc in api worker web; do
img_var="REGISTRY_$(echo $svc | tr '[:lower:]' '[:upper:]')"
img_val=$(eval echo "\$$img_var")
# 1. 检查镜像是否存在
if ! docker image inspect "$img_val" >/dev/null 2>&1; then
echo " ❌ $svc: 镜像不存在 ($img_val)"
VERIFY_FAILED=$((VERIFY_FAILED + 1))
continue
fi
# 2. 检查 layers 有效性
LAYER_COUNT=$(docker inspect --format='{{len .RootFS.Layers}}' "$img_val" 2>/dev/null || echo "0")
if [ "$LAYER_COUNT" -eq 0 ]; then
echo " ❌ $svc: 镜像无有效 layers ($img_val)"
@@ -379,12 +363,14 @@ for svc in api worker web; do
continue
fi
# 3. 获取 digest 和创建时间
IMG_ID=$(docker inspect --format='{{.Id}}' "$img_val")
IMG_CREATED=$(docker inspect --format='{{.Created}}' "$img_val")
IMG_SIZE=$(docker inspect --format='{{.Size}}' "$img_val")
echo " ✅ $svc: ${LAYER_COUNT} layers, size=${IMG_SIZE}, created=${IMG_CREATED}"
echo " id: $IMG_ID"
# 4. 对比上次部署
CHANGED="unchanged"
if [ -n "$PREV_MANIFEST" ]; then
PREV_ID=$(echo "$PREV_MANIFEST" | grep "\"${svc}_id\"" | sed 's/.*: *"\(.*\)".*/\1/' 2>/dev/null || echo "")
@@ -412,6 +398,7 @@ if [ "$VERIFY_FAILED" -gt 0 ]; then
exit 1
fi
# 写入新 manifest
cat > "$DEPLOY_MANIFEST" <<MANIFEST_EOF
{
"deployed_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
@@ -457,14 +444,14 @@ for c in xiaoxia-postgres-staging xiaoxia-redis-staging; do
fi
done
docker network create "xiaoxia-net-${COMPOSE_ENV_VALUE}" 2>/dev/null || true
docker network create xiaoxia-net-staging 2>/dev/null || true
if [ "$SKIP_MIGRATION" != "true" ]; then
echo "Running database migrations..."
docker run --rm \
--env-file "$ENV_FILE" \
--network "xiaoxia-net-${COMPOSE_ENV_VALUE}" \
-e APP_ENV="$COMPOSE_ENV_VALUE" \
--network xiaoxia-net-staging \
-e APP_ENV=staging \
"$REGISTRY_API" sh -c "cd /app && alembic upgrade head" || {
echo "ERROR: Database migration failed"
exit 1
@@ -475,28 +462,29 @@ else
fi
echo "Stopping old containers..."
docker stop -t 10 xiaoxia-web-staging 2>/dev/null || true
docker stop -t 30 xiaoxia-api-staging 2>/dev/null || true
# 优雅关闭:先 stop(发 SIGTERM,等待),再 rm
# Worker 需要更长时间(视频任务最长可能5分钟)
docker stop -t 120 xiaoxia-worker-staging 2>/dev/null || true
docker rm -f xiaoxia-api-staging xiaoxia-web-staging 2>/dev/null || true
docker rm -f xiaoxia-worker-staging 2>/dev/null || true
docker stop -t 30 xiaoxia-api-staging 2>/dev/null || true
docker stop -t 10 xiaoxia-web-staging 2>/dev/null || true
docker rm xiaoxia-worker-staging xiaoxia-api-staging xiaoxia-web-staging 2>/dev/null || true
LOG_OPTS="--log-driver json-file --log-opt max-size=50m --log-opt max-file=3"
echo "Starting all containers..."
# ---- 并行启动三个容器 ----
echo "Starting all containers (parallel)..."
LEGACY_VOLUME=""
if [ -d "$LEGACY_ASSETS_DIR" ] && [ "$(ls -A "$LEGACY_ASSETS_DIR" 2>/dev/null)" ]; then
LEGACY_VOLUME="-v ${LEGACY_ASSETS_DIR}:/usr/share/nginx/html/assets-legacy/assets:ro"
fi
# ── API: 暂保留 docker run(TODO: 后续收敛到 compose)──
docker run -d \
--name xiaoxia-api-staging \
--env-file "$ENV_FILE" \
--network "xiaoxia-net-${COMPOSE_ENV_VALUE}" \
--network xiaoxia-net-staging \
-p 127.0.0.1:8000:8000 \
-e APP_ENV="$COMPOSE_ENV_VALUE" \
-e APP_ENV=staging \
-e APP_VERSION="$IMAGE_TAG" \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
@@ -514,18 +502,31 @@ docker run -d \
"$REGISTRY_API" &
PID_API_START=$!
# ── Worker: 通过 compose 启动(单一事实来源)──
# compose.yml 定义:三进程(beat+generation+transcode)、独立并发、BEAT_ENABLED、
# healthcheck 匹配 'celery.*worker'(不把 beat 算活)、资源限制 4C/8G。
# WORKER_IMAGE 通过环境变量覆盖镜像 tag(compose.yml 默认 :dev)。
echo "Starting worker via docker compose (from $INFRA_DOCKER_DIR)..."
WORKER_IMAGE="$REGISTRY_WORKER" APP_VERSION="$IMAGE_TAG" compose up -d --no-deps worker &
docker run -d \
--name xiaoxia-worker-staging \
--env-file "$ENV_FILE" \
--network xiaoxia-net-staging \
-e APP_ENV=staging \
-e APP_VERSION="$IMAGE_TAG" \
-e WORKER_CONCURRENCY=1 \
-e WORKER_MAX_TASKS_PER_CHILD=100 \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-e PUBLIC_API_BASE_URL=https://staging-api.xiaoxiajianji.com \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--health-cmd "grep -lq celery /proc/[0-9]*/cmdline 2>/dev/null || exit 1" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 30s \
$LOG_OPTS \
"$REGISTRY_WORKER" &
PID_WORKER_START=$!
# ── Web: 暂保留 docker run(TODO: 后续收敛到 compose)──
docker run -d \
--name xiaoxia-web-staging \
--network "xiaoxia-net-${COMPOSE_ENV_VALUE}" \
--network xiaoxia-net-staging \
-p 127.0.0.1:3001:80 \
--restart unless-stopped \
$LEGACY_VOLUME \
@@ -562,8 +563,9 @@ if [ "$START_FAILED" -gt 0 ]; then
rollback
fi
# ---- 并行等待 API 和 Web 健康 ----
echo ""
echo "Waiting for all services health (parallel)..."
echo "Waiting for API + Web health (parallel)..."
HEALTH_LOG_DIR="/tmp/staging-health-$$"
mkdir -p "$HEALTH_LOG_DIR"
@@ -598,60 +600,36 @@ PID_API_HEALTH=$!
) > "$HEALTH_LOG_DIR/web.log" 2>&1 &
PID_WEB_HEALTH=$!
(
i=0
while [ "$i" -lt 20 ]; do
hc=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' xiaoxia-worker-staging 2>/dev/null || echo "missing")
if [ "$hc" = "healthy" ]; then
echo "Worker healthy after $((i * 3))s"
exit 0
fi
if [ "$hc" = "unhealthy" ]; then
echo "Worker UNHEALTHY after $((i * 3))s"
docker logs --tail 30 xiaoxia-worker-staging 2>/dev/null || true
exit 1
fi
i=$((i + 1))
sleep 3
done
echo "Worker health unknown after 60s (last: $hc)"
exit 1
) > "$HEALTH_LOG_DIR/worker.log" 2>&1 &
PID_WORKER_HEALTH=$!
set +e
wait $PID_API_HEALTH
API_EXIT=$?
wait $PID_WEB_HEALTH
WEB_EXIT=$?
wait $PID_WORKER_HEALTH
WORKER_EXIT=$?
set -e
echo ""
echo "健康检查结果:"
API_OK=0
WEB_OK=0
if [ "$API_EXIT" -eq 0 ]; then
echo " OK API: $(cat "$HEALTH_LOG_DIR/api.log")"
echo " OK API: $(cat "$HEALTH_LOG_DIR/api.log")"
API_OK=1
else
echo " FAIL API: 120s未就绪"
docker logs --tail 50 xiaoxia-api-staging 2>/dev/null || true
echo " FAIL API: 120s未就绪"
docker logs --tail 50 xiaoxia-api-staging
fi
if [ "$WEB_EXIT" -eq 0 ]; then
echo " OK Web: $(cat "$HEALTH_LOG_DIR/web.log")"
echo " OK Web: $(cat "$HEALTH_LOG_DIR/web.log")"
WEB_OK=1
else
echo " FAIL Web: 30s未就绪"
docker logs --tail 30 xiaoxia-web-staging 2>/dev/null || true
fi
if [ "$WORKER_EXIT" -eq 0 ]; then
echo " OK Worker: $(cat "$HEALTH_LOG_DIR/worker.log")"
else
echo " FAIL Worker: $(cat "$HEALTH_LOG_DIR/worker.log")"
docker logs --tail 50 xiaoxia-worker-staging 2>/dev/null || true
echo " FAIL Web: 30s未就绪"
docker logs --tail 30 xiaoxia-web-staging
fi
rm -rf "$HEALTH_LOG_DIR"
if [ "$API_EXIT" -ne 0 ] || [ "$WEB_EXIT" -ne 0 ] || [ "$WORKER_EXIT" -ne 0 ]; then
if [ "$API_OK" -eq 0 ] || [ "$WEB_OK" -eq 0 ]; then
echo ""
echo "ERROR: 健康检查失败"
rollback
@@ -662,9 +640,8 @@ docker image prune -af --filter "until=168h" 2>/dev/null || true
docker builder prune -af --filter "until=168h" 2>/dev/null || true
echo ""
echo "=== Staging deployment complete ==="
echo "=== Staging deployment complete (并行优化版) ==="
echo "API: http://127.0.0.1:8000"
echo "Web: http://127.0.0.1:3001"
echo "Worker: managed by docker compose (project=$COMPOSE_PROJECT)"
echo "Version: $IMAGE_TAG"
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Image}}" | grep staging