wangdl 49151117c8
Some checks failed
Deploy API Server / build-and-unit (push) Successful in 32s
Deploy API Server / current-integration (push) Failing after 1m34s
Deploy API Server / backward-compat (push) Successful in 16s
Deploy API Server / m-ai-03-synthetic-e2e (push) Failing after 12s
Deploy API Server / deploy (push) Has been skipped
fix: M-AI-03-GATE-FIX — Admin Retry 统一链路 + 队列测试修正 + E2E 激活 + CI 门禁
### 一、生命周期契约核对
- ADR-003 §1.1: 5 状态(queued/running/succeeded/failed/cancelled)
- cancel_requested = cancelRequestedAt 信号(非状态)
- created/retrying 不在状态机中
- 代码完全一致 

### 二、Admin Retry 修复
- retryJob 不再绕过 Registry/CreationService/Outbox
- 改为调用 AiJobCreationService.createJob()(统一事务链路)
- parentJobId + triggerType=admin_manual 正确传入
- 幂等键: admin-retry:<jobId>:<timestamp>(允许不同重试产生不同 Job)
- 原 Job 不修改,新 Job 使用新 ID
- 新增 AiJobCreationService.retrySnapshotContent 支持复用 Snapshot

### 三、队列定义测试修正
- 旧队列断言保持 6 个 legacy queues 的默认值检查
- 新增 M-AI-03 per-queue 断言(ai-interactive: concurrency=2/attempts=2/backoff=2000; ai-background: concurrency=1/attempts=3/backoff=5000/lockDuration=60s)
- lockDuration 测试改为 >= 30s(允许 per-queue 差异)

### 四、Synthetic E2E 激活
- test/m-ai-03-synthetic.e2e-spec.ts: 12 真实场景(创建/幂等/原子/cancel/JWT/隔离/脱敏/未知type/状态机)
- test/jest-m-ai-03.json: 真实 infra config(无 mock Prisma/BullMQ/Redis)
- CI Workflow 新增 m-ai-03-synthetic-e2e job
- deploy needs 新增 m-ai-03-synthetic-e2e 依赖

### 五、测试结果
- 单元测试: 471/471 passed, 0 failed
- E2E: 12 场景待 CI 真实 infra 执行
- queue-definitions: 19/19 passed

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-21 10:47:09 +08:00

417 lines
20 KiB
YAML

name: Deploy API Server
on:
push:
branches: [main]
workflow_dispatch:
inputs:
force_backward_compat:
description: 'Force M-AI-01 backward compatibility test'
type: boolean
default: false
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
WORKDIR: /tmp/api-server
ARTIFACT_DIR: /tmp/api-server-artifacts
jobs:
# ═══════════════════════════════════════════════════════════
# Job 1: Build + Unit Tests
# ═══════════════════════════════════════════════════════════
build-and-unit:
runs-on: prod
steps:
- name: Checkout
run: |
if [ -d $WORKDIR ]; then
cd $WORKDIR
# Save previous HEAD before reset — used by backward-compat for change detection
git rev-parse HEAD > /tmp/prev-head 2>/dev/null || true
git fetch origin && git reset --hard origin/main
else
git clone http://10.2.0.7:3000/wangdl/api-server.git $WORKDIR
git rev-parse HEAD > /tmp/prev-head 2>/dev/null || true
fi
- name: Install dependencies
run: cd $WORKDIR && npm ci
timeout-minutes: 5
- name: Prisma generate & validate
run: |
cd $WORKDIR
npx prisma generate
npx prisma validate
- name: Build
run: cd $WORKDIR && npm run build
- name: Unit tests
run: cd $WORKDIR && npx jest --passWithNoTests
- name: Save build artifacts
run: |
rm -rf $ARTIFACT_DIR
mkdir -p $ARTIFACT_DIR
cp -a $WORKDIR/dist $ARTIFACT_DIR/dist
cp -a $WORKDIR/node_modules $ARTIFACT_DIR/node_modules
cp -a $WORKDIR/prisma $ARTIFACT_DIR/prisma
cp $WORKDIR/package.json $ARTIFACT_DIR/package.json
cp $WORKDIR/package-lock.json $ARTIFACT_DIR/package-lock.json
# ═══════════════════════════════════════════════════════════
# Job 2: Current-version Integration Tests
# ═══════════════════════════════════════════════════════════
current-integration:
needs: build-and-unit
runs-on: prod
steps:
- name: Restore build artifacts
run: |
cp -a $ARTIFACT_DIR/dist $WORKDIR/dist
cp -a $ARTIFACT_DIR/node_modules $WORKDIR/node_modules
cp -a $ARTIFACT_DIR/prisma $WORKDIR/prisma
- name: Ensure infrastructure
run: |
docker start mysql redis qdrant 2>/dev/null || true
sleep 2
- name: Apply database migrations
run: |
cd $WORKDIR
HAS_MIGS=$(npx prisma db execute --stdin <<< "SELECT COUNT(*) as cnt FROM _prisma_migrations" 2>/dev/null | grep -o '[0-9]*' | head -1 || echo "0")
if [ "$HAS_MIGS" = "0" ] || [ -z "$HAS_MIGS" ]; then
echo "[deploy] Baselining existing migrations..."
for m in prisma/migrations/*/; do
m_name=$(basename "$m")
if [ "$m_name" != "backfill_chat_scope.sql" ] && [ "$m_name" != "migration_lock.toml" ]; then
DATABASE_URL="${{ secrets.DATABASE_URL }}" npx prisma migrate resolve --applied "$m_name" 2>/dev/null || true
fi
done
echo "[deploy] Baseline complete."
fi
DATABASE_URL="${{ secrets.DATABASE_URL }}" npx prisma migrate deploy
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
timeout-minutes: 5
- name: Check if Worker paths changed
run: |
cd $WORKDIR
PREV=$(cat /tmp/prev-head 2>/dev/null || echo "")
if [ -n "$PREV" ]; then
CHANGED=$(git diff --name-only $PREV..HEAD 2>/dev/null || echo "")
else
CHANGED=$(git diff --name-only HEAD~1..HEAD 2>/dev/null || echo "")
fi
echo "Changed files: $CHANGED"
if echo "$CHANGED" | grep -qE "src/workers/|src/modules/ai-analysis/|src/modules/ai/|src/infrastructure/queue/|src/infrastructure/outbox/|prisma/schema.prisma|prisma/migrations/|test/worker-integration|test/run-integration"; then
echo "Worker-related changes detected — running integration tests"
echo "run_int=true" > /tmp/int-decision
else
echo "No Worker-related changes — skipping integration tests"
echo "run_int=false" > /tmp/int-decision
fi
- name: Worker Integration tests
if: success()
run: |
RUN=$(cat /tmp/int-decision 2>/dev/null | grep -o 'true\|false' || echo "false")
if [ "$RUN" != "true" ]; then
echo "[integration] Skipped — no Worker-related changes"
exit 0
fi
cd $WORKDIR
bash test/run-integration-tests.sh
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
REDIS_HOST: 127.0.0.1
REDIS_PORT: '6379'
REDIS_PASSWORD: ${{ secrets.REDIS_PASSWORD }}
JWT_SECRET: test-integration-secret
INTERNAL_API_KEY: test-key
CREDENTIAL_ENCRYPTION_KEY: ${{ secrets.CREDENTIAL_ENCRYPTION_KEY }}
STORAGE_DRIVER: local
MYSQL_ROOT_PASSWORD: ${{ secrets.MYSQL_ROOT_PASSWORD }}
timeout-minutes: 10
# ═══════════════════════════════════════════════════════════
# Job 3: M-AI-01 Backward Compatibility
# ═══════════════════════════════════════════════════════════
backward-compat:
needs: build-and-unit
runs-on: prod
steps:
- name: Check if DB-related paths changed
id: paths
run: |
cd $WORKDIR
FORCE="${{ inputs.force_backward_compat || 'false' }}"
if [ "$FORCE" = "true" ]; then
echo "[backward-compat] Forced by workflow_dispatch input"
echo "run_compat=true" > /tmp/compat-decision
exit 0
fi
# Diff HEAD against previous HEAD saved by build-and-unit checkout step.
# This catches exactly what changed in this push.
PREV=$(cat /tmp/prev-head 2>/dev/null || echo "")
if [ -n "$PREV" ] && [ "$PREV" != "$(git rev-parse HEAD)" ]; then
CHANGED=$(git diff --name-only $PREV..HEAD 2>/dev/null || echo "")
else
# Fallback: check last 3 commits
CHANGED=""
for n in 1 2 3; do
CHANGED="$CHANGED $(git diff --name-only HEAD~$n..HEAD 2>/dev/null || echo "")"
done
fi
echo "Previous HEAD: ${PREV:-none}"
echo "Changed files: $CHANGED"
if echo "$CHANGED" | grep -qE "prisma/schema.prisma|prisma/migrations/|src/modules/ai-analysis/|src/workers/ai-analysis.worker.ts|src/infrastructure/queue/|src/infrastructure/outbox/|src/modules/ai-runtime/ai-job|.gitea/workflows/"; then
echo "DB-related changes detected — running backward compatibility"
echo "run_compat=true" > /tmp/compat-decision
else
echo "No DB-related changes — skipping backward compatibility"
echo "run_compat=false" > /tmp/compat-decision
fi
- name: Ensure infrastructure
if: success()
run: |
RUN=$(cat /tmp/compat-decision 2>/dev/null | grep -o 'true\|false' || echo "false")
if [ "$RUN" = "true" ]; then
docker start mysql redis 2>/dev/null || true
sleep 2
else
echo "[backward-compat] Skipped — no DB changes"
fi
- name: Checkout M-AI-01 baseline
if: success()
run: |
RUN=$(cat /tmp/compat-decision 2>/dev/null | grep -o 'true\|false' || echo "false")
if [ "$RUN" != "true" ]; then exit 0; fi
OLD_DIR=/tmp/api-server-old
mkdir -p $OLD_DIR
cd $OLD_DIR
if [ -d .git ]; then
git fetch origin && git checkout 789d6ec --force
else
git clone http://10.2.0.7:3000/wangdl/api-server.git .
git checkout 789d6ec
fi
- name: Install M-AI-01 dependencies
if: success()
run: |
RUN=$(cat /tmp/compat-decision 2>/dev/null | grep -o 'true\|false' || echo "false")
if [ "$RUN" != "true" ]; then exit 0; fi
cd /tmp/api-server-old && npm ci
timeout-minutes: 5
- name: Generate M-AI-01 Prisma client
if: success()
run: |
RUN=$(cat /tmp/compat-decision 2>/dev/null | grep -o 'true\|false' || echo "false")
if [ "$RUN" != "true" ]; then exit 0; fi
cd /tmp/api-server-old && npx prisma generate
- name: Run backward compatibility tests
if: success()
run: |
RUN=$(cat /tmp/compat-decision 2>/dev/null | grep -o 'true\|false' || echo "false")
if [ "$RUN" != "true" ]; then
echo "[backward-compat] Skipped — no DB-related changes"
exit 0
fi
# Write test inside old code dir so tsx can find node_modules/@prisma/client
cat > /tmp/api-server-old/compat-test.ts << 'TESTEOF'
import { PrismaClient } from '@prisma/client';
const DB_URL = process.env.DATABASE_URL || '';
const prisma = new PrismaClient({ datasources: { db: { url: DB_URL } } });
async function main() {
const TEST_USER = 'backward-compat-test-' + Date.now();
let errors = 0;
await prisma.$executeRawUnsafe("INSERT IGNORE INTO User (id, email, createdAt, updatedAt) VALUES (?, ?, NOW(), NOW())", TEST_USER, TEST_USER + '@test.com');
console.log('1. User created: ' + TEST_USER);
let job: any;
try {
job = await (prisma as any).aiAnalysisJob.create({ data: { userId: TEST_USER, jobType: 'feynman-evaluation', status: 'pending', sessionId: null, answerId: null, queuedAt: new Date() } });
console.log('2. createJob: id=' + job.id + ' status=' + job.status + ' ✅');
} catch (e: any) { console.log('2. createJob FAILED: ' + e.message + ' ❌'); errors++; await prisma.$disconnect(); process.exit(1); }
try {
const found = await (prisma as any).aiAnalysisJob.findUnique({ where: { id: job.id }, include: { results: true } });
if (!found) throw new Error('not found');
console.log('3. findById: id=' + found.id + ' status=' + found.status + ' ✅');
} catch (e: any) { console.log('3. findById FAILED: ' + e.message + ' ❌'); errors++; }
try {
const updated = await (prisma as any).aiAnalysisJob.update({ where: { id: job.id }, data: { status: 'processing', startedAt: new Date() } });
console.log('4. pending→processing: status=' + updated.status + ' ✅');
} catch (e: any) { console.log('4. pending→processing FAILED: ' + e.message + ' ❌'); errors++; }
try {
const updated = await (prisma as any).aiAnalysisJob.update({ where: { id: job.id }, data: { status: 'completed', completedAt: new Date() } });
console.log('5. processing→completed: status=' + updated.status + ' ✅');
} catch (e: any) { console.log('5. processing→completed FAILED: ' + e.message + ' ❌'); errors++; }
let job2: any;
try {
job2 = await (prisma as any).aiAnalysisJob.create({ data: { userId: TEST_USER, jobType: 'active-recall', status: 'pending', queuedAt: new Date() } });
await (prisma as any).aiAnalysisJob.update({ where: { id: job2.id }, data: { status: 'processing', startedAt: new Date() } });
const failed = await (prisma as any).aiAnalysisJob.update({ where: { id: job2.id }, data: { status: 'failed', errorMessage: 'test error', completedAt: new Date() } });
console.log('6. processing→failed: status=' + failed.status + ' errorMessage=' + failed.errorMessage + ' ✅');
} catch (e: any) { console.log('6. processing→failed FAILED: ' + e.message + ' ❌'); errors++; }
const cols = await prisma.$queryRawUnsafe("SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'AiAnalysisJob' AND COLUMN_NAME IN ('errorMessage','completedAt','status','jobType') ORDER BY ORDINAL_POSITION") as any[];
const colNames = cols.map((c: any) => c.COLUMN_NAME);
const required = ['jobType','status','errorMessage','completedAt'];
const missing = required.filter(r => !colNames.includes(r));
if (missing.length === 0) { console.log('7. Physical columns preserved: ' + colNames.join(', ') + ' ✅'); }
else { console.log('7. MISSING: ' + missing.join(', ') + ' ❌'); errors++; }
try {
await (prisma as any).aiAnalysisResult.create({ data: { userId: TEST_USER, jobId: job.id, summary: 'test', masteryScore: 80 } });
const results = await (prisma as any).aiAnalysisResult.findMany({ where: { jobId: job.id } });
console.log('8. AiAnalysisResult FK: ' + results.length + ' results ✅');
} catch (e: any) { console.log('8. AiAnalysisResult FK FAILED: ' + e.message + ' ❌'); errors++; }
try {
await (prisma as any).aiUsageLog.create({ data: { userId: TEST_USER, feature: 'compat-test', provider: 'deepseek', model: 'deepseek-chat', tier: 'primary', promptKey: 'test', promptVersion: 'v1' } });
console.log('9. AiUsageLog insert ✅');
} catch (e: any) { console.log('9. AiUsageLog insert: ' + e.message + ' ⚠️ (non-blocking, table may need migration)'); }
await prisma.$executeRawUnsafe("DELETE FROM AiAnalysisResult WHERE userId = ?", TEST_USER);
try { await (prisma as any).aiUsageLog.deleteMany({ where: { userId: TEST_USER } }); } catch {}
await (prisma as any).aiAnalysisJob.deleteMany({ where: { userId: TEST_USER } });
await prisma.$executeRawUnsafe("DELETE FROM User WHERE id = ?", TEST_USER);
await prisma.$disconnect();
if (errors > 0) { console.log('\n' + errors + ' core test(s) FAILED ❌'); process.exit(1); }
console.log('\nAll backward compatibility tests PASSED ✅');
}
main().catch(e => { console.error(e); process.exit(1); });
TESTEOF
cd /tmp/api-server-old
DATABASE_URL="${{ secrets.DATABASE_URL }}" npx tsx compat-test.ts
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
timeout-minutes: 5
# ═══════════════════════════════════════════════════════════
# Job 4: M-AI-03 Synthetic E2E
# ═══════════════════════════════════════════════════════════
m-ai-03-synthetic-e2e:
needs: build-and-unit
runs-on: prod
steps:
- name: Restore build artifacts
run: |
cp -a $ARTIFACT_DIR/dist $WORKDIR/dist
cp -a $ARTIFACT_DIR/node_modules $WORKDIR/node_modules
cp -a $ARTIFACT_DIR/prisma $WORKDIR/prisma
- name: Ensure infrastructure
run: |
docker start mysql redis 2>/dev/null || true
sleep 2
- name: Apply database migrations
run: |
cd $WORKDIR
DATABASE_URL="${{ secrets.DATABASE_URL }}" npx prisma migrate deploy
timeout-minutes: 5
- name: M-AI-03 Synthetic E2E
run: |
cd $WORKDIR
NODE_ENV=test AI_JOB_SYNTHETIC_ENABLED=true \
DATABASE_URL="${{ secrets.DATABASE_URL }}" \
REDIS_URL="${{ secrets.REDIS_URL || 'redis://localhost:6379' }}" \
npx jest --config test/jest-e2e.json --testPathPatterns="m-ai-03-synthetic" --forceExit --verbose
timeout-minutes: 10
# ═══════════════════════════════════════════════════════════
# Job 5: Deploy (only if all gates pass)
# ═══════════════════════════════════════════════════════════
deploy:
needs: [current-integration, backward-compat, m-ai-03-synthetic-e2e]
runs-on: prod
steps:
- name: Restore build artifacts
run: |
cp -a $ARTIFACT_DIR/dist $WORKDIR/dist
cp -a $ARTIFACT_DIR/node_modules $WORKDIR/node_modules
cp -a $ARTIFACT_DIR/prisma $WORKDIR/prisma
- name: Ensure infrastructure
run: |
docker start mysql redis qdrant 2>/dev/null || true
sleep 2
- name: Resolve failed migrations
run: |
cd $WORKDIR
MYSQL_CMD="docker exec mysql mysql -u ${{ secrets.DB_USER }} -p${{ secrets.DB_PASSWORD }} ${{ secrets.DB_NAME }}"
FAILED=$($MYSQL_CMD -N -e "SELECT migration_name FROM _prisma_migrations WHERE logs LIKE '%failed%' LIMIT 1;" 2>/dev/null || true)
if [ -n "$FAILED" ]; then
echo "[deploy] Found failed migration: $FAILED, cleaning up..."
$MYSQL_CMD -e "DELETE FROM _prisma_migrations WHERE migration_name = '$FAILED';"
echo "[deploy] Cleaned up failed migration $FAILED"
else
echo "[deploy] No failed migrations found"
fi
- name: Apply database migrations
run: |
cd $WORKDIR
HAS_MIGS=$(npx prisma db execute --stdin <<< "SELECT COUNT(*) as cnt FROM _prisma_migrations" 2>/dev/null | grep -o '[0-9]*' | head -1 || echo "0")
if [ "$HAS_MIGS" = "0" ] || [ -z "$HAS_MIGS" ]; then
echo "[deploy] Baselining existing migrations..."
for m in prisma/migrations/*/; do
m_name=$(basename "$m")
if [ "$m_name" != "backfill_chat_scope.sql" ] && [ "$m_name" != "migration_lock.toml" ]; then
DATABASE_URL="${{ secrets.DATABASE_URL }}" npx prisma migrate resolve --applied "$m_name" 2>/dev/null || true
fi
done
echo "[deploy] Baseline complete."
fi
DATABASE_URL="${{ secrets.DATABASE_URL }}" npx prisma migrate deploy
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
- name: Deploy API
run: |
sudo rsync -av --delete $WORKDIR/dist/ /opt/zhixi/backend/dist/
sudo rsync -av --delete $WORKDIR/node_modules/ /opt/zhixi/backend/node_modules/
sudo rsync -av $WORKDIR/prisma/ /opt/zhixi/backend/prisma/
sudo rsync -av $WORKDIR/package.json /opt/zhixi/backend/package.json
cd /opt/zhixi/backend && npx prisma generate
sudo systemctl reset-failed zhixi-api 2>/dev/null || true
sudo systemctl restart zhixi-api
sleep 3
if curl -sf http://localhost:3000/api > /dev/null 2>&1; then
echo "[deploy] API health OK"
else
echo "[deploy] Health check failed"
sudo journalctl -u zhixi-api --no-pager -n 20
exit 1
fi
- name: Deploy Worker
run: |
cd /opt/zhixi/backend
sudo cp deploy/zhixi-nest-worker.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl reset-failed zhixi-nest-worker 2>/dev/null || true
sudo systemctl restart zhixi-nest-worker
sleep 5
echo "[deploy] Worker: $(sudo systemctl is-active zhixi-nest-worker)"
sudo journalctl -u zhixi-nest-worker --no-pager -n 10
- name: Deploy RAG Worker
run: |
set -e
WORKER_DIR="/opt/zhixi/backend/rag-worker"
mkdir -p "$WORKER_DIR"
rsync -av --delete --exclude='.env' --exclude='__pycache__' $WORKDIR/rag-worker/ "$WORKER_DIR/"
sudo cp "$WORKER_DIR/zhixi-worker.service" /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl restart zhixi-worker
sleep 5
sudo systemctl is-active zhixi-worker
echo "[deploy] RAG worker active OK"