Compare commits
No commits in common. "4fb652d2735ccbb06d35ba309e0899d93b837544" and "d7647cbc3d289a6fe1e741c3ae6c94cc19130711" have entirely different histories.
4fb652d273
...
d7647cbc3d
@ -31,4 +31,4 @@ COPY --from=builder /app/package.json ./
|
||||
|
||||
EXPOSE 3000
|
||||
|
||||
CMD ["sh", "-c", "npx prisma migrate deploy && node dist/src/main.js"]
|
||||
CMD ["sh", "-c", "npx prisma migrate deploy && node dist/main.js"]
|
||||
|
||||
@ -26,4 +26,4 @@ COPY --from=builder /app/dist ./dist
|
||||
COPY --from=builder /app/prisma ./prisma
|
||||
COPY --from=builder /app/package.json ./
|
||||
|
||||
CMD ["node", "dist/src/worker.main.js"]
|
||||
CMD ["node", "dist/worker.main.js"]
|
||||
|
||||
@ -1,138 +0,0 @@
|
||||
# 部署流程与回滚方案
|
||||
|
||||
## 部署拓扑
|
||||
|
||||
```
|
||||
┌──────────┐
|
||||
│ NGINX │ :80/:443
|
||||
└────┬─────┘
|
||||
│
|
||||
┌──────────┴──────────┐
|
||||
│ │
|
||||
┌─────▼─────┐ ┌─────▼─────┐
|
||||
│ zhixi-api │ │zhixi-worker│
|
||||
│ :3000 │ │ (no port) │
|
||||
│ main.js │ │worker.main │
|
||||
└─────┬─────┘ └─────┬─────┘
|
||||
│ │
|
||||
│ ┌──────────┐ │
|
||||
└────┤ Redis ├─────┘
|
||||
│ (BullMQ) │
|
||||
└────┬─────┘
|
||||
│
|
||||
┌────▼────┐
|
||||
│ MySQL │
|
||||
└─────────┘
|
||||
|
||||
zhixi-heavy-runtime (独立服务, 本批不修改)
|
||||
```
|
||||
|
||||
## API 与 Worker 职责边界
|
||||
|
||||
| 职责 | API | Worker |
|
||||
|------|:---:|:------:|
|
||||
| HTTP Controller | ✅ | — |
|
||||
| Queue Producer (add) | ✅ | — |
|
||||
| BullMQ @Processor | — | ✅ |
|
||||
| Prisma migration | ✅ (启动时) | — |
|
||||
| 监听端口 | :3000 | 无 |
|
||||
| systemd unit | `zhixi-nest-api` | `zhixi-nest-worker` |
|
||||
| Docker CMD | `node dist/src/main.js` | `node dist/worker.main.js` |
|
||||
|
||||
## Docker 部署
|
||||
|
||||
```bash
|
||||
# 构建
|
||||
docker compose build api worker
|
||||
|
||||
# 启动(含 MySQL/Redis 依赖)
|
||||
docker compose up -d
|
||||
|
||||
# 验证
|
||||
docker compose exec api wget -qO- http://localhost:3000/health # API health
|
||||
docker compose logs worker | grep "Worker version" # Worker startup
|
||||
# Heartbeat 验证见 M-AI-01-07
|
||||
```
|
||||
|
||||
Docker 环境中 API 进程 Processor 数量为 0(`AppModule` 不含任何 `@Processor()` 注册,仅 `WorkerModule` 注册全部 5 个 Processor)。
|
||||
|
||||
## 生产 systemd 部署
|
||||
|
||||
```bash
|
||||
# 1. 复制 unit 文件
|
||||
sudo cp deploy/zhixi-nest-api.service /etc/systemd/system/
|
||||
sudo cp deploy/zhixi-nest-worker.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# 2. 启用自启动
|
||||
sudo systemctl enable zhixi-nest-api zhixi-nest-worker
|
||||
|
||||
# 3. 启动
|
||||
sudo systemctl start zhixi-nest-api
|
||||
sudo systemctl start zhixi-nest-worker
|
||||
|
||||
# 4. 验证
|
||||
sudo systemctl status zhixi-nest-api zhixi-nest-worker
|
||||
curl -s http://localhost:3000/health
|
||||
sudo journalctl -u zhixi-nest-worker -n 20 | grep "Worker version"
|
||||
```
|
||||
|
||||
## 标准部署流程
|
||||
|
||||
无 DB 变更时(回滚同样适用):
|
||||
|
||||
```
|
||||
1. npm run build # 构建
|
||||
2. sudo systemctl restart zhixi-nest-worker # 先重启 Worker
|
||||
3. sleep 5 && journalctl -u zhixi-nest-worker # 验证 Worker heartbeat
|
||||
4. sudo systemctl restart zhixi-nest-api # 再重启 API
|
||||
5. curl http://localhost:3000/health # 验证 API health
|
||||
6. smoke test # 业务验证
|
||||
```
|
||||
|
||||
**原则**:Worker 先于 API 重启。API 重启期间 Worker 继续消费队列。Worker 重启期间 API 继续响应请求(jobs 变为 waiting)。
|
||||
|
||||
## 独立重启验证
|
||||
|
||||
- **API 重启不停止 Worker**:`systemctl restart zhixi-nest-api` 后 `systemctl status zhixi-nest-worker` 仍为 active
|
||||
- **Worker 重启不停止 API**:`systemctl restart zhixi-nest-worker` 后 `curl localhost:3000/health` 返回 200
|
||||
|
||||
## 回滚方案
|
||||
|
||||
### 代码回滚
|
||||
|
||||
```bash
|
||||
git revert <commit> # 回滚代码
|
||||
npm run build # 重新构建
|
||||
sudo systemctl restart zhixi-nest-worker
|
||||
sudo systemctl restart zhixi-nest-api
|
||||
```
|
||||
|
||||
### Processor 恢复(回滚 M-AI-01-03)
|
||||
|
||||
如果拆分导致问题,恢复全部 5 个 Processor 到拆分前位置即可回退为 API 进程内消费模式:
|
||||
|
||||
1. 在 `app.module.ts` providers 中恢复 `AiAnalysisWorker`、`DocumentImportWorker`、`NotificationWorker`
|
||||
2. 在 `admin-audit-log.module.ts` providers 中恢复 `AuditLogProcessor` 及 `BullModule.registerQueue({ name: QUEUE_AUDIT_LOG })`
|
||||
3. 在 `files.module.ts` providers 中恢复 `FileCleanupProcessor` 及 `BullModule.registerQueue({ name: QUEUE_FILE_CLEANUP })`
|
||||
4. 从 `worker.module.ts` providers 移除 `AuditLogProcessor`、`FileCleanupProcessor`
|
||||
5. 停止 `zhixi-nest-worker` service:`sudo systemctl stop zhixi-nest-worker`
|
||||
6. 重启 API:`sudo systemctl restart zhixi-nest-api`
|
||||
|
||||
此时全部 5 个 Processor 回到拆分前位置,API 进程同时担任 Producer 和 Consumer。
|
||||
|
||||
> 推荐使用 `git revert` 作为主回滚方案,可一步覆盖上述所有文件。
|
||||
|
||||
### 数据回滚
|
||||
|
||||
本批不涉及 Prisma Schema 变更,无需数据回滚。
|
||||
|
||||
### 队列安全
|
||||
|
||||
回滚前后验证 BullMQ 无双重消费:
|
||||
|
||||
```bash
|
||||
# 检查 Bull Board 或 Redis 中的 active/waiting 计数
|
||||
redis-cli LLEN bull:ai-analysis:waiting
|
||||
redis-cli LLEN bull:ai-analysis:active
|
||||
```
|
||||
@ -1,34 +0,0 @@
|
||||
[Unit]
|
||||
Description=zhixi NestJS API Server
|
||||
After=network.target mysql.service redis.service
|
||||
Wants=mysql.service redis.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=www-data
|
||||
Group=www-data
|
||||
WorkingDirectory=/opt/zhixi/api-server
|
||||
ExecStart=/usr/bin/node dist/src/main.js
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
TimeoutStopSec=30
|
||||
|
||||
# Environment
|
||||
Environment=NODE_ENV=production
|
||||
Environment=PORT=3000
|
||||
EnvironmentFile=/opt/zhixi/api-server/.env
|
||||
|
||||
# Security
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ReadWritePaths=/opt/zhixi/api-server/uploads
|
||||
ReadWritePaths=/opt/zhixi/api-server/prisma
|
||||
|
||||
# Logging
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=zhixi-nest-api
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@ -1,39 +0,0 @@
|
||||
[Unit]
|
||||
Description=zhixi NestJS Worker (BullMQ Processor)
|
||||
After=network.target mysql.service redis.service
|
||||
Wants=mysql.service redis.service
|
||||
# Start after API is up (API runs Prisma migration on start)
|
||||
After=zhixi-nest-api.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=www-data
|
||||
Group=www-data
|
||||
WorkingDirectory=/opt/zhixi/api-server
|
||||
ExecStart=/usr/bin/node dist/src/worker.main.js
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
TimeoutStopSec=60
|
||||
# Worker needs extra time: BullMQ graceful shutdown waits for active jobs,
|
||||
# then SIGKILL from systemd after TimeoutStopSec provides hard deadline.
|
||||
|
||||
# Environment
|
||||
Environment=NODE_ENV=production
|
||||
EnvironmentFile=/opt/zhixi/api-server/.env
|
||||
|
||||
# Memory limit (Worker jobs may consume memory)
|
||||
MemoryMax=512M
|
||||
|
||||
# Security
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ReadWritePaths=/opt/zhixi/api-server/uploads
|
||||
|
||||
# Logging
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=zhixi-nest-worker
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@ -1,174 +0,0 @@
|
||||
# ADR-001 附录 A:Worker 执行依赖闭包审计
|
||||
|
||||
## 审计日期
|
||||
|
||||
2026-06-19
|
||||
|
||||
## 目标
|
||||
|
||||
确认每个 BullMQ Processor 的完整依赖、事件和副作用,防止 Processor 移到独立 Worker 进程后链路断裂。
|
||||
|
||||
---
|
||||
|
||||
## 一、Processor 依赖矩阵
|
||||
|
||||
### 1.1 AiAnalysisWorker
|
||||
|
||||
| 维度 | 事实 | 证据 |
|
||||
|------|------|------|
|
||||
| **队列** | `ai-analysis` | `@Processor(QUEUE_AI_ANALYSIS)` |
|
||||
| **注册位置** | `AppModule` (line 193) + `WorkerModule` (line 64) — **双重注册** | `app.module.ts:193`、`worker.module.ts:64` |
|
||||
| **注入依赖** | `ActiveRecallAnalysisWorkflow`、`FeynmanEvaluationWorkflow`、`AiAnalysisRepository`、`EventBusService` (@Optional)、`FocusItemsService` (@Optional) | `ai-analysis.worker.ts:22-28` |
|
||||
| **依赖 Module** | `AiModule`(提供 Workflow)、`AiAnalysisModule`(提供 Repository)、`EventBusModule`(提供 EventBusService)、`FocusItemsModule`(提供 FocusItemsService) | — |
|
||||
| **写入的表** | `AiAnalysisJob`(updateJobStatus)、`AiAnalysisResult`(createResult)、`FocusItem`(create) | `ai-analysis.worker.ts:47,66,67,87,101` |
|
||||
| **发布的事件** | `ai.analysis.completed`(通过 `EventBusService.publish()` → `EventEmitter2.emit()` — **进程内**) | `ai-analysis.worker.ts:71` |
|
||||
| **事件订阅者** | `ReviewCardSubscriber`(`@OnEvent('ai.analysis.completed')` → `ReviewModule` → `AppModule`) | `review-card.subscriber.ts:11` |
|
||||
| **WorkerModule 中缺失** | `FocusItemsModule` 未导入 → `FocusItemsService` 在 Worker 进程中为 `null` | `worker.module.ts:1-69` |
|
||||
| **移动后断裂链路** | ⚠️ `ai.analysis.completed` 事件无法到达 `ReviewCardSubscriber`(进程内 EventEmitter2);⚠️ `FocusItemsService` 在独立 Worker 进程中为 null | — |
|
||||
|
||||
### 1.2 DocumentImportWorker
|
||||
|
||||
| 维度 | 事实 | 证据 |
|
||||
|------|------|------|
|
||||
| **队列** | `document-import` | `@Processor(QUEUE_DOCUMENT_IMPORT)` |
|
||||
| **注册位置** | `AppModule` (line 194) + `WorkerModule` (line 65) — **双重注册** | `app.module.ts:194`、`worker.module.ts:65` |
|
||||
| **注入依赖** | `KnowledgeImportWorkflow`、`KnowledgeItemsRepository`、`RedisService` | 需确认 `document-import.worker.ts` |
|
||||
| **写入的表** | `KnowledgeItem` | — |
|
||||
| **发布的事件** | 无 | — |
|
||||
| **移动后断裂链路** | 无事件依赖,但需确认 Redis 连接在 Worker 进程中可用 | — |
|
||||
|
||||
### 1.3 NotificationWorker
|
||||
|
||||
| 维度 | 事实 | 证据 |
|
||||
|------|------|------|
|
||||
| **队列** | `notification` | `@Processor(QUEUE_NOTIFICATION)` |
|
||||
| **注册位置** | `AppModule` (line 195) + `WorkerModule` (line 66) — **双重注册** | `app.module.ts:195`、`worker.module.ts:66` |
|
||||
| **注入依赖** | `NotificationsService` | `notification.worker.ts:8,12` |
|
||||
| **依赖链** | `NotificationsService` → `NotificationsRepository`(必需)+ `EventBusService`(@Optional) | `notifications.service.ts:13-15` |
|
||||
| **依赖 Module** | `NotificationsModule`(提供 NotificationsService + NotificationsRepository)、`EventBusModule`(提供 EventBusService) | `notifications.module.ts:9-14` |
|
||||
| **写入的表** | `Notification`(更新发送状态 — 通过 NotificationsRepository) | — |
|
||||
| **发布的事件** | 无(`EventBusService` 为 @Optional 注入,NotificationWorker 自身不发布事件) | — |
|
||||
| **移动后断裂链路** | 无 — `NotificationsModule` 已在 `WorkerModule.imports` 中(line 61),`NotificationsModule` 自身 imports `EventBusModule`,`PrismaService` 由 `PrismaModule` 提供。依赖链完整,Worker 进程启动不会因缺失依赖崩溃 | `worker.module.ts:61` |
|
||||
|
||||
### 1.4 AuditLogProcessor
|
||||
|
||||
| 维度 | 事实 | 证据 |
|
||||
|------|------|------|
|
||||
| **队列** | `audit-logs` | `@Processor(QUEUE_AUDIT_LOG)` |
|
||||
| **注册位置** | `AdminAuditLogModule` providers(仅注册一次,未双重) | `admin-audit-log.module.ts:13` |
|
||||
| **注入依赖** | `PrismaService` | `audit-log.processor.ts:7` |
|
||||
| **写入的表** | `AdminAuditLog` | `audit-log.processor.ts:12-23` |
|
||||
| **发布的事件** | 无 | — |
|
||||
| **移动后断裂链路** | 无业务事件依赖,但 `AdminAuditLogModule` 同时包含 Controller 和 Processor,需要拆分 | — |
|
||||
|
||||
### 1.5 FileCleanupProcessor
|
||||
|
||||
| 维度 | 事实 | 证据 |
|
||||
|------|------|------|
|
||||
| **队列** | `file-cleanup` | `@Processor(QUEUE_FILE_CLEANUP)` |
|
||||
| **注册位置** | ⚠️ `FilesModule` 导入了 `FileCleanupProcessor` 但**未在 providers 中注册** — `BullExplorer` 扫描可能仍能发现,但依赖注入不可靠 | `files.module.ts:3,12-14` |
|
||||
| **注入依赖** | `CosStorageProvider` | `file-cleanup.processor.ts:10` |
|
||||
| **写入的表** | 无(直接操作 COS) | — |
|
||||
| **发布的事件** | 无 | — |
|
||||
| **移动后断裂链路** | 需确认 `CosStorageProvider` 在 Worker 进程中可注入 | — |
|
||||
|
||||
---
|
||||
|
||||
## 二、事件链路审计
|
||||
|
||||
### 2.1 EventBusService 实现
|
||||
|
||||
| 方法 | 机制 | 跨进程 | 证据 |
|
||||
|------|------|:---:|------|
|
||||
| `publish(event)` | `EventEmitter2.emit()` — 进程内同步 | ❌ | `event-bus.service.ts:19-22` |
|
||||
| `publishAsync(event)` | BullMQ `domain-events` 队列 — 跨进程异步 | ✅ | `event-bus.service.ts:26-36` |
|
||||
|
||||
### 2.2 当前事件使用
|
||||
|
||||
| 事件 | 发布者 | 发布方式 | 订阅者 | 订阅方式 | 位置 | 跨进程安全 |
|
||||
|------|--------|---------|--------|---------|------|:---:|
|
||||
| `ai.analysis.completed` | `AiAnalysisWorker` | `publish()`(进程内) | `ReviewCardSubscriber` | `@OnEvent`(EventEmitter2) | `review-card.subscriber.ts:11` | ❌ |
|
||||
| `task.enqueued` | `QueueService` | `publish()`(进程内) | 无消费者 | — | `queue.service.ts:34` | — |
|
||||
| `streak.updated` | `GrowthService` | `publish()`(进程内) | 无消费者 | — | `growth.service.ts:53` | — |
|
||||
|
||||
### 2.3 Domain Events 队列
|
||||
|
||||
- **生产者**:`EventBusService.publishAsync()` → `queue.add('domain-events', ...)`
|
||||
- **消费者**:无 — 代码库中无任何 `@Processor('domain-events')`
|
||||
- **状态**:死队列 — 事件被写入 Redis 但永不被处理
|
||||
- **结论**:`publishAsync()` 当前无实际效果
|
||||
|
||||
### 2.4 移动后的断裂链路
|
||||
|
||||
```
|
||||
移动前(同进程):
|
||||
AiAnalysisWorker → EventBusService.publish() → EventEmitter2.emit('ai.analysis.completed')
|
||||
→ ReviewCardSubscriber.@OnEvent('ai.analysis.completed')
|
||||
→ reviewService.generateCards() → ReviewCard 创建 ✅
|
||||
|
||||
移动后(跨进程):
|
||||
Worker 进程: AiAnalysisWorker → EventBusService.publish() → EventEmitter2.emit()
|
||||
API 进程: ReviewCardSubscriber.@OnEvent() — 收不到 ❌
|
||||
```
|
||||
|
||||
**影响**:AI 分析完成后不再自动生成复习卡片。
|
||||
|
||||
---
|
||||
|
||||
## 三、Module 注册审计
|
||||
|
||||
### 3.1 双重注册的 Processor
|
||||
|
||||
| Processor | AppModule | WorkerModule | 业务 Module | 问题 |
|
||||
|-----------|:---:|:---:|:---:|------|
|
||||
| AiAnalysisWorker | ✅ (line 193) | ✅ (line 62) | — | 若两个 Module 均加载,BullMQ 创建两个 Worker 实例消费同一队列 → **并发双消费** |
|
||||
| DocumentImportWorker | ✅ (line 194) | ✅ (line 65) | — | 同上 |
|
||||
| NotificationWorker | ✅ (line 195) | ✅ (line 66) | — | 同上 |
|
||||
|
||||
### 3.2 需拆分的 Module(同时包含 Controller 和 Processor)
|
||||
|
||||
| Module | Controller | Processor | Producer |
|
||||
|--------|:---:|:---:|:---:|
|
||||
| `AdminAuditLogModule` | ✅ | ✅ (`AuditLogProcessor`) | — |
|
||||
| `FilesModule` | ✅ | ⚠️ (`FileCleanupProcessor` 导入但未在 providers 注册) | ✅ (`FilesService`) |
|
||||
|
||||
---
|
||||
|
||||
## 四、WorkerModule 缺失依赖
|
||||
|
||||
当前 `WorkerModule`(`worker.module.ts:1-69`)导入的 Module:
|
||||
|
||||
```
|
||||
PrismaModule, RedisModule, QueueModule, AiModule, StorageModule,
|
||||
LoggerModule, AiAnalysisModule, DocumentImportModule, KnowledgeItemsModule, NotificationsModule
|
||||
```
|
||||
|
||||
**缺失但 AiAnalysisWorker 需要的 Module**:
|
||||
|
||||
| 缺失 Module | 提供的 Service | 影响 |
|
||||
|------------|---------------|------|
|
||||
| `FocusItemsModule` | `FocusItemsService` | AiAnalysisWorker 中 `focusItems` 为 null,不创建 FocusItem |
|
||||
| `EventBusModule` | `EventBusService` | AiAnalysisWorker 中 `eventBus` 为 null,不发布事件。但当前有 `@Optional()` 标记,不会崩溃 |
|
||||
| `ReviewModule` | `ReviewCardSubscriber` | 即使事件发布成功(使用 publishAsync),无订阅者处理 |
|
||||
|
||||
---
|
||||
|
||||
## 五、修复建议矩阵(供 01-03 使用)
|
||||
|
||||
| 问题 | 修复方向 | 优先级 |
|
||||
|------|---------|:---:|
|
||||
| AiAnalysisWorker/NotificationWorker/DocumentImportWorker 双重注册 | 从 `AppModule` providers 中移除,仅保留在 `WorkerModule` | **阻塞** |
|
||||
| `EventBusService.publish()` 无法跨进程 | 选项 A:将 `ReviewCardSubscriber` 移入 `WorkerModule`;选项 B:改为 `publishAsync()` + 新增 `DomainEventsProcessor` 消费 `domain-events` 队列 | **阻塞** |
|
||||
| `FocusItemsModule` 未在 WorkerModule | 添加 `FocusItemsModule` 到 `WorkerModule.imports` | **阻塞** |
|
||||
| `ReviewModule` 未在 WorkerModule | 如果选 A,需在 WorkerModule 中导入 `ReviewModule`(或拆出 `ReviewWorkerModule`) | **阻塞** |
|
||||
| `AuditLogProcessor` 在 `AdminAuditLogModule` 中 | 拆分为 `AdminAuditLogProducerModule` + `AdminAuditLogWorkerModule`,或将其移入 `WorkerModule` 并保留 Controller 在原 Module | 高 |
|
||||
| `FileCleanupProcessor` 未在 providers 中注册 | 补充注册,或显式移入 `WorkerModule` | 高 |
|
||||
|
||||
---
|
||||
|
||||
## 六、无法确认项
|
||||
|
||||
| 项目 | 状态 | 原因 |
|
||||
|------|:---:|------|
|
||||
| Docker Compose 当前是否同时启动 API 和 Worker | 无法确认 | 需检查 `docker-compose.yml` 中 `command` 或 `entrypoint` 指向 `dist/main.js` 还是两者 |
|
||||
| 生产 systemd 是否使用独立 Worker 进程 | 无法确认 | 无法访问生产服务器 |
|
||||
@ -1,169 +0,0 @@
|
||||
# ADR-001:统一 AI 架构迁移决策
|
||||
|
||||
## 状态
|
||||
|
||||
已接受(2026-06-19)
|
||||
|
||||
## 背景
|
||||
|
||||
当前知习项目存在两套独立的 AI 分析系统,以及一套规则出题系统:
|
||||
|
||||
- **系统 A(BullMQ + TypeScript,运行中)**:处理 active-recall 和 feynman-evaluation 分析,通过 `AiAnalysisWorker` 在 NestJS 进程内执行,写入 `AiAnalysisJob`/`AiAnalysisResult` 表,创建 `FocusItem`,发布 `ai.analysis.completed` 事件驱动 `ReviewCardSubscriber` 生成复习卡片。
|
||||
- **系统 B(Rust + HTTP 轮询,闲置)**:`zhixi-heavy-runtime` 独立进程,支持 5 种分析类型(learning_state_analysis、weak_point_analysis、next_action_planning、quiz_generation、flashcard_generation),通过轮询 `AiRuntimeJob` 表获取任务,调用 DeepSeek API,结果写入多个相互独立的领域表。
|
||||
- **系统 C(规则出题,运行中)**:`QuizService.create()` 通过纯文本拼接生成测验题目,不调用任何 AI 模型。
|
||||
|
||||
系统 B 存在以下关键问题:
|
||||
1. **无任务生产者** — 唯一创建 `AiRuntimeJob` 的入口 `POST /ai/jobs` 无人调用
|
||||
2. **Snapshot 结构不兼容** — Rust 端 `LearningAnalysisSnapshot` 结构体与 API 端 `SnapshotBuilderService` 产出的 JSON 结构完全不匹配
|
||||
3. **Rust 无重计算** — 所有 Pipeline 的独有业务逻辑总和不足 50 行,实际是一个 HTTP 代理层
|
||||
4. **数据断层** — Rust 结果写入的表(`AiLearningAnalysis`、`WeakPointCandidate`、`NextActionRecommendation`)无 iOS/Admin 消费端
|
||||
|
||||
同时,当前 API 进程内注册 BullMQ Processor 的部署拓扑存在以下问题:
|
||||
- API 进程与 Worker 进程边界模糊,Docker 与生产环境配置不一致
|
||||
- `AppModule` 中同时注册 `AiAnalysisWorker`、`DocumentImportWorker`、`NotificationWorker`
|
||||
- `WorkerModule` 和 `worker.main.ts` 已存在但未独立使用
|
||||
|
||||
## 决策
|
||||
|
||||
采用 **8 批次渐进式迁移**,将 AI 分析能力从 BullMQ + Rust 双系统统一到单一 BullMQ + TypeScript 架构。
|
||||
|
||||
### 目标架构拓扑
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ iOS App │
|
||||
└──────┬──────┘
|
||||
│ HTTP
|
||||
┌──────▼──────┐
|
||||
│ zhixi-api │ NestJS (仅 Controller + Queue Producer)
|
||||
│ :3000 │ 不注册任何 @Processor()
|
||||
└──────┬──────┘
|
||||
│ Redis (BullMQ)
|
||||
┌──────▼──────┐
|
||||
│ zhixi-worker│ NestJS (仅 @Processor())
|
||||
│ (无端口) │ AiAnalysisWorker + DocumentImportWorker
|
||||
└──────┬──────┘ + NotificationWorker + AuditLogProcessor
|
||||
│ HTTP + FileCleanupProcessor
|
||||
┌──────▼──────┐
|
||||
│ DeepSeek │
|
||||
│ API │
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
### API Process 职责
|
||||
|
||||
- 暴露所有 HTTP Controller
|
||||
- 注入 Queue Producer(`QueueService.add()`)
|
||||
- 不注册任何 `@Processor()` 或 `WorkerHost`
|
||||
- 不启动 BullMQ Worker 实例
|
||||
|
||||
### Worker Process 职责
|
||||
|
||||
- 注册所有 `@Processor()` 和 `WorkerHost`
|
||||
- 消费 BullMQ 队列
|
||||
- 调用 AI Workflow → DeepSeek API
|
||||
- 写入分析结果表
|
||||
- 发布领域事件
|
||||
- 不暴露 HTTP 端口
|
||||
|
||||
### Heavy Runtime 当前处置
|
||||
|
||||
- 第一阶段保持部署但不修改
|
||||
- 不停止 Rust 进程
|
||||
- 不删除 `AiRuntimeJob` 及相关表
|
||||
- M-AI-08 正式退场
|
||||
|
||||
## 迁移批次
|
||||
|
||||
| 批次 | 名称 | 范围 |
|
||||
|------|------|------|
|
||||
| M-AI-01 | Worker 进程边界与部署收口 | 拆分 API/Worker 进程,不改变业务 |
|
||||
| M-AI-02 | 统一 AiJob 数据库 Expand | 扩展 `AiAnalysisJob`,新增 Artifact/Snapshot/Outbox |
|
||||
| M-AI-03 | 统一 Job Engine、Registry 与 Outbox | AiJobService + Registry + 双队列 |
|
||||
| M-AI-04 | Active Recall 端到端迁移 | 切换生产者,iOS 展示真实结果 |
|
||||
| M-AI-05 | Feynman 与 ReviewCard 可靠链路 | Outbox 替代进程内 EventEmitter |
|
||||
| M-AI-06 | AI Quiz 端到端迁移 | 迁移 Rust Prompt/Schema,异步生成 |
|
||||
| M-AI-07 | 后台学习分析与推荐闭环 | learning_state/weak_point/next_action |
|
||||
| M-AI-08 | Heavy Runtime 退场与 Legacy Contract | 停止 Rust,归档表 |
|
||||
|
||||
## 第一阶段非目标(M-AI-01)
|
||||
|
||||
1. 不修改 `AiAnalysisJob` / `AiRuntimeJob` 表结构
|
||||
2. 不新建统一 AiJob 模型
|
||||
3. 不迁移 Active Recall、Feynman 或 Quiz 业务链路
|
||||
4. 不停止 `zhixi-heavy-runtime` 服务
|
||||
5. 不修改 iOS 代码
|
||||
6. 不删除任何数据库表
|
||||
7. 不修改 AI Prompt、模型或业务结果结构
|
||||
8. 不新增 Prisma Migration
|
||||
|
||||
## 架构约束
|
||||
|
||||
以下约束在所有批次中持续生效:
|
||||
|
||||
1. `AppModule` 不允许注册 `@Processor()` 或 `WorkerHost`
|
||||
2. `WorkerModule` 不允许暴露公开 HTTP Controller
|
||||
3. 不允许新建第三套 AI Job 模型(除 M-AI-02 统一扩展外)
|
||||
4. 不允许顺带修改 Prompt、模型选择或业务结果 Schema(各自独立 Issue)
|
||||
5. 没有 iOS/Admin 消费者的任务类型不得启用自动生成
|
||||
6. 自动触发必须有频率限制、幂等保护、成本上限
|
||||
|
||||
## 回滚原则
|
||||
|
||||
每个 Issue 必须提供独立回滚方案:
|
||||
|
||||
1. 代码层面:回滚提交或 revert PR
|
||||
2. 数据库:本批次不涉及 Schema 变更,无需数据回滚
|
||||
3. 部署:恢复旧 AppModule Processor 注册,停止独立 Worker service
|
||||
4. 队列:验证无双重消费
|
||||
|
||||
## 现有 Processor 清单
|
||||
|
||||
| Processor | 队列 | 文件 |
|
||||
|-----------|------|------|
|
||||
| `AiAnalysisWorker` | `ai-analysis` | `workers/ai-analysis.worker.ts` |
|
||||
| `DocumentImportWorker` | `document-import` | `workers/document-import.worker.ts` |
|
||||
| `NotificationWorker` | `notification` | `workers/notification.worker.ts` |
|
||||
| `AuditLogProcessor` | `audit-logs` | `modules/admin-audit-log/audit-log.processor.ts` |
|
||||
| `FileCleanupProcessor` | `file-cleanup` | `modules/files/file-cleanup.processor.ts` |
|
||||
|
||||
## 模块依赖图(当前)
|
||||
|
||||
```
|
||||
AppModule
|
||||
├── WorkerModule (已存在但未独立使用)
|
||||
│ ├── AiAnalysisWorker
|
||||
│ ├── DocumentImportWorker
|
||||
│ └── NotificationWorker
|
||||
├── QueueModule
|
||||
│ ├── BullModule.forRootAsync (Redis 连接)
|
||||
│ └── BullModule.registerQueue (6 个队列)
|
||||
├── AiAnalysisModule
|
||||
│ ├── AiAnalysisController (HTTP 端点)
|
||||
│ └── AiAnalysisService (BullMQ Producer)
|
||||
├── ActiveRecallModule
|
||||
│ └── ActiveRecallService (BullMQ Producer 调用者)
|
||||
├── ReviewModule
|
||||
│ ├── ReviewController (HTTP 端点)
|
||||
│ ├── ReviewCardSubscriber (监听 ai.analysis.completed)
|
||||
│ └── ReviewService
|
||||
├── AiRuntimeModule
|
||||
│ ├── UserAiController (HTTP 端点)
|
||||
│ └── RuntimeInternalController (Rust 内部端点)
|
||||
├── AdminAuditLogModule
|
||||
│ └── AuditLogProcessor (@Processor)
|
||||
└── FilesModule
|
||||
└── FileCleanupProcessor (@Processor)
|
||||
```
|
||||
|
||||
## 后续 Issue 引用
|
||||
|
||||
后续 M-AI-01 Issue 必须引用本文档(ADR-001)作为架构约束依据。
|
||||
|
||||
## 证据
|
||||
|
||||
- 当前部署:`docker-compose.yml:1-70` — API 和 Worker 共享同一 NestJS 实例
|
||||
- Worker 模块:`worker.module.ts:1-20` — 已定义但未独立使用
|
||||
- Worker 入口:`worker.main.ts:1-10` — 已存在但未被 Docker/生产独立调用
|
||||
- 现有架构文档:`docs/ai-runtime-architecture.md` — 描述 Rust Heavy Runtime 架构
|
||||
- BullMQ 版本:`bullmq@5.77.1`、`@nestjs/bullmq@11.0.4`
|
||||
@ -11,8 +11,7 @@
|
||||
"start": "nest start",
|
||||
"start:dev": "nest start --watch",
|
||||
"start:debug": "nest start --debug --watch",
|
||||
"start:prod": "node dist/src/main",
|
||||
"start:worker": "node dist/src/worker.main",
|
||||
"start:prod": "node dist/main",
|
||||
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
|
||||
"test": "jest",
|
||||
"test:watch": "jest --watch",
|
||||
|
||||
@ -79,6 +79,9 @@ import { MetricsInterceptor } from './common/interceptors/metrics.interceptor';
|
||||
import { TimeoutInterceptor } from './common/interceptors/timeout.interceptor';
|
||||
import { AppThrottleModule } from './common/throttle/throttle.module';
|
||||
|
||||
import { AiAnalysisWorker } from './workers/ai-analysis.worker';
|
||||
import { DocumentImportWorker } from './workers/document-import.worker';
|
||||
import { NotificationWorker } from './workers/notification.worker';
|
||||
|
||||
import appConfig from './config/app.config';
|
||||
import databaseConfig from './config/database.config';
|
||||
@ -187,6 +190,9 @@ import appleConfig from './config/apple.config';
|
||||
{ provide: APP_INTERCEPTOR, useClass: MetricsInterceptor },
|
||||
{ provide: APP_INTERCEPTOR, useClass: TimeoutInterceptor },
|
||||
{ provide: APP_INTERCEPTOR, useClass: ResponseInterceptor },
|
||||
AiAnalysisWorker,
|
||||
DocumentImportWorker,
|
||||
NotificationWorker,
|
||||
],
|
||||
})
|
||||
|
||||
|
||||
@ -1,196 +0,0 @@
|
||||
import {
|
||||
QUEUE_DEFINITIONS,
|
||||
queueDef,
|
||||
workerOptionsFor,
|
||||
defaultJobOptionsFor,
|
||||
} from './queue-definitions';
|
||||
import {
|
||||
QUEUE_AI_ANALYSIS,
|
||||
QUEUE_DOCUMENT_IMPORT,
|
||||
QUEUE_NOTIFICATION,
|
||||
QUEUE_DOMAIN_EVENTS,
|
||||
QUEUE_AUDIT_LOG,
|
||||
QUEUE_FILE_CLEANUP,
|
||||
} from './queue.service';
|
||||
|
||||
describe('Queue Definition Registry', () => {
|
||||
const allQueues = [
|
||||
QUEUE_AI_ANALYSIS,
|
||||
QUEUE_DOCUMENT_IMPORT,
|
||||
QUEUE_NOTIFICATION,
|
||||
QUEUE_DOMAIN_EVENTS,
|
||||
QUEUE_AUDIT_LOG,
|
||||
QUEUE_FILE_CLEANUP,
|
||||
];
|
||||
|
||||
describe('QUEUE_DEFINITIONS', () => {
|
||||
it('contains all 6 queues', () => {
|
||||
expect(Object.keys(QUEUE_DEFINITIONS)).toHaveLength(6);
|
||||
for (const name of allQueues) {
|
||||
expect(QUEUE_DEFINITIONS[name]).toBeDefined();
|
||||
}
|
||||
});
|
||||
|
||||
it('every queue has required workerOptions fields', () => {
|
||||
for (const def of Object.values(QUEUE_DEFINITIONS)) {
|
||||
expect(def.workerOptions.concurrency).toBeGreaterThanOrEqual(1);
|
||||
expect(def.workerOptions.lockDuration).toBeGreaterThan(0);
|
||||
expect(def.workerOptions.stalledInterval).toBeGreaterThan(0);
|
||||
expect(def.workerOptions.maxStalledCount).toBeGreaterThanOrEqual(1);
|
||||
}
|
||||
});
|
||||
|
||||
it('every queue has required defaultJobOptions fields', () => {
|
||||
for (const def of Object.values(QUEUE_DEFINITIONS)) {
|
||||
expect(def.defaultJobOptions.attempts).toBeGreaterThanOrEqual(1);
|
||||
expect(def.defaultJobOptions.backoff.type).toBe('exponential');
|
||||
expect(def.defaultJobOptions.backoff.delay).toBeGreaterThan(0);
|
||||
expect(def.defaultJobOptions.removeOnComplete.count).toBeGreaterThan(0);
|
||||
expect(def.defaultJobOptions.removeOnComplete.age).toBeGreaterThan(0);
|
||||
expect(def.defaultJobOptions.removeOnFail.count).toBeGreaterThan(0);
|
||||
expect(def.defaultJobOptions.removeOnFail.age).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
|
||||
it('applies first-stage default values', () => {
|
||||
for (const def of Object.values(QUEUE_DEFINITIONS)) {
|
||||
expect(def.workerOptions.concurrency).toBe(1);
|
||||
expect(def.defaultJobOptions.attempts).toBe(3);
|
||||
expect(def.defaultJobOptions.backoff.delay).toBe(1000);
|
||||
expect(def.defaultJobOptions.backoff.type).toBe('exponential');
|
||||
expect(def.defaultJobOptions.removeOnComplete.count).toBe(1000);
|
||||
expect(def.defaultJobOptions.removeOnComplete.age).toBe(24 * 3600);
|
||||
expect(def.defaultJobOptions.removeOnFail.count).toBe(5000);
|
||||
expect(def.defaultJobOptions.removeOnFail.age).toBe(7 * 24 * 3600);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('queueDef', () => {
|
||||
it('returns definition for valid queue name', () => {
|
||||
const def = queueDef(QUEUE_AI_ANALYSIS);
|
||||
expect(def.name).toBe(QUEUE_AI_ANALYSIS);
|
||||
});
|
||||
|
||||
it('throws for unknown queue name', () => {
|
||||
expect(() => queueDef('nonexistent')).toThrow('Unknown queue');
|
||||
});
|
||||
});
|
||||
|
||||
describe('workerOptionsFor', () => {
|
||||
it('returns worker options for a queue', () => {
|
||||
const opts = workerOptionsFor(QUEUE_DOCUMENT_IMPORT);
|
||||
expect(opts.concurrency).toBe(1);
|
||||
expect(opts.lockDuration).toBe(30_000);
|
||||
});
|
||||
});
|
||||
|
||||
describe('defaultJobOptionsFor', () => {
|
||||
it('returns job options for a queue', () => {
|
||||
const opts = defaultJobOptionsFor(QUEUE_NOTIFICATION);
|
||||
expect(opts.attempts).toBe(3);
|
||||
expect(opts.backoff.type).toBe('exponential');
|
||||
});
|
||||
});
|
||||
|
||||
describe('environment variable overrides', () => {
|
||||
let queueDefs: typeof import('./queue-definitions');
|
||||
const savedEnv = { ...process.env };
|
||||
|
||||
beforeEach(() => {
|
||||
jest.resetModules();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
// Restore env to prevent cross-test leakage
|
||||
for (const key of Object.keys(process.env)) {
|
||||
if (!(key in savedEnv)) delete process.env[key];
|
||||
else process.env[key] = savedEnv[key];
|
||||
}
|
||||
});
|
||||
|
||||
const loadDefs = () => {
|
||||
queueDefs = require('./queue-definitions');
|
||||
};
|
||||
|
||||
it('respects BULL_AI_ANALYSIS_CONCURRENCY override', () => {
|
||||
process.env.BULL_AI_ANALYSIS_CONCURRENCY = '5';
|
||||
loadDefs();
|
||||
const def = queueDefs.QUEUE_DEFINITIONS[QUEUE_AI_ANALYSIS];
|
||||
expect(def.workerOptions.concurrency).toBe(5);
|
||||
});
|
||||
|
||||
it('respects BULL_AI_ANALYSIS_ATTEMPTS override', () => {
|
||||
process.env.BULL_AI_ANALYSIS_ATTEMPTS = '1';
|
||||
loadDefs();
|
||||
const def = queueDefs.QUEUE_DEFINITIONS[QUEUE_AI_ANALYSIS];
|
||||
expect(def.defaultJobOptions.attempts).toBe(1);
|
||||
});
|
||||
|
||||
it('rejects concurrency=0 via min bound and falls back', () => {
|
||||
process.env.BULL_AI_ANALYSIS_CONCURRENCY = '0';
|
||||
loadDefs();
|
||||
const def = queueDefs.QUEUE_DEFINITIONS[QUEUE_AI_ANALYSIS];
|
||||
expect(def.workerOptions.concurrency).toBe(1);
|
||||
});
|
||||
|
||||
it('uses fallback when env var is empty string', () => {
|
||||
process.env.BULL_AI_ANALYSIS_ATTEMPTS = '';
|
||||
loadDefs();
|
||||
const def = queueDefs.QUEUE_DEFINITIONS[QUEUE_AI_ANALYSIS];
|
||||
expect(def.defaultJobOptions.attempts).toBe(3);
|
||||
});
|
||||
|
||||
it('uses fallback when env var is non-numeric', () => {
|
||||
process.env.BULL_AI_ANALYSIS_CONCURRENCY = 'abc';
|
||||
loadDefs();
|
||||
const def = queueDefs.QUEUE_DEFINITIONS[QUEUE_AI_ANALYSIS];
|
||||
expect(def.workerOptions.concurrency).toBe(1);
|
||||
});
|
||||
|
||||
it('uses fallback when env var is negative', () => {
|
||||
process.env.BULL_AI_ANALYSIS_CONCURRENCY = '-1';
|
||||
loadDefs();
|
||||
const def = queueDefs.QUEUE_DEFINITIONS[QUEUE_AI_ANALYSIS];
|
||||
expect(def.workerOptions.concurrency).toBe(1);
|
||||
});
|
||||
|
||||
it('respects global BULL_COMPLETED_MAX_COUNT for all queues', () => {
|
||||
process.env.BULL_COMPLETED_MAX_COUNT = '500';
|
||||
loadDefs();
|
||||
for (const def of Object.values(queueDefs.QUEUE_DEFINITIONS)) {
|
||||
expect(def.defaultJobOptions.removeOnComplete.count).toBe(500);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('global cleanup env vars', () => {
|
||||
it('applies BULL_COMPLETED_MAX_COUNT to all queues', () => {
|
||||
for (const def of Object.values(QUEUE_DEFINITIONS)) {
|
||||
expect(def.defaultJobOptions.removeOnComplete.count).toBe(1000);
|
||||
}
|
||||
});
|
||||
|
||||
it('applies BULL_FAILED_MAX_AGE_SEC to all queues', () => {
|
||||
for (const def of Object.values(QUEUE_DEFINITIONS)) {
|
||||
expect(def.defaultJobOptions.removeOnFail.age).toBe(7 * 24 * 3600);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('lockDuration vs stalled vs business timeout', () => {
|
||||
it('lockDuration (30s) ≠ business timeout — documented constraint', () => {
|
||||
// lockDuration is BullMQ lock renewal interval.
|
||||
// If the processor runs past lockDuration but the worker is alive,
|
||||
// BullMQ auto-extends the lock. Only if the worker crashes does
|
||||
// stalledInterval (30s) fire, and maxStalledCount (1) controls
|
||||
// how many times the job can be retried by another worker.
|
||||
// Business timeouts MUST be enforced inside processor logic.
|
||||
for (const def of Object.values(QUEUE_DEFINITIONS)) {
|
||||
expect(def.workerOptions.lockDuration).toBe(30_000);
|
||||
expect(def.workerOptions.stalledInterval).toBe(30_000);
|
||||
expect(def.workerOptions.maxStalledCount).toBe(1);
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
@ -1,115 +0,0 @@
|
||||
import {
|
||||
QUEUE_AI_ANALYSIS,
|
||||
QUEUE_DOCUMENT_IMPORT,
|
||||
QUEUE_NOTIFICATION,
|
||||
QUEUE_DOMAIN_EVENTS,
|
||||
QUEUE_AUDIT_LOG,
|
||||
QUEUE_FILE_CLEANUP,
|
||||
} from './queue.service';
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════
|
||||
// Queue Definition Registry
|
||||
//
|
||||
// Centralizes BullMQ worker options and default job options for every queue.
|
||||
// Environment variables override the defaults defined here.
|
||||
//
|
||||
// lockDuration is NOT a business timeout — it is the BullMQ lock renewal
|
||||
// interval. A processor that runs longer than lockDuration will have its
|
||||
// lock automatically extended by the worker, unless the worker crashes
|
||||
// (then stalledInterval detects the stall and maxStalledCount controls
|
||||
// retry). Business timeouts must be enforced inside the processor logic.
|
||||
// ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
function envInt(key: string, fallback: number, min?: number): number {
|
||||
const raw = process.env[key];
|
||||
if (raw === undefined || raw === '') return fallback;
|
||||
const n = parseInt(raw, 10);
|
||||
if (isNaN(n) || n < 0) return fallback;
|
||||
if (min !== undefined && n < min) return fallback;
|
||||
return n;
|
||||
}
|
||||
|
||||
export interface QueueWorkerOptions {
|
||||
concurrency: number;
|
||||
lockDuration: number;
|
||||
stalledInterval: number;
|
||||
maxStalledCount: number;
|
||||
}
|
||||
|
||||
export interface QueueDefaultJobOptions {
|
||||
attempts: number;
|
||||
backoff: { type: 'exponential'; delay: number };
|
||||
removeOnComplete: { count: number; age: number };
|
||||
removeOnFail: { count: number; age: number };
|
||||
}
|
||||
|
||||
export interface QueueDefinition {
|
||||
name: string;
|
||||
workerOptions: QueueWorkerOptions;
|
||||
defaultJobOptions: QueueDefaultJobOptions;
|
||||
}
|
||||
|
||||
const DEFAULT_WORKER: QueueWorkerOptions = {
|
||||
concurrency: 1,
|
||||
lockDuration: 30_000,
|
||||
stalledInterval: 30_000,
|
||||
maxStalledCount: 1,
|
||||
};
|
||||
|
||||
const DEFAULT_JOB: QueueDefaultJobOptions = {
|
||||
attempts: 3,
|
||||
backoff: { type: 'exponential', delay: 1000 },
|
||||
removeOnComplete: { count: 1000, age: 24 * 3600 },
|
||||
removeOnFail: { count: 5000, age: 7 * 24 * 3600 },
|
||||
};
|
||||
|
||||
function buildDef(name: string): QueueDefinition {
|
||||
const prefix = name.toUpperCase().replace(/-/g, '_');
|
||||
return {
|
||||
name,
|
||||
workerOptions: {
|
||||
concurrency: envInt(`BULL_${prefix}_CONCURRENCY`, DEFAULT_WORKER.concurrency, 1),
|
||||
lockDuration: envInt(`BULL_${prefix}_LOCK_DURATION`, DEFAULT_WORKER.lockDuration, 1),
|
||||
stalledInterval: envInt(`BULL_${prefix}_STALLED_INTERVAL`, DEFAULT_WORKER.stalledInterval, 1),
|
||||
maxStalledCount: envInt(`BULL_${prefix}_MAX_STALLED`, DEFAULT_WORKER.maxStalledCount, 1),
|
||||
},
|
||||
defaultJobOptions: {
|
||||
attempts: envInt(`BULL_${prefix}_ATTEMPTS`, DEFAULT_JOB.attempts, 1),
|
||||
backoff: {
|
||||
type: 'exponential',
|
||||
delay: envInt(`BULL_${prefix}_BACKOFF_DELAY`, DEFAULT_JOB.backoff.delay),
|
||||
},
|
||||
removeOnComplete: {
|
||||
count: envInt('BULL_COMPLETED_MAX_COUNT', DEFAULT_JOB.removeOnComplete.count),
|
||||
age: envInt('BULL_COMPLETED_MAX_AGE_SEC', DEFAULT_JOB.removeOnComplete.age),
|
||||
},
|
||||
removeOnFail: {
|
||||
count: envInt('BULL_FAILED_MAX_COUNT', DEFAULT_JOB.removeOnFail.count),
|
||||
age: envInt('BULL_FAILED_MAX_AGE_SEC', DEFAULT_JOB.removeOnFail.age),
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export const QUEUE_DEFINITIONS: Record<string, QueueDefinition> = {
|
||||
[QUEUE_AI_ANALYSIS]: buildDef(QUEUE_AI_ANALYSIS),
|
||||
[QUEUE_DOCUMENT_IMPORT]: buildDef(QUEUE_DOCUMENT_IMPORT),
|
||||
[QUEUE_NOTIFICATION]: buildDef(QUEUE_NOTIFICATION),
|
||||
[QUEUE_DOMAIN_EVENTS]: buildDef(QUEUE_DOMAIN_EVENTS),
|
||||
[QUEUE_AUDIT_LOG]: buildDef(QUEUE_AUDIT_LOG),
|
||||
[QUEUE_FILE_CLEANUP]: buildDef(QUEUE_FILE_CLEANUP),
|
||||
};
|
||||
|
||||
export function queueDef(name: string): QueueDefinition {
|
||||
const def = QUEUE_DEFINITIONS[name];
|
||||
if (!def) throw new Error(`Unknown queue: ${name}`);
|
||||
return def;
|
||||
}
|
||||
|
||||
export function workerOptionsFor(name: string): QueueWorkerOptions {
|
||||
return queueDef(name).workerOptions;
|
||||
}
|
||||
|
||||
export function defaultJobOptionsFor(name: string): QueueDefaultJobOptions {
|
||||
return queueDef(name).defaultJobOptions;
|
||||
}
|
||||
@ -4,7 +4,6 @@ import { EventBusService } from '../../common/event-bus/event-bus.service';
|
||||
import { BaseDomainEvent } from '../../common/events/base-domain.event';
|
||||
import { PrismaService } from '../database/prisma.service';
|
||||
import { Queue } from 'bullmq';
|
||||
import { defaultJobOptionsFor } from './queue-definitions';
|
||||
|
||||
export const QUEUE_AI_ANALYSIS = 'ai-analysis';
|
||||
export const QUEUE_DOCUMENT_IMPORT = 'document-import';
|
||||
@ -22,22 +21,12 @@ export class QueueService {
|
||||
@InjectQueue(QUEUE_AI_ANALYSIS) private readonly aiQueue: Queue,
|
||||
@InjectQueue(QUEUE_DOCUMENT_IMPORT) private readonly importQueue: Queue,
|
||||
@InjectQueue(QUEUE_NOTIFICATION) private readonly notifyQueue: Queue,
|
||||
@InjectQueue(QUEUE_DOMAIN_EVENTS) private readonly domainEventsQueue: Queue,
|
||||
@InjectQueue(QUEUE_AUDIT_LOG) private readonly auditLogQueue: Queue,
|
||||
@InjectQueue(QUEUE_FILE_CLEANUP) private readonly fileCleanupQueue: Queue,
|
||||
@Optional() private readonly eventBus?: any,
|
||||
) {}
|
||||
|
||||
async add(queueName: string, data: any, opts?: { jobId?: string; attempts?: number; backoff?: number }) {
|
||||
const queue = this.getQueue(queueName);
|
||||
const jobOpts = defaultJobOptionsFor(queueName);
|
||||
const job = await queue.add(queueName, data, {
|
||||
attempts: jobOpts.attempts,
|
||||
backoff: jobOpts.backoff,
|
||||
removeOnComplete: jobOpts.removeOnComplete,
|
||||
removeOnFail: jobOpts.removeOnFail,
|
||||
...opts,
|
||||
});
|
||||
const job = await queue.add(queueName, data, { attempts: 3, backoff: { type: 'exponential', delay: 1000 }, ...opts });
|
||||
|
||||
// Log to DB
|
||||
await this.prisma.taskLog.create({ data: { queueName, jobId: job.id || '', status: 'enqueued', payload: JSON.parse(JSON.stringify(data)) } }).catch(() => {});
|
||||
@ -57,9 +46,6 @@ export class QueueService {
|
||||
case QUEUE_AI_ANALYSIS: return this.aiQueue;
|
||||
case QUEUE_DOCUMENT_IMPORT: return this.importQueue;
|
||||
case QUEUE_NOTIFICATION: return this.notifyQueue;
|
||||
case QUEUE_DOMAIN_EVENTS: return this.domainEventsQueue;
|
||||
case QUEUE_AUDIT_LOG: return this.auditLogQueue;
|
||||
case QUEUE_FILE_CLEANUP: return this.fileCleanupQueue;
|
||||
default: throw new Error(`Unknown queue: ${name}`);
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,12 +1,15 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { BullModule } from '@nestjs/bullmq';
|
||||
import { AuditLogProcessor } from './audit-log.processor';
|
||||
import { PrismaService } from '../../infrastructure/database/prisma.service';
|
||||
import { QUEUE_AUDIT_LOG } from '../../infrastructure/queue/queue.service';
|
||||
import { AdminAuditLogController } from './admin-audit-log.controller';
|
||||
import { AdminAuditLogService } from './admin-audit-log.service';
|
||||
import { AdminAuthModule } from '../admin-auth/admin-auth.module';
|
||||
|
||||
@Module({
|
||||
imports: [AdminAuthModule],
|
||||
imports: [AdminAuthModule, BullModule.registerQueue({ name: QUEUE_AUDIT_LOG })],
|
||||
controllers: [AdminAuditLogController],
|
||||
providers: [PrismaService, AdminAuditLogService],
|
||||
providers: [AuditLogProcessor, PrismaService,AdminAuditLogService],
|
||||
})
|
||||
export class AdminAuditLogModule {}
|
||||
|
||||
@ -2,9 +2,8 @@ import { Processor, WorkerHost } from '@nestjs/bullmq';
|
||||
import { Job } from 'bullmq';
|
||||
import { PrismaService } from '../../infrastructure/database/prisma.service';
|
||||
import { QUEUE_AUDIT_LOG } from '../../infrastructure/queue/queue.service';
|
||||
import { workerOptionsFor } from '../../infrastructure/queue/queue-definitions';
|
||||
|
||||
@Processor(QUEUE_AUDIT_LOG, workerOptionsFor(QUEUE_AUDIT_LOG))
|
||||
@Processor(QUEUE_AUDIT_LOG)
|
||||
export class AuditLogProcessor extends WorkerHost {
|
||||
constructor(private readonly prisma: PrismaService) { super(); }
|
||||
|
||||
|
||||
@ -26,11 +26,4 @@ export class AdminServersController {
|
||||
async getHealth() {
|
||||
return this.serversService.getHealthChecks();
|
||||
}
|
||||
|
||||
@Get('workers')
|
||||
@AdminRoles('SUPER_ADMIN' as AdminRole)
|
||||
@ApiOperation({ summary: 'Worker 心跳状态(online/stale/offline)' })
|
||||
async getWorkers() {
|
||||
return this.serversService.getWorkerHeartbeats();
|
||||
}
|
||||
}
|
||||
|
||||
@ -2,8 +2,6 @@ import { Injectable, Logger } from '@nestjs/common';
|
||||
import * as os from 'os';
|
||||
import { exec } from 'child_process';
|
||||
import { promisify } from 'util';
|
||||
import { RedisService } from '../../infrastructure/redis/redis.service';
|
||||
import { WorkerHeartbeatService } from '../../workers/worker-heartbeat.service';
|
||||
|
||||
const execAsync = promisify(exec);
|
||||
|
||||
@ -71,8 +69,6 @@ function chineseUptime(sec: number): string {
|
||||
export class AdminServersService {
|
||||
private readonly logger = new Logger(AdminServersService.name);
|
||||
|
||||
constructor(private readonly redis: RedisService) {}
|
||||
|
||||
async getLocalMetrics(): Promise<ServerInfo> {
|
||||
const cpus = os.cpus();
|
||||
const total = os.totalmem(), free = os.freemem(), used = total - free;
|
||||
@ -157,18 +153,6 @@ export class AdminServersService {
|
||||
return { servers: [local, ...(remote ? [remote] : [])] };
|
||||
}
|
||||
|
||||
// ═══ Worker Heartbeats ═══
|
||||
|
||||
async getWorkerHeartbeats() {
|
||||
try {
|
||||
const statuses = await WorkerHeartbeatService.computeStatuses(this.redis);
|
||||
return { workers: statuses };
|
||||
} catch (err: any) {
|
||||
this.logger.warn(`Worker heartbeat query failed: ${err.message}`);
|
||||
return { workers: [] };
|
||||
}
|
||||
}
|
||||
|
||||
// ═══ Health Checks ═══
|
||||
|
||||
async getHealthChecks() {
|
||||
|
||||
@ -2,10 +2,9 @@ import { Processor, WorkerHost } from '@nestjs/bullmq';
|
||||
import { Job } from 'bullmq';
|
||||
import { Logger } from '@nestjs/common';
|
||||
import { QUEUE_FILE_CLEANUP } from '../../infrastructure/queue/queue.service';
|
||||
import { workerOptionsFor } from '../../infrastructure/queue/queue-definitions';
|
||||
import { CosStorageProvider } from '../../infrastructure/storage/cos-storage.provider';
|
||||
|
||||
@Processor(QUEUE_FILE_CLEANUP, workerOptionsFor(QUEUE_FILE_CLEANUP))
|
||||
@Processor(QUEUE_FILE_CLEANUP)
|
||||
export class FileCleanupProcessor extends WorkerHost {
|
||||
private readonly logger = new Logger(FileCleanupProcessor.name);
|
||||
|
||||
|
||||
@ -1,4 +1,7 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { BullModule } from '@nestjs/bullmq';
|
||||
import { FileCleanupProcessor } from './file-cleanup.processor';
|
||||
import { QUEUE_FILE_CLEANUP } from '../../infrastructure/queue/queue.service';
|
||||
import { FilesController } from './files.controller';
|
||||
import { AdminFilesController } from './admin-files.controller';
|
||||
import { FilesService } from './files.service';
|
||||
|
||||
@ -1,120 +1,12 @@
|
||||
import { NestFactory } from '@nestjs/core';
|
||||
import { WorkerModule } from './worker.module';
|
||||
import { ConfigService } from '@nestjs/config';
|
||||
import { PrismaService } from './infrastructure/database/prisma.service';
|
||||
import { RedisService } from './infrastructure/redis/redis.service';
|
||||
import { Logger } from '@nestjs/common';
|
||||
import * as os from 'os';
|
||||
|
||||
function generateInstanceId(): string {
|
||||
const hostname = os.hostname();
|
||||
const pid = process.pid;
|
||||
const timestamp = Date.now();
|
||||
return `worker-${hostname}-${pid}-${timestamp}`;
|
||||
}
|
||||
|
||||
async function validatePrerequisites(
|
||||
prisma: PrismaService,
|
||||
redis: RedisService,
|
||||
config: ConfigService,
|
||||
logger: Logger,
|
||||
): Promise<void> {
|
||||
// Validate MySQL
|
||||
try {
|
||||
await prisma.$queryRaw`SELECT 1`;
|
||||
logger.log('MySQL connection verified');
|
||||
} catch (err: any) {
|
||||
logger.error(`MySQL connection failed: ${err.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Validate Redis
|
||||
try {
|
||||
await redis.get('__worker_health_check__');
|
||||
logger.log('Redis connection verified');
|
||||
} catch (err: any) {
|
||||
logger.error(`Redis connection failed: ${err.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Validate required config keys
|
||||
const requiredKeys = ['database.url', 'redis.host', 'jwt.secret'];
|
||||
const missing = requiredKeys.filter(k => {
|
||||
const val = config.get(k);
|
||||
if (val === undefined || val === null || val === '') return true;
|
||||
if (typeof val === 'string' && val.startsWith('change_me')) return true;
|
||||
return false;
|
||||
});
|
||||
|
||||
if (missing.length > 0) {
|
||||
logger.error(`Missing required config keys: ${missing.join(', ')}`);
|
||||
process.exit(1);
|
||||
}
|
||||
logger.log('Required configuration verified');
|
||||
}
|
||||
|
||||
async function bootstrap() {
|
||||
const logger = new Logger('WorkerBootstrap');
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-var-requires
|
||||
const pkg = require('../package.json');
|
||||
const instanceId = generateInstanceId();
|
||||
process.env.WORKER_INSTANCE_ID = instanceId;
|
||||
logger.log(`Worker version ${pkg.version} — instance ${instanceId}`);
|
||||
|
||||
const app = await NestFactory.createApplicationContext(WorkerModule);
|
||||
|
||||
const config = app.get(ConfigService);
|
||||
const prisma = app.get(PrismaService);
|
||||
const redis = app.get(RedisService);
|
||||
|
||||
await validatePrerequisites(prisma, redis, config, logger);
|
||||
|
||||
// Log queue configuration
|
||||
try {
|
||||
const { QUEUE_DEFINITIONS } = require('./infrastructure/queue/queue-definitions');
|
||||
logger.log('Queue configuration:');
|
||||
for (const [name, def] of Object.entries(QUEUE_DEFINITIONS) as [string, any][]) {
|
||||
logger.log(
|
||||
` ${name}: concurrency=${def.workerOptions.concurrency} attempts=${def.defaultJobOptions.attempts} backoff=exponential:${def.defaultJobOptions.backoff.delay}ms`,
|
||||
);
|
||||
}
|
||||
} catch {
|
||||
logger.warn('Could not load queue definitions for startup log');
|
||||
}
|
||||
|
||||
app.enableShutdownHooks();
|
||||
|
||||
logger.log(`Worker ${instanceId} started — waiting for jobs`);
|
||||
|
||||
// Graceful shutdown with forced timeout
|
||||
const SHUTDOWN_TIMEOUT_MS = 30_000;
|
||||
let shuttingDown = false;
|
||||
|
||||
const shutdown = (signal: string) => {
|
||||
if (shuttingDown) return;
|
||||
shuttingDown = true;
|
||||
|
||||
logger.log(`Received ${signal} — stopping job acquisition, waiting for active jobs (timeout ${SHUTDOWN_TIMEOUT_MS / 1000}s)...`);
|
||||
|
||||
const forceExit = setTimeout(() => {
|
||||
logger.error(`Shutdown timed out after ${SHUTDOWN_TIMEOUT_MS / 1000}s — forcing exit`);
|
||||
process.exit(1);
|
||||
}, SHUTDOWN_TIMEOUT_MS);
|
||||
|
||||
app.close().then(() => {
|
||||
clearTimeout(forceExit);
|
||||
logger.log('Worker shut down gracefully');
|
||||
process.exit(0);
|
||||
}).catch((err: any) => {
|
||||
clearTimeout(forceExit);
|
||||
logger.error(`Worker shutdown error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
};
|
||||
|
||||
process.on('SIGTERM', () => shutdown('SIGTERM'));
|
||||
process.on('SIGINT', () => shutdown('SIGINT'));
|
||||
console.log('[Worker] BullMQ workers started — waiting for jobs');
|
||||
}
|
||||
|
||||
bootstrap();
|
||||
|
||||
@ -13,17 +13,10 @@ import { AiAnalysisWorker } from './workers/ai-analysis.worker';
|
||||
import { DocumentImportWorker } from './workers/document-import.worker';
|
||||
import { NotificationWorker } from './workers/notification.worker';
|
||||
|
||||
import { AuditLogProcessor } from './modules/admin-audit-log/audit-log.processor';
|
||||
import { FileCleanupProcessor } from './modules/files/file-cleanup.processor';
|
||||
import { WorkerHeartbeatService } from './workers/worker-heartbeat.service';
|
||||
|
||||
import { AiAnalysisModule } from './modules/ai-analysis/ai-analysis.module';
|
||||
import { DocumentImportModule } from './modules/document-import/document-import.module';
|
||||
import { KnowledgeItemsModule } from './modules/knowledge-items/knowledge-items.module';
|
||||
import { NotificationsModule } from './modules/notifications/notifications.module';
|
||||
import { EventBusModule } from './common/event-bus/event-bus.module';
|
||||
import { FocusItemsModule } from './modules/focus-items/focus-items.module';
|
||||
import { ReviewModule } from './modules/review/review.module';
|
||||
|
||||
import appConfig from './config/app.config';
|
||||
import databaseConfig from './config/database.config';
|
||||
@ -66,17 +59,11 @@ import appleConfig from './config/apple.config';
|
||||
DocumentImportModule,
|
||||
KnowledgeItemsModule,
|
||||
NotificationsModule,
|
||||
EventBusModule,
|
||||
FocusItemsModule,
|
||||
ReviewModule,
|
||||
],
|
||||
providers: [
|
||||
AiAnalysisWorker,
|
||||
DocumentImportWorker,
|
||||
NotificationWorker,
|
||||
AuditLogProcessor,
|
||||
FileCleanupProcessor,
|
||||
WorkerHeartbeatService,
|
||||
],
|
||||
})
|
||||
export class WorkerModule {}
|
||||
|
||||
@ -2,7 +2,6 @@ import { Processor, WorkerHost } from '@nestjs/bullmq';
|
||||
import { Logger, Optional } from '@nestjs/common';
|
||||
import { Job } from 'bullmq';
|
||||
import { QUEUE_AI_ANALYSIS } from '../infrastructure/queue/queue.service';
|
||||
import { workerOptionsFor } from '../infrastructure/queue/queue-definitions';
|
||||
import { EventBusService } from '../common/event-bus/event-bus.service';
|
||||
import { BaseDomainEvent } from '../common/events/base-domain.event';
|
||||
import { ActiveRecallAnalysisWorkflow } from '../modules/ai/workflows/active-recall-analysis.workflow';
|
||||
@ -15,7 +14,7 @@ class AIAnalysisCompleted extends BaseDomainEvent {
|
||||
constructor(public readonly payload: Record<string, any>) { super(); }
|
||||
}
|
||||
|
||||
@Processor(QUEUE_AI_ANALYSIS, workerOptionsFor(QUEUE_AI_ANALYSIS))
|
||||
@Processor(QUEUE_AI_ANALYSIS)
|
||||
export class AiAnalysisWorker extends WorkerHost {
|
||||
private readonly logger = new Logger(AiAnalysisWorker.name);
|
||||
|
||||
|
||||
@ -2,13 +2,12 @@ import { Processor, WorkerHost } from '@nestjs/bullmq';
|
||||
import { Logger } from '@nestjs/common';
|
||||
import { Job } from 'bullmq';
|
||||
import { QUEUE_DOCUMENT_IMPORT } from '../infrastructure/queue/queue.service';
|
||||
import { workerOptionsFor } from '../infrastructure/queue/queue-definitions';
|
||||
import { DocumentImportRepository } from '../modules/document-import/document-import.repository';
|
||||
import { KnowledgeItemsRepository } from '../modules/knowledge-items/knowledge-items.repository';
|
||||
import { KnowledgeImportWorkflow } from '../modules/ai/workflows/knowledge-import.workflow';
|
||||
import { RedisService } from '../infrastructure/redis/redis.service';
|
||||
|
||||
@Processor(QUEUE_DOCUMENT_IMPORT, workerOptionsFor(QUEUE_DOCUMENT_IMPORT))
|
||||
@Processor(QUEUE_DOCUMENT_IMPORT)
|
||||
export class DocumentImportWorker extends WorkerHost {
|
||||
private readonly logger = new Logger(DocumentImportWorker.name);
|
||||
|
||||
|
||||
@ -2,10 +2,9 @@ import { Processor, WorkerHost } from '@nestjs/bullmq';
|
||||
import { Logger } from '@nestjs/common';
|
||||
import { Job } from 'bullmq';
|
||||
import { QUEUE_NOTIFICATION } from '../infrastructure/queue/queue.service';
|
||||
import { workerOptionsFor } from '../infrastructure/queue/queue-definitions';
|
||||
import { NotificationsService } from '../modules/notifications/notifications.service';
|
||||
|
||||
@Processor(QUEUE_NOTIFICATION, workerOptionsFor(QUEUE_NOTIFICATION))
|
||||
@Processor(QUEUE_NOTIFICATION)
|
||||
export class NotificationWorker extends WorkerHost {
|
||||
private readonly logger = new Logger(NotificationWorker.name);
|
||||
|
||||
|
||||
@ -1,169 +0,0 @@
|
||||
import { WorkerHeartbeatService, WorkerStatus } from './worker-heartbeat.service';
|
||||
import { RedisService } from '../infrastructure/redis/redis.service';
|
||||
|
||||
describe('WorkerHeartbeatService', () => {
|
||||
let service: WorkerHeartbeatService;
|
||||
let mockRedis: jest.Mocked<Pick<RedisService, 'set' | 'get' | 'del' | 'keys' | 'ttl'>>;
|
||||
|
||||
beforeEach(() => {
|
||||
mockRedis = {
|
||||
set: jest.fn().mockResolvedValue(undefined),
|
||||
get: jest.fn().mockResolvedValue(null),
|
||||
del: jest.fn().mockResolvedValue(undefined),
|
||||
keys: jest.fn().mockResolvedValue([]),
|
||||
ttl: jest.fn().mockResolvedValue(-2),
|
||||
};
|
||||
|
||||
process.env.WORKER_INSTANCE_ID = 'test-worker-1';
|
||||
service = new WorkerHeartbeatService(mockRedis as any);
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await service.onModuleDestroy().catch(() => {});
|
||||
delete process.env.WORKER_INSTANCE_ID;
|
||||
jest.useRealTimers();
|
||||
});
|
||||
|
||||
describe('onModuleInit', () => {
|
||||
it('writes heartbeat immediately and starts interval', async () => {
|
||||
jest.useFakeTimers();
|
||||
await service.onModuleInit();
|
||||
expect(mockRedis.set).toHaveBeenCalledTimes(1);
|
||||
expect(mockRedis.set).toHaveBeenCalledWith(
|
||||
'worker:heartbeat:test-worker-1',
|
||||
expect.any(String),
|
||||
90,
|
||||
);
|
||||
|
||||
// Parse the heartbeat data
|
||||
const data = JSON.parse((mockRedis.set as jest.Mock).mock.calls[0][1]);
|
||||
expect(data.instanceId).toBe('test-worker-1');
|
||||
expect(data.queues).toBeInstanceOf(Array);
|
||||
expect(data.queues.length).toBeGreaterThanOrEqual(1);
|
||||
expect(data.hostname).toBeDefined();
|
||||
expect(data.appVersion).toBeDefined();
|
||||
expect(data.startedAt).toBeDefined();
|
||||
});
|
||||
|
||||
it('writes heartbeat periodically', async () => {
|
||||
jest.useFakeTimers();
|
||||
await service.onModuleInit();
|
||||
expect(mockRedis.set).toHaveBeenCalledTimes(1);
|
||||
|
||||
jest.advanceTimersByTime(30_000);
|
||||
await Promise.resolve();
|
||||
expect(mockRedis.set).toHaveBeenCalledTimes(2);
|
||||
|
||||
jest.advanceTimersByTime(30_000);
|
||||
await Promise.resolve();
|
||||
expect(mockRedis.set).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
it('uses instanceId from WORKER_INSTANCE_ID env var', async () => {
|
||||
process.env.WORKER_INSTANCE_ID = 'custom-instance';
|
||||
const svc = new WorkerHeartbeatService(mockRedis as any);
|
||||
await svc.onModuleInit();
|
||||
expect(mockRedis.set).toHaveBeenCalledWith(
|
||||
'worker:heartbeat:custom-instance',
|
||||
expect.any(String),
|
||||
90,
|
||||
);
|
||||
await svc.onModuleDestroy();
|
||||
});
|
||||
});
|
||||
|
||||
describe('onModuleDestroy', () => {
|
||||
it('stops interval and deletes heartbeat key', async () => {
|
||||
jest.useFakeTimers();
|
||||
await service.onModuleInit();
|
||||
expect(mockRedis.set).toHaveBeenCalledTimes(1);
|
||||
|
||||
await service.onModuleDestroy();
|
||||
|
||||
// Advance time — no more beats
|
||||
jest.advanceTimersByTime(30_000);
|
||||
await Promise.resolve();
|
||||
expect(mockRedis.set).toHaveBeenCalledTimes(1);
|
||||
|
||||
// Key deleted
|
||||
expect(mockRedis.del).toHaveBeenCalledWith('worker:heartbeat:test-worker-1');
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeStatuses', () => {
|
||||
it('returns empty array when no heartbeat keys', async () => {
|
||||
(mockRedis.keys as jest.Mock).mockResolvedValue([]);
|
||||
const result = await WorkerHeartbeatService.computeStatuses(mockRedis as any);
|
||||
expect(result).toEqual([]);
|
||||
});
|
||||
|
||||
it('returns online when TTL > 30', async () => {
|
||||
(mockRedis.keys as jest.Mock).mockResolvedValue(['worker:heartbeat:w1']);
|
||||
(mockRedis.get as jest.Mock).mockResolvedValue(JSON.stringify({
|
||||
instanceId: 'w1', appVersion: '0.0.1', startedAt: new Date().toISOString(),
|
||||
queues: ['ai-analysis'], hostname: 'host1',
|
||||
}));
|
||||
(mockRedis.ttl as jest.Mock).mockResolvedValue(60);
|
||||
|
||||
const result = await WorkerHeartbeatService.computeStatuses(mockRedis as any);
|
||||
expect(result[0].status).toBe('online');
|
||||
});
|
||||
|
||||
it('returns stale when TTL between 1 and 30', async () => {
|
||||
(mockRedis.keys as jest.Mock).mockResolvedValue(['worker:heartbeat:w1']);
|
||||
(mockRedis.get as jest.Mock).mockResolvedValue(JSON.stringify({
|
||||
instanceId: 'w1', appVersion: '0.0.1', startedAt: new Date().toISOString(),
|
||||
queues: ['ai-analysis'], hostname: 'host1',
|
||||
}));
|
||||
(mockRedis.ttl as jest.Mock).mockResolvedValue(15);
|
||||
|
||||
const result = await WorkerHeartbeatService.computeStatuses(mockRedis as any);
|
||||
expect(result[0].status).toBe('stale');
|
||||
});
|
||||
|
||||
it('returns offline when TTL < 0', async () => {
|
||||
(mockRedis.keys as jest.Mock).mockResolvedValue(['worker:heartbeat:w1']);
|
||||
(mockRedis.get as jest.Mock).mockResolvedValue(JSON.stringify({
|
||||
instanceId: 'w1', appVersion: '0.0.1', startedAt: new Date().toISOString(),
|
||||
queues: ['ai-analysis'], hostname: 'host1',
|
||||
}));
|
||||
(mockRedis.ttl as jest.Mock).mockResolvedValue(-1);
|
||||
|
||||
const result = await WorkerHeartbeatService.computeStatuses(mockRedis as any);
|
||||
expect(result[0].status).toBe('offline');
|
||||
});
|
||||
|
||||
it('handles multiple worker instances', async () => {
|
||||
(mockRedis.keys as jest.Mock).mockResolvedValue(['worker:heartbeat:w1', 'worker:heartbeat:w2']);
|
||||
(mockRedis.get as jest.Mock)
|
||||
.mockResolvedValueOnce(JSON.stringify({ instanceId: 'w1', appVersion: '0.0.1', startedAt: new Date().toISOString(), queues: ['ai-analysis'], hostname: 'host1' }))
|
||||
.mockResolvedValueOnce(JSON.stringify({ instanceId: 'w2', appVersion: '0.0.1', startedAt: new Date().toISOString(), queues: ['ai-analysis'], hostname: 'host2' }));
|
||||
(mockRedis.ttl as jest.Mock)
|
||||
.mockResolvedValueOnce(60)
|
||||
.mockResolvedValueOnce(10);
|
||||
|
||||
const result = await WorkerHeartbeatService.computeStatuses(mockRedis as any);
|
||||
expect(result).toHaveLength(2);
|
||||
expect(result[0].status).toBe('online');
|
||||
expect(result[1].status).toBe('stale');
|
||||
});
|
||||
|
||||
it('skips malformed JSON data', async () => {
|
||||
(mockRedis.keys as jest.Mock).mockResolvedValue(['worker:heartbeat:w1']);
|
||||
(mockRedis.get as jest.Mock).mockResolvedValue('not-json');
|
||||
const result = await WorkerHeartbeatService.computeStatuses(mockRedis as any);
|
||||
expect(result).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('API heartbeat key ≠ Worker heartbeat key', () => {
|
||||
it('API process does not have WorkerHeartbeatService in its DI graph', () => {
|
||||
// WorkerHeartbeatService is only registered in WorkerModule,
|
||||
// not in AppModule. This is verified by the module configuration:
|
||||
// - AppModule does not import WorkerHeartbeatService
|
||||
// - WorkerModule imports and provides WorkerHeartbeatService
|
||||
// This test documents the architectural constraint.
|
||||
expect(WorkerHeartbeatService).toBeDefined();
|
||||
});
|
||||
});
|
||||
});
|
||||
@ -1,137 +0,0 @@
|
||||
import { Injectable, OnModuleInit, OnModuleDestroy, Logger } from '@nestjs/common';
|
||||
import { RedisService } from '../infrastructure/redis/redis.service';
|
||||
import { QUEUE_DEFINITIONS } from '../infrastructure/queue/queue-definitions';
|
||||
import * as os from 'os';
|
||||
|
||||
export interface WorkerHeartbeatData {
|
||||
instanceId: string;
|
||||
appVersion: string;
|
||||
startedAt: string;
|
||||
queues: string[];
|
||||
concurrency: number;
|
||||
activeJobCount: number;
|
||||
hostname: string;
|
||||
}
|
||||
|
||||
export interface WorkerStatus {
|
||||
instanceId: string;
|
||||
appVersion: string;
|
||||
startedAt: string;
|
||||
queues: string[];
|
||||
hostname: string;
|
||||
lastHeartbeatAgo: number; // seconds since last beat
|
||||
status: 'online' | 'stale' | 'offline';
|
||||
}
|
||||
|
||||
@Injectable()
|
||||
export class WorkerHeartbeatService implements OnModuleInit, OnModuleDestroy {
|
||||
private readonly logger = new Logger(WorkerHeartbeatService.name);
|
||||
private intervalId: ReturnType<typeof setInterval> | null = null;
|
||||
private startedAt: Date;
|
||||
|
||||
private static readonly HEARTBEAT_INTERVAL_MS = 30_000;
|
||||
private static readonly HEARTBEAT_TTL_SEC = 90;
|
||||
|
||||
constructor(private readonly redis: RedisService) {}
|
||||
|
||||
async onModuleInit() {
|
||||
this.startedAt = new Date();
|
||||
await this.beat();
|
||||
this.intervalId = setInterval(() => this.beat(), WorkerHeartbeatService.HEARTBEAT_INTERVAL_MS);
|
||||
this.logger.log(`Heartbeat started — instance=${this.resolveInstanceId()}`);
|
||||
}
|
||||
|
||||
async onModuleDestroy() {
|
||||
if (this.intervalId) {
|
||||
clearInterval(this.intervalId);
|
||||
this.intervalId = null;
|
||||
}
|
||||
try {
|
||||
await this.redis.del(this.heartbeatKey());
|
||||
} catch {
|
||||
// Best-effort cleanup
|
||||
}
|
||||
this.logger.log('Heartbeat stopped');
|
||||
}
|
||||
|
||||
// ── Public static helpers for Admin API ──
|
||||
|
||||
static heartbeatKeyPrefix = 'worker:heartbeat:';
|
||||
|
||||
static heartbeatKeyFor(instanceId: string): string {
|
||||
return `${WorkerHeartbeatService.heartbeatKeyPrefix}${instanceId}`;
|
||||
}
|
||||
|
||||
static async computeStatuses(redis: RedisService): Promise<WorkerStatus[]> {
|
||||
const keys = await redis.keys(`${WorkerHeartbeatService.heartbeatKeyPrefix}*`);
|
||||
const now = Date.now();
|
||||
const statuses: WorkerStatus[] = [];
|
||||
|
||||
for (const key of keys) {
|
||||
const raw = await redis.get(key);
|
||||
if (!raw) continue;
|
||||
|
||||
let data: WorkerHeartbeatData;
|
||||
try {
|
||||
data = JSON.parse(raw);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
|
||||
const ttl = await redis.ttl(key);
|
||||
const lastHeartbeatAgo = WorkerHeartbeatService.HEARTBEAT_TTL_SEC - ttl;
|
||||
|
||||
let status: WorkerStatus['status'];
|
||||
if (ttl < 0) {
|
||||
status = 'offline';
|
||||
} else if (ttl <= 30) {
|
||||
// TTL ≤ 30s means haven't heard from worker in 60+ seconds
|
||||
status = 'stale';
|
||||
} else {
|
||||
status = 'online';
|
||||
}
|
||||
|
||||
statuses.push({
|
||||
instanceId: data.instanceId,
|
||||
appVersion: data.appVersion,
|
||||
startedAt: data.startedAt,
|
||||
queues: data.queues,
|
||||
hostname: data.hostname,
|
||||
lastHeartbeatAgo,
|
||||
status,
|
||||
});
|
||||
}
|
||||
|
||||
return statuses;
|
||||
}
|
||||
|
||||
// ── Private ──
|
||||
|
||||
private resolveInstanceId(): string {
|
||||
return process.env.WORKER_INSTANCE_ID || `worker-${os.hostname()}-${process.pid}`;
|
||||
}
|
||||
|
||||
private heartbeatKey(): string {
|
||||
return WorkerHeartbeatService.heartbeatKeyFor(this.resolveInstanceId());
|
||||
}
|
||||
|
||||
private async beat() {
|
||||
const instanceId = this.resolveInstanceId();
|
||||
const data: WorkerHeartbeatData = {
|
||||
instanceId,
|
||||
// eslint-disable-next-line @typescript-eslint/no-var-requires
|
||||
appVersion: require('../../package.json').version,
|
||||
startedAt: this.startedAt.toISOString(),
|
||||
queues: Object.keys(QUEUE_DEFINITIONS),
|
||||
concurrency: Object.values(QUEUE_DEFINITIONS)[0]?.workerOptions.concurrency ?? 1,
|
||||
activeJobCount: 0,
|
||||
hostname: os.hostname(),
|
||||
};
|
||||
|
||||
try {
|
||||
await this.redis.set(this.heartbeatKey(), JSON.stringify(data), WorkerHeartbeatService.HEARTBEAT_TTL_SEC);
|
||||
} catch (err: any) {
|
||||
this.logger.warn(`Heartbeat write failed: ${err.message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,281 +0,0 @@
|
||||
#!/bin/bash
|
||||
# M-AI-01-08 集成验收脚本
|
||||
# 前置条件:docker compose up -d(MySQL + Redis + API + Worker 全部启动)
|
||||
# 运行方式:bash test/m-ai-01-08-integration.sh
|
||||
|
||||
set -e
|
||||
|
||||
API="http://localhost:3000"
|
||||
RED="\033[0;31m"
|
||||
GREEN="\033[0;32m"
|
||||
NC="\033[0m"
|
||||
|
||||
pass() { echo -e "${GREEN}[PASS]${NC} $1"; }
|
||||
fail() { echo -e "${RED}[FAIL]${NC} $1"; exit 1; }
|
||||
|
||||
echo "============================================"
|
||||
echo " M-AI-01-08 集成验收"
|
||||
echo "============================================"
|
||||
|
||||
# ─── 1. 基础健康检查 ───
|
||||
|
||||
echo ""
|
||||
echo "── 1. 基础健康检查 ──"
|
||||
|
||||
echo -n " API health: "
|
||||
HEALTH=$(curl -s "$API/health")
|
||||
if echo "$HEALTH" | grep -q '"ok"'; then
|
||||
pass "API healthy"
|
||||
else
|
||||
fail "API not healthy: $HEALTH"
|
||||
fi
|
||||
|
||||
echo -n " Worker heartbeat: "
|
||||
sleep 2 # Wait for first heartbeat
|
||||
HB=$(curl -s "$API/admin-api/servers/workers" \
|
||||
-H "Authorization: Bearer ${ADMIN_TOKEN:-test}" 2>/dev/null || echo '{"workers":[]}')
|
||||
WORKER_COUNT=$(echo "$HB" | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d.get('workers',[])))" 2>/dev/null || echo "0")
|
||||
if [ "$WORKER_COUNT" -gt 0 ]; then
|
||||
pass "Worker heartbeat detected ($WORKER_COUNT instance(s))"
|
||||
else
|
||||
echo " (Worker heartbeat check skipped — requires ADMIN_TOKEN)"
|
||||
fi
|
||||
|
||||
# ─── 2. 进程边界验证 ───
|
||||
|
||||
echo ""
|
||||
echo "── 2. 进程边界 ──"
|
||||
|
||||
echo " Stopping Worker container..."
|
||||
docker compose stop worker
|
||||
|
||||
echo -n " API still responds: "
|
||||
HEALTH2=$(curl -s "$API/health" || echo 'fail')
|
||||
if echo "$HEALTH2" | grep -q '"ok"'; then
|
||||
pass "API responds while Worker is stopped"
|
||||
else
|
||||
fail "API not responding while Worker stopped"
|
||||
fi
|
||||
|
||||
echo " Starting Worker container..."
|
||||
docker compose start worker
|
||||
sleep 5
|
||||
|
||||
echo -n " Worker heartbeat restored: "
|
||||
HB2=$(curl -s "$API/admin-api/servers/workers" \
|
||||
-H "Authorization: Bearer ${ADMIN_TOKEN:-test}" 2>/dev/null || echo '{"workers":[]}')
|
||||
WC2=$(echo "$HB2" | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d.get('workers',[])))" 2>/dev/null || echo "0")
|
||||
if [ "$WC2" -gt 0 ]; then
|
||||
pass "Worker heartbeat restored after restart"
|
||||
else
|
||||
echo " (heartbeat check skipped — requires ADMIN_TOKEN)"
|
||||
fi
|
||||
|
||||
# ─── 3. Document Import 链路 ──
|
||||
|
||||
echo ""
|
||||
echo "── 3. Document Import 链路 ──"
|
||||
|
||||
echo -n " Creating knowledge base..."
|
||||
KB=$(curl -s -X POST "$API/api/knowledge-bases" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${USER_TOKEN:-test}" \
|
||||
-d '{"title":"Integration Test KB"}' 2>/dev/null)
|
||||
KB_ID=$(echo "$KB" | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('id',''))" 2>/dev/null || echo "")
|
||||
if [ -n "$KB_ID" ]; then
|
||||
echo " KB=$KB_ID"
|
||||
else
|
||||
echo " (skipped — requires USER_TOKEN; creating mock KB_ID=mock-kb-1)"
|
||||
KB_ID="mock-kb-1"
|
||||
fi
|
||||
|
||||
echo -n " Creating import job..."
|
||||
IMPORT=$(curl -s -X POST "$API/api/knowledge-bases/$KB_ID/sources" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${USER_TOKEN:-test}" \
|
||||
-d '{"title":"Test Source","type":"markdown","content":"# Test"}' 2>/dev/null)
|
||||
IMPORT_JOB_ID=$(echo "$IMPORT" | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('jobId',json.load(sys.stdin).get('data',{}).get('id','')))" 2>/dev/null || echo "")
|
||||
if [ -n "$IMPORT_JOB_ID" ]; then
|
||||
echo " DocumentImport Job ID: $IMPORT_JOB_ID"
|
||||
pass "Import job created — traceable Job ID"
|
||||
else
|
||||
echo " (skipped — requires USER_TOKEN and real service)"
|
||||
fi
|
||||
|
||||
# ─── 4. Notification 链路 ──
|
||||
|
||||
echo ""
|
||||
echo "── 4. Notification 链路 ──"
|
||||
|
||||
echo -n " Triggering notification..."
|
||||
NOTIF=$(curl -s -X POST "$API/api/notifications/test" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${USER_TOKEN:-test}" \
|
||||
-d '{"type":"ai_job_succeeded","title":"Integration Test","content":"Verifying notification chain"}' 2>/dev/null)
|
||||
NOTIF_STATUS=$(echo "$NOTIF" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('success', False))" 2>/dev/null || echo "")
|
||||
if [ "$NOTIF_STATUS" = "True" ]; then
|
||||
pass "Notification enqueued"
|
||||
else
|
||||
echo " (skipped — requires USER_TOKEN)"
|
||||
fi
|
||||
|
||||
echo " Worker processes notification queue:"
|
||||
NOTIF_LOG=$(docker compose logs worker 2>/dev/null | grep -i "notification" | tail -3 || echo "")
|
||||
if [ -n "$NOTIF_LOG" ]; then
|
||||
echo "$NOTIF_LOG"
|
||||
pass "NotificationWorker log entries found"
|
||||
else
|
||||
echo " (no notification log entries yet — Worker may not have processed the job)"
|
||||
fi
|
||||
|
||||
echo " Verify notification state in DB:"
|
||||
echo " docker compose exec mysql mysql -u zhixi_user -p\$MYSQL_PASSWORD zhixi -e \""
|
||||
echo " SELECT id, userId, type, title, readAt, createdAt FROM Notification WHERE type='ai_job_succeeded' ORDER BY createdAt DESC LIMIT 3;\" "
|
||||
|
||||
# ─── 5. Active Recall 全链路 ──
|
||||
|
||||
echo ""
|
||||
echo "── 5. Active Recall 全链路 ──"
|
||||
|
||||
echo -n " Creating learning session..."
|
||||
SESSION=$(curl -s -X POST "$API/api/learning-sessions" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${USER_TOKEN:-test}" \
|
||||
-d '{"knowledgeBaseId":"'"$KB_ID"'","mode":"active_recall"}' 2>/dev/null)
|
||||
SESSION_ID=$(echo "$SESSION" | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('id',''))" 2>/dev/null || echo "")
|
||||
if [ -n "$SESSION_ID" ]; then
|
||||
echo " Session ID: $SESSION_ID"
|
||||
else
|
||||
echo " (skipped — requires USER_TOKEN)"
|
||||
fi
|
||||
|
||||
echo -n " Triggering AI analysis..."
|
||||
ANALYSIS=$(curl -s -X POST "$API/api/ai-analysis" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer ${USER_TOKEN:-test}" \
|
||||
-d '{"sessionId":"'"$SESSION_ID"'","type":"active_recall"}' 2>/dev/null)
|
||||
ANALYSIS_JOB_ID=$(echo "$ANALYSIS" | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('jobId',''))" 2>/dev/null || echo "")
|
||||
if [ -n "$ANALYSIS_JOB_ID" ]; then
|
||||
echo " AiAnalysisJob ID: $ANALYSIS_JOB_ID"
|
||||
pass "AI analysis job created — traceable Job ID"
|
||||
else
|
||||
echo " (skipped — requires USER_TOKEN)"
|
||||
fi
|
||||
|
||||
# ─── 5. Worker 日志验证 ──
|
||||
|
||||
echo ""
|
||||
echo "── 6. Worker 日志验证 ──"
|
||||
|
||||
echo " Worker startup log:"
|
||||
docker compose logs worker 2>/dev/null | grep "Worker version" | tail -1 || echo " (no logs available)"
|
||||
|
||||
echo ""
|
||||
echo " Worker heartbeat log:"
|
||||
docker compose logs worker 2>/dev/null | grep "Heartbeat" | tail -3 || echo " (no logs available)"
|
||||
|
||||
echo ""
|
||||
echo " Worker job processing log:"
|
||||
docker compose logs worker 2>/dev/null | grep -E "Job.*added|completed|failed" | tail -5 || echo " (no jobs processed yet)"
|
||||
|
||||
# ─── 6. ID 可追溯性验证 ──
|
||||
|
||||
echo ""
|
||||
echo "── 7. ID 可追溯性验证 ──"
|
||||
|
||||
echo " Query chain via DB (requires mysql client or docker exec):"
|
||||
echo " docker compose exec mysql mysql -u zhixi_user -p\$MYSQL_PASSWORD zhixi -e \""
|
||||
echo " SELECT id, userId, jobType, targetType, targetId, status, createdAt"
|
||||
echo " FROM AiAnalysisJob WHERE id='$ANALYSIS_JOB_ID';\" "
|
||||
echo ""
|
||||
echo " docker compose exec mysql mysql -u zhixi_user -p\$MYSQL_PASSWORD zhixi -e \""
|
||||
echo " SELECT id, jobId, userId, status, createdAt"
|
||||
echo " FROM AiAnalysisResult WHERE jobId='$ANALYSIS_JOB_ID';\" "
|
||||
echo ""
|
||||
echo " docker compose exec mysql mysql -u zhixi_user -p\$MYSQL_PASSWORD zhixi -e \""
|
||||
echo " SELECT id, userId, analysisJobId, title, status, createdAt"
|
||||
echo " FROM FocusItem WHERE analysisJobId='$ANALYSIS_JOB_ID';\" "
|
||||
echo ""
|
||||
echo " docker compose exec mysql mysql -u zhixi_user -p\$MYSQL_PASSWORD zhixi -e \""
|
||||
echo " SELECT id, userId, reviewItemId, front, back, status, createdAt"
|
||||
echo " FROM ReviewCard WHERE reviewItemId IN (SELECT id FROM FocusItem WHERE analysisJobId='$ANALYSIS_JOB_ID');\" "
|
||||
|
||||
# ─── 7. 失败场景 ──
|
||||
|
||||
echo ""
|
||||
echo "── 8. 失败场景 ──"
|
||||
|
||||
echo ""
|
||||
echo " 7a. Redis 不可用时 API 仍接受请求:"
|
||||
echo -n " Stopping Redis..."
|
||||
docker compose stop redis 2>/dev/null || true
|
||||
sleep 3
|
||||
REDIS_DOWN_CHECK=$(curl -s -o /dev/null -w "%{http_code}" "$API/health" 2>/dev/null || echo "000")
|
||||
if [ "$REDIS_DOWN_CHECK" = "200" ]; then
|
||||
pass "API /health returns 200 while Redis is down"
|
||||
else
|
||||
echo " (API returned $REDIS_DOWN_CHECK — may fail if health check requires Redis)"
|
||||
fi
|
||||
echo " Starting Redis..."
|
||||
docker compose start redis 2>/dev/null || true
|
||||
sleep 2
|
||||
|
||||
echo ""
|
||||
echo " 7b. Worker SIGTERM 优雅关闭:"
|
||||
echo -n " Sending SIGTERM to Worker..."
|
||||
docker compose exec -T worker kill -TERM 1 2>/dev/null && sleep 3 || echo " (container not running)"
|
||||
SHUTDOWN_LOG=$(docker compose logs worker 2>/dev/null | grep -c "shut down" || echo "0")
|
||||
if [ "$SHUTDOWN_LOG" -gt 0 ]; then
|
||||
pass "Worker logged graceful shutdown"
|
||||
else
|
||||
echo " (Worker shutdown log not found — may have already stopped)"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo " 7c. 强制终止恢复:"
|
||||
echo -n " Killing Worker..."
|
||||
docker compose kill worker 2>/dev/null || true
|
||||
sleep 2
|
||||
echo -n " Starting Worker..."
|
||||
docker compose start worker 2>/dev/null || true
|
||||
sleep 5
|
||||
RECOVERY_CHECK=$(curl -s -o /dev/null -w "%{http_code}" "$API/health" 2>/dev/null || echo "000")
|
||||
if [ "$RECOVERY_CHECK" = "200" ]; then
|
||||
pass "API still responding after Worker kill+start cycle"
|
||||
else
|
||||
echo " (API not responding — may need manual investigation)"
|
||||
fi
|
||||
echo " Stalled jobs should be re-processed after lockUntil expiry (~30s)"
|
||||
|
||||
echo ""
|
||||
echo " 7d. 重复 jobId 幂等性:"
|
||||
echo " BullMQ uses jobId for dedup. Verify by checking Redis:"
|
||||
echo " docker compose exec redis redis-cli GET \"bull:ai-analysis:id\" # check no duplicate entries"
|
||||
echo " (Automated dedup verified at BullMQ level — Queue.add with same jobId is idempotent)"
|
||||
|
||||
echo ""
|
||||
echo " 7e. attempts/backoff 验证:"
|
||||
BACKOFF_LOG=$(docker compose logs worker 2>/dev/null | grep -E "retry|backoff|attempt" | tail -5 || echo "")
|
||||
if [ -n "$BACKOFF_LOG" ]; then
|
||||
echo "$BACKOFF_LOG"
|
||||
pass "Attempts/backoff log entries found"
|
||||
else
|
||||
echo " (No retry/backoff events in logs — all jobs succeeded or no jobs ran)"
|
||||
fi
|
||||
|
||||
# ─── 9. 汇总 ──
|
||||
|
||||
echo ""
|
||||
echo "============================================"
|
||||
echo " 验收完成"
|
||||
echo "============================================"
|
||||
echo ""
|
||||
echo "将以上输出粘贴到 Issue #252 评论中。"
|
||||
echo ""
|
||||
echo "手动补充验证项:"
|
||||
echo " [ ] Worker 启动后 Admin /workers 显示 online"
|
||||
echo " [ ] Worker 停止后 90s 内 /workers 转为 offline"
|
||||
echo " [ ] API 重启期间 Worker 继续处理 job"
|
||||
echo " [ ] Worker 重启期间 API 继续响应请求"
|
||||
echo " [ ] 所有 Job/AnalysisResult/FocusItem/ReviewCard ID 通过 MySQL 查询可追溯"
|
||||
echo " [ ] attempts/backoff 在 provider 超时时正确递增"
|
||||
Loading…
x
Reference in New Issue
Block a user