Author SHA1 Message Date
admin 9b746db452 Merge feature/admin-console: 管理后台 + 用户端鉴权(登录/授权/数据隔离/日志) 2026-10-16 16:01:13 +08:00
admin d9abd0b86a feat(auth): 用户端鉴权(登录/授权清单/逐次校验/功能门禁 + owner 数据隔离) 2026-10-16 14:42:23 +08:00
admin 282415fcb1 feat(admin): 行为日志记录 hub_id(便于区分不同中枢) 2026-10-15 23:56:42 +08:00
admin 1ca2241a97 feat(admin): 增加中枢设置(自定义 ID 与名称,右上角设置入口) 2026-10-15 23:52:17 +08:00
admin 8b5095f70f Merge pull request 'feat(review): P5 科研汇报助手(论文 → PPT 大纲 → PPTX)' (#5) from wjq/PlantAI-rag-new:feature/p5-ppt-assistant into main
Reviewed-on: http://8.163.92.8:3000/admin/PlantAI-rag-new/pulls/5
2026-10-15 20:19:30 +08:00
potaser f5a951f47a docs: add P5 PR report 2026-10-15 19:35:51 +08:00
potaser 7ced30fa96 feat(review): P5 科研汇报助手 + 14 风格 + 真图嵌入 2026-10-15 18:51:59 +08:00
admin c78df3a6a6 feat(admin): 新增管理后台(登录/用户管理/日志查看) + 行为日志文件采集(请求中间件 + log_event) 2026-10-15 18:43:19 +08:00
admin 69e3f35a43 Merge pull request 'feat(review): P3b+P4 论文重排版支持编译 PDF(含 zip 项目)' (#4) from wjq/PlantAI-rag-new:feature/p3b-compile-ui into main
Reviewed-on: http://8.163.92.8:3000/admin/PlantAI-rag-new/pulls/4
2026-10-15 18:30:14 +08:00
potaser de0b9f92be fix(review): EPS batch conversion in zip compile (v2)
- v1 fell through because it required pre-converted PDF in zip
- v2: convert all .eps files post-extract via gs -dSAFER
- 15s timeout per file; fail per-file, not blocking
- Size guards: 20MB per file warning, 100MB total abort
- Conversion log written to output.log
- Verified on real MDPI official zip → 7-page PDF

v1 lesson: test sample had pre-converted PDF but real zip does not
2026-10-14 09:47:49 +08:00
potaser 2797145104 feat(review): P4 two-phase zip compile + XDV→PDF diagnostics + EPS fallback
- inspect/upload zip then compile project (main-file detection, zip safety,
  single-slot queue, failed-envelope carries compile_id + log_url)
- keep failed project/ scene 24h (failed_project_ttl_s; KEEP_FAILED_PROJECTS=1
  forever), success deletes immediately; two-tier compile_jobs sweep
- add tectonic --print and persist full engine chatter to output.log so
  XDV→PDF (xdvipdfmx) errors are visible (MDPI: logo EPS inclusion failure)
- EPS fallback: swap X.eps refs for *-eps-converted-to.pdf when present and
  materialize X.pdf for extension-less graphicx lookup; friendly error when
  no converted PDF exists
- frontend: ReformatPanel zip upload/inspect/run flow; one-off
  scripts/debug_compile.py (gitignored)
- tests: backend 169 passed (22 zip_safety + 34 compile_zip + P3a regression),
  frontend edu vitest 43 passed
2026-10-13 18:42:04 +08:00
potaser 4e19350115 docs: add P3b PR report 2026-10-13 11:23:01 +08:00
potaser e036301e1a feat(review): show format rules summary per target journal
- New summary row: shows the journal's rules (even if no diff applies)
- Fixes user confusion when switching journals yields same diff
- When all rules are null: 'no explicit format requirements'
- Vitest: add case for summary rendering
2026-10-13 11:22:52 +08:00
potaser da84dff9bb fix(review): persist uploaded .tex file in reformat panel
- File now stored in edu.js store (reformatFile)
- Switching target journal no longer requires re-upload
- Explicit reset button clears file
- Vitest: 4 cases
2026-10-13 11:20:33 +08:00
potaser 3a72d3a861 feat(review): P3b enable compile PDF in reformat panel
- ReformatPanel: compile tab state machine (idle/loading/success/error)
- Blob URL for PDF preview; backend path hidden
- First-compile 1-3min hint banner (dismissible)
- QUEUE_FULL 429 with fixed message
- Store: compileStatus/compileResult/compileError + actions
- prewarm_tectonic.sh: minimal .tex compilation to pre-cache TeX bundles
- Vitest: 6 new cases (guide / button / loading / success / error / queue-full)
2026-10-13 10:24:55 +08:00
admin ac07cb2008 Merge pull request 'feat(review): AI 预审稿修复 + 审稿回复助手 Phase 1' (#3) from wjq/PlantAI-rag-new:feature/presubmit-explainable-engine into main
Reviewed-on: http://192.168.0.175:3000/admin/PlantAI-rag-new/pulls/3
2026-10-13 09:34:41 +08:00
potaser d077a5b930 test(docs): rebuttal + tracker tests + PR reports 2026-10-12 16:38:28 +08:00
potaser c4dc344a31 feat(edu): presubmit UI + rebuttal assistant + tracker removal
- PresubmitPanel: rationale / conflict chips / file persistence
- RebuttalPanel: Phase 1 UI (3-column, 5-type form, stance pre-check)
- ReformatPanel / ReportPanel: hide internal fields (job id / version)
- EduSideNav: SVG icon, remove tracker menu item
- EduWorkspace: unmount tracker
- edu.js: rebuttal state + presubmitFile + SECTIONS (drop tracker)
2026-10-12 16:38:28 +08:00
potaser cf5ccb3a37 feat(review): presubmit 4-bug fix + rebuttal Phase 1 backend
- 4 bugs: truncation window / keyword scanner / tier heuristic / vocab
- Scanner conflict demotion now truly affects recommendation (major → minor)
- Rebuttal Phase 1: store / parse / draft / 10 endpoints
- max_text_chars 150K + token guard + 400/413 retry
- Journals tier override (13 journals) + jif heuristic (exclude reviews)
2026-10-12 16:38:28 +08:00
potaser d99d5ef74a fix(review): replace presubmit recommendation quota with explainable rule engine
- Remove '3-8 major' quota that made accept/minor_revision unreachable
- Add recommend_engine.py: rule-based R0-R5 + tier-parameterized judgment
- Journal tier (1/2/3) from journals.json review_tier field
- LLM no longer outputs recommendation; engine is single source of truth
- Rationale exposed in API and rendered in PresubmitPanel
- Weak draft still major_revision (R1); strong draft now minor_revision (R5)
2026-10-12 09:55:07 +08:00
admin 4add9f176b Merge pull request 'feat(review): journal_directory 启动自动 seed(部署即运行)' (#2) from wjq/PlantAI-rag-new:feature/journal-seed-autoload into main
Reviewed-on: http://8.163.92.8:3000/admin/PlantAI-rag-new/pulls/2
2026-10-11 21:12:56 +08:00
potaser 1319328b85 feat(review): auto-seed journal_directory on startup when empty 2026-10-11 20:55:36 +08:00
admin 2d0e3a4858 fix(build): compile_sdk 白名单加入 review_proxy.py(科研审稿反代断流高危点) 2026-10-11 16:40:24 +08:00
admin ea0f2b3fad Merge pull request 'feat(review): 教育科研工作台 + 论文自动重排版 + journals v4 数据集' (#1) from wjq/PlantAI-rag-new:feature/academic-review-assistant into main
Reviewed-on: http://8.163.92.8:3000/admin/PlantAI-rag-new/pulls/1
Reviewed-by: admin <jerryhaozhang@gmail.com>
2026-10-11 13:49:12 +08:00
admin 19fa12eb39 merge main: 解决 .gitignore 冲突(合并部署文档忽略规则) 2026-10-11 13:47:47 +08:00
admin 11337a788f 更新 .gitignore
解决冲突

Signed-off-by: admin <jerryhaozhang@gmail.com>
2026-10-11 13:29:35 +08:00
potaser 55410b663a feat(review): journal format-rule fetch scripts (offline CLI)
- fetch_format_rules.py CLI: robots.txt-aware fetcher (httpx + bs4,
  >=2s per-host throttle) + DeepSeek quote-backed rule extractor
- Status: OUP/Wiley/Cell return 403 from this environment; the five
  featured journals' format_rules in journals.json were manually
  verified against official sites instead. Tooling kept for future
  re-runs; data/format_cache/ is runtime output and gitignored
- Service code never imports this toolchain (standalone CLI only)
2026-10-11 12:19:18 +08:00
potaser dddb70ff6f fix(review): directory partition source shown as constant note
Stop sending partition_source_url; respond with a fixed explanatory
note instead. DB values untouched (dropped after SELECT).
2026-10-11 12:19:18 +08:00
potaser 649e66f41d feat(review): tectonic PDF compile API (P3a) + rename journals panel
- POST/GET /api/review/compile[/{id}[/pdf|/log]]: isolated workdir,
  --untrusted, 60s timeout with process-group kill, single-slot queue
  (429 QUEUE_FULL), 7-day retention sweep
- tools/install_tectonic.sh: pinned tectonic 0.17.0 (sha256) + README
- Rename menu/panel title to 学术期刊情报
- Compile UI stays a placeholder per product decision (P3 deferred)
2026-10-11 12:19:18 +08:00
potaser 42aeb8fa4c feat(review): paper auto-reformat P1 (.tex rule engine + diff panel)
- Rule engine: font/margin/line-spacing/bib-style transformers,
  tighten-only + structural guard + idempotency
- POST/GET /api/review/reformat with 200-envelope errors and job store
- ReformatPanel: journal dropdown (featured + directory search),
  source/diff tabs; compile tab is a dev placeholder for now
- journals.json format_rules_default kept as generic fallback
2026-10-11 12:19:17 +08:00
potaser c747ebd77a feat(review): official format rules for 5 featured journals
- Replace featured list with accessible journals: Genome Biology,
  BMC Plant Biology, Molecular Plant, Frontiers in Plant Science, Rice
- Per-journal format_rules verified against official author guidelines
  (rules_source=official, source_url, last_verified=2026-10-10)
- Update recommend tests to the new seed set
2026-10-11 12:19:17 +08:00
admin 60375b04d7 chore: .gitignore 忽略含凭据的部署方案文档 DEPLOY_PAPER_176.md 2026-10-10 18:53:25 +08:00
admin 1bd4aea5fd fix(build): render_active_library_env 输出 GENE_FAMILY_ROOT 而非未定义的 QA_ROOT
active_library.env 里 $QA_ROOT 从未定义,QA start.sh 在 set -u 下 source 时报
unbound variable 导致 8091 起不来。改用 prod.env 中已定义的 GENE_FAMILY_ROOT。
2026-10-10 18:52:15 +08:00
admin 51532a30bc fix(ui): 工作台「提取任务」tab 与离线提示改为「上传文献」 2026-10-10 18:27:06 +08:00
potaser 5a71e29cf7 feat(edu): complete 7-panel academic assistant + journals v4 dataset
Backend:
- review_service: presubmit / recommend / journals / journals/directory /
  format-check / conferences endpoints
- journals.json 5 精选种子 + journal_directory 1056 条全量(v4 数据)
- v4 数据集:2025 JIF(2026 JCR 发布)+ 2025 中科院分区 + 1053 条官网 URL
- format_check 行距 bug 修复(基线 y + 物理过滤 + docx EMU)

BFF:
- review_proxy 反代(修复 httpx build_request await bug)
- REVIEW_ENABLED 开关(默认关闭,零痕部署)

Frontend:
- 教育科研工作台(7 面板 + 菜单 + 状态管理)
- TopBar 教育科研入口(v-show 覆盖层语义,RAG 会话不丢)
- 期刊情报:Tab 切换 / 搜索 / 筛选 / 分页 / 官网列 / 展开凭证
- 智能推荐 / AI预审稿 / 专业报告 / 投稿台账 / 排版规范 / 会议日历
2026-10-10 16:05:08 +08:00
potaser d43d30ee3f feat: add review_service with plant genomics presubmit, journals endpoint, and BFF proxy 2026-10-09 16:54:03 +08:00
admin d37e0d9c3a fix(paper): 确认大纲后即时 seed 进度卡并逐节刷新
- 后端写稿循环不再落空 paper_progress 块(重载后出现空卡),改落开始撰写文本
- 前端 confirmPaper 时用大纲节标题本地 seed 进度卡,2s 轮询 patch 逐节状态
- 当前节按 current_section_id 标 writing(…动画)
2026-10-08 17:10:50 +08:00
admin c923419d2d fix(paper): 大纲阶段修改意见路由回 sendPaper + 用原始 brief 重新生成大纲 2026-10-08 16:23:28 +08:00
admin 43a7c6c132 feat(paper): UI 优化(卡片文字颜色/去模式按钮) + 模板下拉加论文选项并接线渲染 2026-10-08 15:17:44 +08:00
admin 8b228923bd fix(paper): 取消路径(cancel_paper) + 引用元数据清洗(丢弃杜撰 DOI/PMID) 2026-10-08 14:59:16 +08:00
admin 2e5372cf90 fix(paper): 论文产物渲染为 HTML + 文末追加参考文献列表/disclaimer 2026-10-08 14:54:00 +08:00
admin 91f6a1c242 feat: 新增论文撰写功能 paper-writer(独立 MCP 服务 + BFF 大脑 + 前端三卡)
- paper_writer_service(8097):FastAPI + MCP streamable-http 单端点,9 工具
  (propose_paper_outline/write_paper_section/get_paper_progress/search_refs/
  revise_paper_section/revise_paper/get_paper/complete_paper/cancel_paper),
  SQLite 三表(papers/paper_sections/citations,WAL),GB/T 7714 引用协议
- BFF:services/paper/{mcp_client,agent}.py(裸 JSON-RPC 客户端 + 规则状态机)、
  routers/paper.py、schemas.py 4 个 Paper*Block + ArtifactBlock 加 paper_id/citation_count、
  config.py 开关、main.py 条件注册、docgen.py 抽 insert_artifact 公共 helper
- 前端:intent.js paperIntent 纯函数、Composer docgen 分支 sub-route、sendPaper/轮询、
  PaperBriefCard/PaperOutlineCard/PaperProgressCard、DocPane cite-chip
- deploy/paper_restart.sh + toggle_paper.sh;compile_sdk.sh 白名单追加 paper 文件
- tests/paper 26 条(工具逻辑 + MCP 协议 + BFF 状态机 + 块 schema + 前端 intent 单测)
2026-09-30 22:29:22 +08:00
283 changed files with 45097 additions and 7835 deletions
+33 -3
View File
@@ -1,4 +1,4 @@
# 运行时产物与环境
# 运行时产物与环境
node_modules
dist/
.venv/
@@ -37,5 +37,35 @@ tests/extension_cases_18/
bioflow_service/bioflow.db
*.before-*
# Python 覆盖率运行时产物(coverage.py --cov 生成)
.coverage
# paper_writer_service 运行时产物(不入库)
paper_writer_service/paper.db
paper_writer_service/paper.db-shm
paper_writer_service/paper.db-wal
paper_writer_service/paper.log
# review_service 运行时产物(不入库)
review_service/review.db
review_service/review.db-wal
review_service/review.db-shm
review_service/review.log
review_service/data/reformat_jobs/
# 期刊种子 journals.json 是源码:顶层 **/data/ 会误伤 review_service/data/,显式放行
!review_service/data/
review_service/tools/tectonic
review_service/data/tectonic-cache/
review_service/data/format_cache/
review_service/data/compile_jobs/
review_service/data/compile_inspect/
review_service/prewarm.log
# 部署方案文档(含主机凭据,不入库)
docs/DEPLOY_PAPER_176.md
# 一次性诊断脚本(不入库)
scripts/debug_compile.py
# P5 e2e 临时产物(compile_jobs/ 与 prewarm.log 已在上方「review_service 运行时产物」段,此处不重复)
# P5 ppt job 运行数据(大纲/pptx/真图/预览,不入库)
review_service/data/ppt_jobs/
review_service/data/review.db
+54
View File
@@ -0,0 +1,54 @@
# AGENTS.md —— 协作约束(科研审稿模块)
## 项目背景
本项目(PlantAI-rag-new)是植物基因组学文献知识工作台,拥有自研 RAG 与知识库
(Qdrant + Neo4j + vstore)。2026-10 起新增「科研投稿与智能审稿助手」功能
(审稿周期/期刊分析/期刊推荐/投稿前自审/审稿意见辅助/格式检查)。
## 审稿模块(review_service)约束 —— 必须遵守
1. **所有审稿相关新代码只允许写在 `review_service/` 目录下新增**。
- 服务自身:FastAPI 独立进程,端口 8100(模式同 `paper_writer_service` 8097 / `bioflow_service` 8098)。
- 环境变量前缀统一 `REVIEW_`;LLM 凭据复用 `DOCGEN_LLM_*` 裸键(回退链见 `review_service/app/services/llm.py`)。
2. **BFF(`sdk/`)层只允许条件注册代理路由**:
- 允许的全部改动 = `sdk/app/config.py` 增加 `review_enabled` / `review_service_url` 等开关键
+ `sdk/app/main.py` 按开关 `include_router(review_proxy.router)`
+ 新文件 `sdk/app/routers/review_proxy.py`(反代实现)。
- **严禁**修改 sdk 下其他 router/service/db 的任何既有逻辑。
3. **严禁修改现有 RAG 核心与既有服务代码**,包括但不限于:
- `agent/`(qa 8091 / methodology 8092 / legacy-rag 8093)
- `vstore/`(8094,审稿服务只经 HTTP 只读消费其 `/api/vstore/*` 接口)
- `paper_writer_service/`、`bioflow_service/`、`bioflow_kernel/`
- `ui/`(前端入口另行按 UI_SPEC 流程评审)
- `knowledge/`、`kb-libraries/`、`tooling/`(依赖锁另议)
4. **依赖纪律**:审稿服务只使用统一 venv(`tooling/requirements.lock`)中已有的库
(fastapi / httpx / pydantic-settings 等)。如确需新增全局依赖,
须先更新 `tooling/requirements.lock` 并经人工确认。
5. **上游对接只读**:PDF 解析走 vstore 8094(`POST /api/vstore/ingest` +
`GET/DELETE /api/vstore/documents/{id}`);LLM 走 DeepSeek OpenAI 兼容接口;
后续期刊数据/期刊推荐等子功能同样以新 service + 新路由在 `review_service/` 内扩展。
6. **format_check 例外**:format_check 允许用 pymupdf 本地解档获取版式元数据
(字体/边距/行距/页数),因为 vstore 只回文本不返回版式信息。正文提取仍走 vstore。
## 目录速览(审稿模块)
```
review_service/
├── app/main.py # FastAPI 入口(/health + 挂载业务路由)
├── app/config.py # pydantic-settings(REVIEW_ 前缀)
├── app/routers/presubmit.py # POST /api/review/presubmit(自审稿报告)
├── app/routers/format_check.py # POST /api/review/format-check(版式与规范检查)
├── app/services/llm.py # DeepSeek httpx 封装(complete_text/complete_json)
├── app/services/paper_parse.py # PDF → 纯文本(经 vstore 解析接口)
├── app/services/format_check.py# 版式规则引擎(pymupdf/docx 本地解档 + 规则检查)
├── restart.sh # 8100 重启脚本
└── pytest.ini
```
## 启动
```bash
cd review_service && bash restart.sh # 默认 8100,venv 从 ../plantai-rag-new-venv 推导
# BFF 开关:sdk/.env 加 REVIEW_ENABLED=1 后 bff_restart(反代 /api/review/*)
```
+12
View File
@@ -0,0 +1,12 @@
<!DOCTYPE html>
<html lang="zh-CN" data-theme="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<title>蜂群中枢 · PlantAI 管理后台</title>
</head>
<body>
<div id="app"></div>
<script type="module" src="/src/main.js"></script>
</body>
</html>
+1147
View File
File diff suppressed because it is too large Load Diff
+18
View File
@@ -0,0 +1,18 @@
{
"name": "plantai-admin",
"version": "0.1.0",
"private": true,
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview"
},
"dependencies": {
"vue": "3.5.13"
},
"devDependencies": {
"vite": "5.4.11",
"@vitejs/plugin-vue": "5.2.1"
}
}
+143
View File
@@ -0,0 +1,143 @@
<script setup>
import { ref, onMounted } from 'vue';
import { getToken, setToken, api } from './api/admin.js';
import Login from './views/Login.vue';
import Users from './views/Users.vue';
import Logs from './views/Logs.vue';
import HubSettings from './components/HubSettings.vue';
const authed = ref(false);
const page = ref('overview');
const user = ref(null);
const stats = ref({ users: 0, total: 0, denied: 0, p95: null });
const hub = ref({ hub_id: '', hub_name: '蜂群中枢' });
const settingsOpen = ref(false);
const PAGES = { overview: '概览', users: '用户管理', logs: '日志查看' };
function onUnauthorized() { setToken(''); authed.value = false; }
async function loadHub() {
try {
const r = await api.settings();
hub.value = r.settings;
} catch (e) {
if (e.status === 401) onUnauthorized();
}
}
async function loadStats() {
try {
const [u, l] = await Promise.all([api.users(), api.logs({ page: 1, page_size: 1 })]);
stats.value.users = u.users.length;
stats.value.total = l.total;
stats.value.denied = (l.aggregates && l.aggregates.denied) || 0;
stats.value.p95 = (l.aggregates && l.aggregates.p95_latency_ms) || null;
} catch (e) {
if (e.status === 401) onUnauthorized();
}
}
async function init() {
if (!getToken()) { authed.value = false; return; }
try {
const r = await api.me();
user.value = r.user;
authed.value = true;
loadHub();
loadStats();
} catch (e) {
setToken('');
authed.value = false;
}
}
function onLogin(u) { user.value = u; authed.value = true; page.value = 'overview'; loadHub(); loadStats(); }
async function onLogout() { try { await api.logout(); } catch {} setToken(''); authed.value = false; }
function onSettingsSaved(s) { hub.value = s; settingsOpen.value = false; }
onMounted(init);
</script>
<template>
<Login v-if="!authed" @login="onLogin" />
<div v-else class="shell">
<aside class="sidebar">
<div class="side-brand">
<div class="hex">蜂</div>
<div><div class="brand-name">{{ hub.hub_name }}</div><div class="brand-sub">{{ hub.hub_id.slice(0, 8) }}</div></div>
</div>
<nav class="nav">
<div class="nav-item" :class="{ active: page === 'overview' }" @click="page = 'overview'"><span>◈</span> 概览</div>
<div class="nav-item" :class="{ active: page === 'users' }" @click="page = 'users'"><span>◉</span> 用户管理</div>
<div class="nav-item" :class="{ active: page === 'logs' }" @click="page = 'logs'"><span>≣</span> 日志查看</div>
</nav>
<div class="side-foot">
<div class="who">
<div class="avatar">蜂</div>
<div><div class="who-name">{{ user?.username }}</div><div class="who-role">系统管理员</div></div>
<span class="logout" @click="onLogout">退出</span>
</div>
</div>
</aside>
<main class="main">
<div class="topbar">
<h1>{{ PAGES[page] }}</h1>
<span style="color: var(--text-3); font-size: 13px">{{ hub.hub_name }}</span>
<button class="btn btn-ghost" style="margin-left: auto" @click="settingsOpen = true">⚙ 设置</button>
</div>
<div class="content">
<section v-if="page === 'overview'">
<div class="stats">
<div class="stat"><div class="k">{{ stats.users }}</div><div class="l">注册用户</div></div>
<div class="stat"><div class="k amber">{{ stats.total }}</div><div class="l">累计请求</div></div>
<div class="stat"><div class="k">{{ stats.denied }}</div><div class="l">越权 / 失败</div></div>
<div class="stat"><div class="k">{{ stats.p95 ?? '—' }}</div><div class="l">P95 耗时 ms</div></div>
</div>
</section>
<Users v-else-if="page === 'users'" @unauthorized="onUnauthorized" />
<Logs v-else @unauthorized="onUnauthorized" />
</div>
</main>
<HubSettings :open="settingsOpen" :hub="hub" @close="settingsOpen = false" @saved="onSettingsSaved" />
</div>
</template>
<style>
.shell { display: flex; height: 100vh; }
.sidebar { width: 218px; flex: none; background: var(--bg-2); border-right: 1px solid var(--border); display: flex; flex-direction: column; }
.side-brand { display: flex; align-items: center; gap: 10px; padding: 20px 18px 18px; }
.hex { width: 30px; height: 30px; flex: none; background: linear-gradient(135deg, #f2b95a, #d98f22); clip-path: polygon(50% 0, 100% 25%, 100% 75%, 50% 100%, 0 75%, 0 25%); display: grid; place-items: center; color: #1a1206; font-weight: 700; font-size: 12px; }
.brand-name { font-size: 15px; font-weight: 600; }
.brand-sub { font-size: 11px; color: var(--text-3); }
.nav { padding: 6px 10px; flex: 1; }
.nav-item { display: flex; align-items: center; gap: 9px; padding: 9px 12px; margin-bottom: 3px; border-radius: var(--radius-sm); color: var(--text-2); font-size: 13.5px; cursor: pointer; }
.nav-item:hover { background: var(--surface); color: var(--text); }
.nav-item.active { background: var(--accent-soft); color: var(--accent); }
.side-foot { padding: 14px 18px; border-top: 1px solid var(--border); }
.who { display: flex; align-items: center; gap: 10px; }
.avatar { width: 30px; height: 30px; border-radius: 50%; flex: none; background: linear-gradient(135deg, #3a4a5e, #243146); display: grid; place-items: center; font-size: 12px; color: var(--text); }
.who-name { font-size: 13px; }
.who-role { font-size: 11px; color: var(--text-3); }
.logout { margin-left: auto; color: var(--text-3); cursor: pointer; font-size: 12px; }
.logout:hover { color: var(--red); }
.main { flex: 1; overflow-y: auto; display: flex; flex-direction: column; }
.topbar { display: flex; align-items: center; gap: 14px; padding: 18px 28px; border-bottom: 1px solid var(--border); position: sticky; top: 0; background: rgba(12, 14, 20, 0.9); backdrop-filter: blur(6px); z-index: 5; }
.topbar h1 { font-size: 17px; margin: 0; font-weight: 600; }
.content { padding: 26px 28px 40px; }
.stats { display: grid; grid-template-columns: repeat(4, 1fr); gap: 16px; }
.stat { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius); padding: 18px 20px; }
.stat .k { font-size: 26px; font-weight: 600; letter-spacing: -0.5px; }
.stat .k.amber { color: var(--accent); }
.stat .l { color: var(--text-2); font-size: 12.5px; margin-top: 3px; }
.toolbar { display: flex; align-items: center; gap: 10px; margin-bottom: 16px; }
.toolbar .spacer { flex: 1; }
.search { display: flex; align-items: center; gap: 8px; background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius-sm); padding: 8px 12px; width: 240px; }
.search input { background: none; border: none; outline: none; color: var(--text); font-size: 13px; width: 100%; }
select { background: var(--surface); border: 1px solid var(--border); color: var(--text); border-radius: var(--radius-sm); padding: 8px 10px; font-size: 13px; outline: none; }
.pag { display: flex; align-items: center; gap: 8px; padding: 12px 18px; border-top: 1px solid var(--border); color: var(--text-3); font-size: 12.5px; }
.pag .spacer { flex: 1; }
.pag button { background: var(--surface-2); border: 1px solid var(--border); color: var(--text-2); border-radius: 6px; padding: 5px 11px; font-size: 12.5px; }
.pag button.on { color: var(--accent); border-color: var(--accent); }
.pag button:disabled { opacity: 0.4; cursor: default; }
</style>
+48
View File
@@ -0,0 +1,48 @@
const TOKEN_KEY = 'pa-admin-token';
let token = localStorage.getItem(TOKEN_KEY) || '';
export function getToken() { return token; }
export function setToken(t) {
token = t || '';
if (t) localStorage.setItem(TOKEN_KEY, t);
else localStorage.removeItem(TOKEN_KEY);
}
async function request(path, { method = 'GET', body } = {}) {
const res = await fetch('/api/admin' + path, {
method,
headers: {
...(body ? { 'Content-Type': 'application/json' } : {}),
...(token ? { Authorization: 'Bearer ' + token } : {}),
},
body: body ? JSON.stringify(body) : undefined,
});
let data = null;
try { data = await res.json(); } catch { data = null; }
if (!res.ok) {
const err = new Error((data && data.error && data.error.message) || ('请求失败 HTTP ' + res.status));
err.status = res.status;
err.code = data && data.error && data.error.code;
throw err;
}
return data;
}
export const api = {
login: (username, password) => request('/login', { method: 'POST', body: { username, password } }),
logout: () => request('/logout', { method: 'POST' }),
me: () => request('/me'),
units: () => request('/units'),
users: () => request('/users'),
createUser: (body) => request('/users', { method: 'POST', body }),
updateUser: (id, body) => request('/users/' + id, { method: 'PATCH', body }),
deleteUser: (id) => request('/users/' + id, { method: 'DELETE' }),
logs: (params = {}) => {
const q = new URLSearchParams();
Object.entries(params).forEach(([k, v]) => { if (v !== undefined && v !== null && v !== '') q.set(k, v); });
const qs = q.toString();
return request('/logs' + (qs ? '?' + qs : ''));
},
settings: () => request('/settings'),
updateSettings: (body) => request('/settings', { method: 'PUT', body }),
};
+57
View File
@@ -0,0 +1,57 @@
<script setup>
import { ref, watch } from 'vue';
import { api } from '../api/admin.js';
const props = defineProps({ open: Boolean, hub: Object });
const emit = defineEmits(['close', 'saved']);
const hubId = ref('');
const hubName = ref('');
const busy = ref(false);
const error = ref('');
watch(() => props.open, (open) => {
if (open) {
hubId.value = (props.hub && props.hub.hub_id) || '';
hubName.value = (props.hub && props.hub.hub_name) || '';
error.value = '';
busy.value = false;
}
});
async function save() {
if (busy.value) return;
if (!hubName.value.trim()) { error.value = '中枢名称不能为空'; return; }
busy.value = true;
error.value = '';
try {
const r = await api.updateSettings({ hub_id: hubId.value.trim(), hub_name: hubName.value.trim() });
emit('saved', r.settings);
} catch (e) {
error.value = e.message || '保存失败';
} finally {
busy.value = false;
}
}
</script>
<template>
<div v-if="open" class="overlay" @click.self="emit('close')">
<div class="modal">
<div class="modal-head">中枢设置</div>
<div class="modal-body">
<div class="field"><label>中枢名称</label><input v-model="hubName" placeholder="例如:湘湖实验室中枢"></div>
<div class="field" style="margin-bottom: 0"><label>中枢 ID</label><input v-model="hubId" class="mono" placeholder="唯一标识,留空则由系统生成"></div>
<p v-if="error" class="err">{{ error }}</p>
</div>
<div class="modal-foot">
<button class="btn btn-ghost" @click="emit('close')">取消</button>
<button class="btn btn-primary" :disabled="busy" @click="save">{{ busy ? '保存中…' : '保存' }}</button>
</div>
</div>
</div>
</template>
<style scoped>
.err { color: var(--red); font-size: 12px; margin: 12px 0 0; }
</style>
+101
View File
@@ -0,0 +1,101 @@
<script setup>
import { ref, watch } from 'vue';
import { api } from '../api/admin.js';
const props = defineProps({
open: Boolean,
mode: { type: String, default: 'create' },
user: Object,
});
const emit = defineEmits(['close', 'saved']);
const UNITS = [
['qa', '问答'], ['kg', '知识图谱'], ['extraction', '文献抽取'], ['docgen', '文档生成'],
['figures', '图片图库'], ['private_docs', '私有资料'], ['bioflow', '生信分析'],
['paper', '论文撰写'], ['legacy', '旧版接口'],
];
const username = ref('');
const password = ref('');
const role = ref('researcher');
const authorizations = ref([]);
const busy = ref(false);
const error = ref('');
watch(() => props.open, (open) => {
if (open) {
error.value = '';
busy.value = false;
if (props.mode === 'edit' && props.user) {
username.value = props.user.username;
password.value = '';
role.value = props.user.role === 'admin' ? 'admin' : (props.user.role === 'guest' ? 'guest' : 'researcher');
authorizations.value = [...(props.user.authorizations || [])];
} else {
username.value = '';
password.value = '';
role.value = 'researcher';
authorizations.value = ['qa'];
}
}
});
async function save() {
if (busy.value) return;
if (props.mode === 'create' && (!username.value.trim() || !password.value)) {
error.value = '用户名和密码必填';
return;
}
busy.value = true;
error.value = '';
try {
if (props.mode === 'create') {
await api.createUser({ username: username.value.trim(), password: password.value, role: role.value, authorizations: authorizations.value });
} else {
const body = { role: role.value, authorizations: authorizations.value };
if (password.value) body.password = password.value;
await api.updateUser(props.user.id, body);
}
emit('saved');
} catch (e) {
error.value = e.message || '保存失败';
} finally {
busy.value = false;
}
}
</script>
<template>
<div v-if="open" class="overlay" @click.self="emit('close')">
<div class="modal">
<div class="modal-head">{{ mode === 'create' ? '新建用户' : '编辑用户' }}</div>
<div class="modal-body">
<div class="field"><label>用户名</label><input v-model="username" :disabled="mode === 'edit'"></div>
<div class="field"><label>{{ mode === 'create' ? '初始密码' : '重置密码(留空不改)' }}</label><input v-model="password" type="password"></div>
<div class="field">
<label>角色</label>
<div class="seg">
<button :class="{ on: role === 'researcher' }" @click="role = 'researcher'">研究员</button>
<button :class="{ on: role === 'guest' }" @click="role = 'guest'">访客</button>
<button :class="{ on: role === 'admin' }" @click="role = 'admin'">管理员</button>
</div>
</div>
<div class="field" style="margin-bottom: 0">
<label>授权清单</label>
<div class="checks">
<label v-for="[k, label] in UNITS" :key="k" class="check"><input type="checkbox" :value="k" v-model="authorizations"> {{ label }}</label>
</div>
</div>
<p v-if="error" class="err">{{ error }}</p>
</div>
<div class="modal-foot">
<button class="btn btn-ghost" @click="emit('close')">取消</button>
<button class="btn btn-primary" :disabled="busy" @click="save">{{ busy ? '保存中…' : (mode === 'create' ? '创建用户' : '保存') }}</button>
</div>
</div>
</div>
</template>
<style scoped>
.err { color: var(--red); font-size: 12px; margin: 12px 0 0; }
</style>
+5
View File
@@ -0,0 +1,5 @@
import { createApp } from 'vue';
import App from './App.vue';
import './style.css';
createApp(App).mount('#app');
+96
View File
@@ -0,0 +1,96 @@
:root {
--bg: #0c0e14;
--bg-2: #10131b;
--surface: #151924;
--surface-2: #1a1f2c;
--border: #232a38;
--border-2: #2e3648;
--text: #e9edf5;
--text-2: #9aa4b6;
--text-3: #5f6a7d;
--accent: #e8a33d;
--accent-ink: #14100a;
--accent-soft: rgba(232, 163, 61, 0.14);
--green: #3ecf8e;
--green-soft: rgba(62, 207, 142, 0.14);
--red: #ef6b6b;
--red-soft: rgba(239, 107, 107, 0.14);
--blue: #6ea8ff;
--radius: 10px;
--radius-sm: 7px;
}
* { box-sizing: border-box; }
html, body, #app { margin: 0; height: 100%; }
body {
background: var(--bg);
color: var(--text);
font-family: -apple-system, BlinkMacSystemFont, "PingFang SC", "Microsoft YaHei", "Segoe UI", Roboto, sans-serif;
font-size: 14px;
line-height: 1.5;
-webkit-font-smoothing: antialiased;
}
.mono { font-family: ui-monospace, "SF Mono", "JetBrains Mono", Menlo, Consolas, monospace; }
button { font-family: inherit; cursor: pointer; }
.btn {
display: inline-flex; align-items: center; justify-content: center; gap: 7px;
padding: 9px 15px; font-size: 13.5px; border-radius: var(--radius-sm);
border: 1px solid transparent; transition: filter .15s; background: transparent;
}
.btn:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
.btn-primary { background: var(--accent); color: var(--accent-ink); font-weight: 600; }
.btn-primary:hover { filter: brightness(1.06); }
.btn-ghost { border-color: var(--border-2); color: var(--text-2); }
.btn-ghost:hover { color: var(--text); border-color: var(--text-3); }
.btn-danger { background: var(--red-soft); color: var(--red); }
.btn-block { width: 100%; }
.field { margin-bottom: 15px; }
.field label { display: block; font-size: 12px; color: var(--text-2); margin-bottom: 7px; }
.field input, .field select {
width: 100%; padding: 10px 12px; font-size: 14px; color: var(--text);
background: var(--bg-2); border: 1px solid var(--border); border-radius: var(--radius-sm); outline: none;
}
.field input:focus, .field select:focus { border-color: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
.chip {
display: inline-block; padding: 3px 9px; border-radius: 20px; font-size: 11.5px;
border: 1px solid var(--border-2); color: var(--text-2); margin: 1.5px 3px 1.5px 0;
}
.chip.amber { border-color: transparent; background: var(--accent-soft); color: var(--accent); }
.badge { display: inline-block; padding: 2px 8px; border-radius: 5px; font-size: 11.5px; font-family: ui-monospace, Menlo, monospace; }
.badge.green { background: var(--green-soft); color: var(--green); }
.badge.red { background: var(--red-soft); color: var(--red); }
.badge.amber { background: var(--accent-soft); color: var(--accent); }
.badge.muted { background: var(--surface-2); color: var(--text-3); }
.panel { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius); }
table { width: 100%; border-collapse: collapse; }
th { text-align: left; font-size: 11.5px; font-weight: 500; color: var(--text-3); padding: 10px 18px; border-bottom: 1px solid var(--border); }
td { padding: 13px 18px; border-bottom: 1px solid var(--border); font-size: 13.5px; }
tr:last-child td { border-bottom: none; }
tbody tr:hover { background: rgba(255,255,255,0.015); }
.cell-id { color: var(--text-2); }
.row-actions { display: flex; gap: 6px; }
.row-actions .act { color: var(--blue); cursor: pointer; font-size: 13px; padding: 3px 6px; border-radius: 5px; }
.row-actions .act:hover { background: rgba(110,168,255,0.1); }
.row-actions .act.danger { color: var(--red); }
.row-actions .act.danger:hover { background: var(--red-soft); }
.role { font-weight: 600; }
.role.admin { color: var(--accent); }
.role.researcher { color: var(--blue); }
.role.guest { color: var(--text-2); }
.overlay { position: fixed; inset: 0; background: rgba(0,0,0,0.55); display: flex; align-items: center; justify-content: center; z-index: 50; }
.modal { width: 460px; max-width: calc(100vw - 32px); background: var(--surface); border: 1px solid var(--border-2); border-radius: 14px; }
.modal-head { padding: 18px 22px 0; font-size: 15px; font-weight: 600; }
.modal-body { padding: 18px 22px; }
.modal-foot { padding: 14px 22px 20px; display: flex; justify-content: flex-end; gap: 10px; border-top: 1px solid var(--border); }
.checks { display: grid; grid-template-columns: 1fr 1fr; gap: 9px 14px; margin-top: 8px; }
.check { display: flex; align-items: center; gap: 8px; font-size: 13px; color: var(--text-2); }
.check input { accent-color: var(--accent); width: 15px; height: 15px; }
.seg { display: flex; gap: 6px; }
.seg button { background: transparent; border: 1px solid var(--border-2); color: var(--text-2); border-radius: 7px; padding: 7px 14px; font-size: 13px; }
.seg button.on { background: var(--accent-soft); border-color: transparent; color: var(--accent); }
+53
View File
@@ -0,0 +1,53 @@
<script setup>
import { ref } from 'vue';
import { api, setToken } from '../api/admin.js';
const emit = defineEmits(['login']);
const username = ref('');
const password = ref('');
const error = ref('');
const busy = ref(false);
async function submit() {
if (busy.value) return;
if (!username.value || !password.value) { error.value = '请输入用户名和密码'; return; }
busy.value = true;
error.value = '';
try {
const r = await api.login(username.value, password.value);
setToken(r.token);
emit('login', r.user);
} catch (e) {
error.value = e.message || '登录失败';
} finally {
busy.value = false;
}
}
</script>
<template>
<div class="login">
<div class="card">
<div class="brand">
<div class="hex">蜂</div>
<div><div class="brand-name">蜂群中枢</div><div class="brand-sub">PlantAI 管理后台</div></div>
</div>
<div class="field"><label>用户名</label><input v-model="username" autocomplete="username" @keyup.enter="submit"></div>
<div class="field"><label>密码</label><input v-model="password" type="password" autocomplete="current-password" @keyup.enter="submit"></div>
<button class="btn btn-primary btn-block" :disabled="busy" @click="submit">{{ busy ? '登录中…' : '登录' }}</button>
<p v-if="error" class="err">{{ error }}</p>
<div class="hint">仅限中枢管理员。登录与所有操作均会留痕。</div>
</div>
</div>
</template>
<style scoped>
.login { height: 100vh; display: flex; align-items: center; justify-content: center; background: radial-gradient(1200px 600px at 70% -10%, rgba(232, 163, 61, 0.08), transparent 60%), var(--bg); }
.card { width: 360px; background: var(--surface); border: 1px solid var(--border); border-radius: 14px; padding: 34px 32px 30px; }
.brand { display: flex; align-items: center; gap: 11px; margin-bottom: 26px; }
.hex { width: 34px; height: 34px; flex: none; background: linear-gradient(135deg, #f2b95a, #d98f22); clip-path: polygon(50% 0, 100% 25%, 100% 75%, 50% 100%, 0 75%, 0 25%); display: grid; place-items: center; color: #1a1206; font-weight: 700; font-size: 14px; }
.brand-name { font-size: 16px; font-weight: 600; }
.brand-sub { color: var(--text-3); font-size: 12px; }
.err { color: var(--red); font-size: 12px; margin: 10px 0 0; }
.hint { margin-top: 20px; padding-top: 16px; border-top: 1px solid var(--border); font-size: 12px; color: var(--text-3); }
</style>
+102
View File
@@ -0,0 +1,102 @@
<script setup>
import { ref, onMounted } from 'vue';
import { api } from '../api/admin.js';
const emit = defineEmits(['unauthorized']);
const from = ref('');
const event = ref('');
const q = ref('');
const page = ref(1);
const pageSize = 30;
const items = ref([]);
const total = ref(0);
const aggregates = ref({});
const loading = ref(false);
const EVENT_LABEL = {
http_request: '调用', admin_login_ok: '登录成功', admin_login_fail: '登录失败',
call_denied: '越权', user_created: '建用户', user_updated: '改用户', user_deleted: '删用户',
};
function statusBadge(s) {
if (!s) return 'muted';
if (s >= 200 && s < 300) return 'green';
if (s >= 500) return 'red';
return 'amber';
}
async function load() {
loading.value = true;
try {
const r = await api.logs({
from_date: from.value || undefined,
event: event.value || undefined,
q: q.value || undefined,
page: page.value,
page_size: pageSize,
});
items.value = r.items;
total.value = r.total;
aggregates.value = r.aggregates || {};
} catch (e) {
if (e.status === 401) emit('unauthorized');
} finally {
loading.value = false;
}
}
function search() { page.value = 1; load(); }
function prev() { if (page.value > 1) { page.value -= 1; load(); } }
function next() { if (page.value * pageSize < total.value) { page.value += 1; load(); } }
onMounted(load);
</script>
<template>
<div>
<div class="stats" style="margin-bottom: 16px">
<div class="stat"><div class="k">{{ total }}</div><div class="l">匹配日志</div></div>
<div class="stat"><div class="k amber">{{ aggregates.denied ?? 0 }}</div><div class="l">越权 / 失败</div></div>
<div class="stat"><div class="k">{{ aggregates.avg_latency_ms ?? '—' }}</div><div class="l">平均耗时 ms</div></div>
<div class="stat"><div class="k">{{ aggregates.p95_latency_ms ?? '—' }}</div><div class="l">P95 耗时 ms</div></div>
</div>
<div class="toolbar">
<input class="date" v-model="from" type="date">
<select v-model="event">
<option value="">全部事件</option>
<option v-for="(l, k) in EVENT_LABEL" :key="k" :value="k">{{ l }}</option>
</select>
<div class="search"><input v-model="q" placeholder="按用户 / 路径过滤…" @keyup.enter="search"></div>
<button class="btn btn-ghost" @click="search">查询</button>
</div>
<div class="panel">
<table>
<thead><tr><th>时间</th><th>用户</th><th>事件</th><th>方法</th><th>路径</th><th>状态</th><th>耗时</th></tr></thead>
<tbody>
<tr v-if="loading"><td colspan="7" class="cell-id">加载中…</td></tr>
<tr v-else-if="!items.length"><td colspan="7" class="cell-id">暂无日志</td></tr>
<tr v-for="(e, i) in items" :key="i">
<td class="cell-id mono">{{ (e.ts || '').replace('T', ' ').slice(0, 19) }}</td>
<td class="mono cell-id">{{ e.user_id ? e.user_id.slice(0, 8) : '—' }}</td>
<td>{{ EVENT_LABEL[e.event] || e.event }}</td>
<td class="cell-id mono">{{ e.method || '—' }}</td>
<td class="mono">{{ e.path || '—' }}</td>
<td><span v-if="e.status" class="badge" :class="statusBadge(e.status)">{{ e.status }}</span><span v-else class="badge muted">—</span></td>
<td class="cell-id mono">{{ e.latency_ms != null ? e.latency_ms + 'ms' : '—' }}</td>
</tr>
</tbody>
</table>
<div class="pag">
<span>共 {{ total }} 条</span><span class="spacer"></span>
<button :disabled="page <= 1" @click="prev">上一页</button>
<button class="on">{{ page }}</button>
<button :disabled="page * pageSize >= total" @click="next">下一页</button>
</div>
</div>
</div>
</template>
<style scoped>
input.date { background: var(--surface); border: 1px solid var(--border); color: var(--text); border-radius: var(--radius-sm); padding: 8px 10px; font-size: 13px; outline: none; }
</style>
+72
View File
@@ -0,0 +1,72 @@
<script setup>
import { ref, onMounted } from 'vue';
import { api } from '../api/admin.js';
import UserForm from '../components/UserForm.vue';
const emit = defineEmits(['unauthorized']);
const users = ref([]);
const loading = ref(true);
const formOpen = ref(false);
const formMode = ref('create');
const editing = ref(null);
const ROLE_LABEL = { admin: '管理员', researcher: '研究员', guest: '访客' };
const UNIT_LABEL = { qa: '问答', kg: '知识图谱', extraction: '文献抽取', docgen: '文档生成', figures: '图片图库', private_docs: '私有资料', bioflow: '生信分析', paper: '论文撰写', legacy: '旧版接口' };
async function load() {
loading.value = true;
try {
const r = await api.users();
users.value = r.users;
} catch (e) {
if (e.status === 401) emit('unauthorized');
} finally {
loading.value = false;
}
}
function create() { formMode.value = 'create'; editing.value = null; formOpen.value = true; }
function edit(u) { formMode.value = 'edit'; editing.value = u; formOpen.value = true; }
async function del(u) {
if (!confirm(`删除用户「${u.username}」?此操作不可撤销。`)) return;
try { await api.deleteUser(u.id); load(); } catch (e) { alert(e.message || '删除失败'); }
}
function onSaved() { formOpen.value = false; load(); }
onMounted(load);
</script>
<template>
<div>
<div class="toolbar">
<div class="spacer"></div>
<button class="btn btn-primary" @click="create">+ 新建用户</button>
</div>
<div class="panel">
<table>
<thead><tr><th>用户</th><th>角色</th><th>授权范围</th><th>创建时间</th><th style="width: 120px"></th></tr></thead>
<tbody>
<tr v-if="loading"><td colspan="5" class="cell-id">加载中…</td></tr>
<tr v-else-if="!users.length"><td colspan="5" class="cell-id">暂无用户</td></tr>
<tr v-for="u in users" :key="u.id">
<td>
<div style="display: flex; align-items: center; gap: 10px">
<div class="avatar">{{ u.username[0] }}</div>
<div><div>{{ u.username }}</div><div class="cell-id" style="font-size: 12px">{{ u.id.slice(0, 8) }}</div></div>
</div>
</td>
<td><span class="role" :class="u.role">{{ ROLE_LABEL[u.role] || u.role }}</span></td>
<td>
<span v-if="u.role === 'admin'" class="chip amber">全部权限</span>
<template v-else><span v-for="a in u.authorizations" :key="a" class="chip">{{ UNIT_LABEL[a] || a }}</span></template>
</td>
<td class="cell-id mono">{{ (u.created_at || '').slice(0, 10) }}</td>
<td><div class="row-actions"><span class="act" @click="edit(u)">编辑</span><span class="act danger" @click="del(u)">删除</span></div></td>
</tr>
</tbody>
</table>
</div>
<UserForm :open="formOpen" :mode="formMode" :user="editing" @close="formOpen = false" @saved="onSaved" />
</div>
</template>
+15
View File
@@ -0,0 +1,15 @@
import { defineConfig } from 'vite';
import vue from '@vitejs/plugin-vue';
export default defineConfig({
base: '/admin/',
plugins: [vue()],
server: {
host: '127.0.0.1',
port: 5181,
proxy: {
'/api': { target: 'http://127.0.0.1:8090', changeOrigin: false },
},
},
build: { outDir: 'dist', target: 'es2020' },
});
-67
View File
@@ -16,70 +16,3 @@ capabilities:
endpoint: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi
method: GET
timeout: 8
# ---- 生信能力单元(S1):两个 MCP 端点,各自独立装卸 ----
- name: bioflow-gene-family
provider: mcp
permission: read_only
description: 基因家族分析(路径规划/建任务/参数确认/文件获取与预处理/状态查询),自然语言触发
session_default: true
nl_triggers:
- 基因家族
- 家族成员
- 亚家族
- 系统发育
- 建树
- 进化树
- 保守基序
- motif
- 结构域
- 基因结构
- 共线性
- Ka/Ks
- 选择压力
- 组合证据
mcp:
url: http://127.0.0.1:8098/mcp/gene-family/
prompt_pack:
file: bioflow_gene_family_agent_rules.md
- name: bioflow-rnaseq
provider: mcp
permission: read_only
description: 转录组分析(实验设计确认/样本配置/路径规划/建任务/差异表达/富集),自然语言触发
session_default: true
nl_triggers:
- 转录组
- RNA-seq
- RNAseq
- rnaseq
- 差异表达
- 表达定量
- 富集
- WGCNA
- GSEA
- 实验设计
- fastq
mcp:
url: http://127.0.0.1:8098/mcp/rnaseq/
prompt_pack:
file: bioflow_rnaseq_agent_rules.md
# ---- 生信能力单元(S4):第三个 MCP 端点,用户自定义扩展分析 ----
- name: bioflow-extension
provider: mcp
permission: read_only
description: 用户自定义扩展分析(创建/转换/试运行/校验/发布/运行),自然语言触发
session_default: true
nl_triggers:
- 扩展
- 拓展分析
- 扩展分析
- 自定义分析
- 分析模板
- 分析脚本
- 分析工具
mcp:
url: http://127.0.0.1:8098/mcp/extension/
prompt_pack:
file: bioflow_extension_agent_rules.md
-3
View File
@@ -31,6 +31,3 @@ QA_V1_WEB_SEARCH_MAX_RESULTS=4
QA_V1_WEB_SEARCH_TIMEOUT=8
QA_V1_WEB_SEARCH_MAX_CHARS=1200
QA_V1_ENABLE_SUPERPOWERS=1
# 生信对话式能力熔断开关(S1):默认关(=0)。置 1 才启用 QA 前门 bioflow_agent。
QA_V1_ENABLE_BIOFLOW_AGENT=0
@@ -1,31 +0,0 @@
# 用户自定义扩展分析对话规范(bioflow_extension_agent_rules)
你是用户自定义扩展分析助手。引导用户把一段可运行的分析脚本/工具,通过「创建 → AI 转换 → 试运行 → 校验 → 确认 → 发布 → 运行」流程,做成 PlantAI 可自然语言触发的自定义扩展。
## 对话节奏(自然语言主导 + 卡片辅助)
每次需要用户输入/选择时,遵循固定节奏:
1. 先用自然语言说明并列表展示需要什么;
2. 提示用户可用卡片快捷操作,也可直接打字(两条通道语义一致);
3. 用户回答后,用自然语言复述,确认无误再继续。
## 创建(create_extension)
- 需要用户提供:显示名、运行命令、代码文件(可上传/粘贴文本内容)、运行时镜像、可选测试数据/金标。
- 代码文件以「路径 + 文本内容」提交;引导用户说明脚本是做什么的、依赖什么。
## AI 转换与校验(build → trial_run → validate)
- 创建后调用 build 做 AI 转换(原始命令 → command_template + 参数 + 依赖),失败时规则版兜底。
- 转换结果先向用户说明(识别到的参数/依赖/输入角色),请用户确认参数名与默认值。
- 试运行/校验的结果要转述给用户,失败时说明原因并引导修复(repair 循环)。
## 确认与发布(confirm_extension → publish_extension)
- 确认前请用户确认触发词(nl_triggers)与参数——触发词决定用户以后说什么话能唤起这个扩展。
- 发布后扩展才可被自然语言触发运行;发布是不可逆的对外可用状态,须经用户确认。
## 运行(run_extension)
- 仅已发布扩展可运行;运行参数须在白名单内(参数 schema),未知参数明确追问。
- 运行结果(run_result)转述给用户。
## 其它
- 工具名/参数名/内部实现一律不外泄;
- 状态不确定时主动查 extension_status 并向用户转述当前阶段与下一步;
- 人工确认边界(确认触发词、发布)不可跳过、不可代用户拍板。
@@ -1,32 +0,0 @@
# 基因家族分析对话规范(bioflow_gene_family_agent_rules)
你是基因家族分析助手。用自然语言引导用户从需求到分析完成,全程对话式交互。
## 人工确认边界(不可跳过、不可代确认)
- **subfamily_confirm(亚家族预测人工确认)是硬性人工边界**:系统预测的亚家族划分结果,
必须停下来请用户查看并确认或修改。你绝不能代用户确认,也不能在用户未确认前继续下游分析。
- 用户提出修改(如"把某成员从亚家族 A 移到亚家族 B")时,先复述修改内容,再请用户做最终确认。
## 对话节奏(自然语言主导 + 卡片辅助)
每次需要用户输入/选择时,遵循固定节奏:
1. 先用自然语言说明并**列表展示**选项及含义;
2. 提示用户可用卡片快捷操作,也可直接打字回复(两条通道语义一致);
3. 用户选择后,用自然语言**复述**其选择,确认无误再继续。
## 路径规划与建任务
- 规划后列出"核心流程(必选)+ 可选分析"候选路径,说明每个可选分析的生物学意义;
- **建任务前必须经用户确认阶段链**,收到确认后再创建任务,绝不擅自建任务。
## 文件获取与预处理
- 源文件支持上传、http/https 链接、OSS 地址、局域网机器路径;
- 文件到达后先探测格式,**格式异常(如 Excel/文本)时先向用户说明列结构并提问确认**
(哪列是序列、是否去重、是否去表头等),再由模型整理成 fasta/gff 等标准格式;
- 转换结果过格式验证,失败时说明问题并追问修正(循环不超过 3 次后建议人工整理)。
## 参数确认
- 列"参数名(默认值,含义)"请用户确认或修改;未识别参数时明确指出并给出允许范围。
## 其它
- 工具名、参数名、内部实现一律不外泄;
- 任务运行中用户提问(流程/参数/生物学意义)随时回答,不打断任务;
- 卡住或不确定时主动查询任务状态,向用户转述当前进度与下一步。
@@ -1,29 +0,0 @@
# 转录组分析对话规范(bioflow_rnaseq_agent_rules)
你是转录组分析助手。用自然语言引导用户从实验设计确认到差异表达/富集分析完成。
## 人工确认边界(不可跳过、不可代确认)
- **实验设计确认(rnaseq_de 之前的必确认信息)是硬性边界**:开始分析前必须与用户确认——
① 物种与参考基因组版本;② 每个样本的 FASTQ(上传/链接/局域网路径)及单端双端;
③ 对照组/处理组各几个重复、有无时间点/批次;④ 建库是否链特异性。
- 统计模型正确性依赖上述信息,未确认前不得创建差异表达任务,也不能代用户拍板基线/比较方向。
## 对话节奏(自然语言主导 + 卡片辅助)
每次需要用户输入/选择时,遵循固定节奏:
1. 先用自然语言说明并**列表展示**必确认信息与可选分析;
2. 提示用户可用卡片快捷操作,也可直接打字回复(两条通道语义一致);
3. 用户回答后,用自然语言**复述**其样本设计与选择(如"干旱 3 vs 对照 3,双端,非链特异,对照为基线"),
确认无误再继续。
## 样本配置与文件预处理
- 用户用自然语言描述样本时,可解析生成 samples.yaml(样本名/对照处理组/重复/单双端);
- FASTQ 支持上传、http/https 链接、OSS 地址、局域网路径;核对成对完整;
- 非 FASTQ 的输入文件(表格/文本等)先探测格式,**异常时先说明列结构并提问确认**,再整理为可用格式。
## 参数确认
- 列"参数名(默认值,含义)"请用户确认或修改;未识别参数时明确指出并给出允许范围。
## 其它
- 工具名、参数名、内部实现一律不外泄;
- 任务运行中用户提问(流程/参数/实验设计意义)随时回答,不打断任务;
- 卡住或不确定时主动查询任务状态,向用户转述当前进度与下一步(含样本质控 PCA、差异基因数目等结果转述)。
-29
View File
@@ -13,7 +13,6 @@ from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from . import __version__
from .bioflow_agent import try_bioflow_agent
from .config import QAConfig
from .pipeline import QAAnswerError, QAV1Pipeline
@@ -48,9 +47,6 @@ class QARequest(BaseModel):
# ---- Capability Gateway(S1):纯数据字段,无 validator(Cython 编译安全)----
history: List[Dict[str, str]] = Field(default_factory=list)
capabilities: Dict[str, Any] = Field(default_factory=dict)
# ---- Bioflow 对话式编排(S2):卡片回填协议 + session↔job 绑定 ----
card_result: Optional[Dict[str, Any]] = None
bioflow: Optional[Dict[str, Any]] = None
@lru_cache(maxsize=1)
@@ -154,18 +150,6 @@ async def chat(request: QARequest) -> Dict[str, Any]:
detail={"error_code": "llm_unavailable", "answer": "", "message": "Final-answer LLM is not configured"},
)
try:
# ---- Bioflow Agent 前门(S1):命中则短路返回,否则走原 pipeline ----
bioflow_result = await asyncio.to_thread(
try_bioflow_agent,
request.question,
llm_client=pipeline.llm,
history=request.history,
capabilities=request.capabilities,
card_result=request.card_result,
bioflow=request.bioflow,
)
if bioflow_result is not None:
return bioflow_result
ablation = dict(request.ablation or {})
ablation["use_final_llm"] = True
return await asyncio.to_thread(
@@ -232,19 +216,6 @@ async def chat_stream(request: QARequest) -> StreamingResponse:
def run() -> None:
try:
# ---- Bioflow Agent 前门(S1):命中则短路,否则走原 pipeline ----
bioflow_result = try_bioflow_agent(
request.question,
llm_client=pipeline.llm,
history=request.history,
capabilities=request.capabilities,
card_result=request.card_result,
bioflow=request.bioflow,
progress_callback=emit,
)
if bioflow_result is not None:
events.put(("complete", bioflow_result))
return
ablation = dict(request.ablation or {})
ablation["use_final_llm"] = True
result = pipeline.answer(
@@ -1,684 +0,0 @@
from __future__ import annotations
"""bioflow_agent.py —— 生信 MCP 能力前门路由(S1 + S2)。
S1:三重门(会话开关/注册表/意图预筛)→ ToolRunner 工具循环 → 自然语言转述。
S2:对话式编排闭环——card_result 卡片回填协议 + session↔job 绑定 + 多轮记忆
+ 参数抽取闭环 + job_status 主动查询转述。
card_result 协议(前端卡片操作 → 结构化用户消息 → QA 流):
{card: path_selection|form_confirm|file_upload|archive_choice, payload: {...}}
BioflowAgent 读取后经工具执行,再自然语言复述;卡片与自然语言两条通道在
服务端语义一致(QA 对话状态机是唯一决策点)。
session↔job 绑定:BFF sessions.meta.bioflow = {active_job_id, chain},经请求
body["bioflow"] 注入(含 session_id);job_status 工具是任务真相唯一权威。
金标路径零改动:任何不命中/异常都返回 None 回退原 pipeline。
"""
import json
import logging
import os
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
from .capabilities.manifests import CapabilityManifest
from .capabilities.providers import PromptPackProvider, get_dispatcher
from .capabilities.registry import get_registry
from .tool_runner import ToolRunner, make_llm_complete
LOGGER = logging.getLogger(__name__)
# 三个生信能力单元(与 bioflow_service 三个 MCP 端点一一对应)
BIOFLOW_CAPABILITY_NAMES = ("bioflow-gene-family", "bioflow-rnaseq", "bioflow-extension")
ENV_FUSE = "QA_V1_ENABLE_BIOFLOW_AGENT"
MAX_ROUNDS = 5
# card_result 卡片类型(S2 前端卡片回填协议)
CARD_PATH_SELECTION = "path_selection"
CARD_FORM_CONFIRM = "form_confirm"
CARD_FILE_UPLOAD = "file_upload"
CARD_ARCHIVE_CHOICE = "archive_choice"
CARD_EXTENSION_ACTION = "extension_action" # S4:扩展能力单元操作(build/trial/validate/confirm/publish/run)
CARD_TYPES = (CARD_PATH_SELECTION, CARD_FORM_CONFIRM, CARD_FILE_UPLOAD, CARD_ARCHIVE_CHOICE, CARD_EXTENSION_ACTION)
# chain 名 → 能力名(建任务时选择哪条链的 create 工具)
_CHAIN_TO_CAPABILITY = {
"gene_family": "bioflow-gene-family",
"rnaseq": "bioflow-rnaseq",
}
def _chain_to_capability(chain: str) -> Optional[str]:
"""把 chain 名(gene_family/rnaseq)映射为能力名;未知返回 None。"""
return _CHAIN_TO_CAPABILITY.get(str(chain or "").strip().lower())
_SYSTEM_BASE = (
"你是 PlantAI 生信分析助手,用自然语言引导用户完成基因家族或转录组分析。\n"
"你可以调用生信工具完成规划、建任务、确认参数、登记文件、查状态等操作。\n"
"硬性规则:\n"
"1. 回复只使用自然语言,绝不向用户泄露工具名、参数名或内部实现细节。\n"
"2. 需要用户提供文件/参数/做选择时,先自然语言说明并列表展示选项与含义,再请用户回答,"
"绝不擅自替用户做不可逆决策。\n"
"3. 人工确认边界(基因家族 subfamily_confirm、转录组实验设计确认)不可跳过、不可代用户确认。\n"
"4. 任务运行中可随时回答流程/参数/生物学意义类问题;卡住或不确定时主动查询任务状态,"
"再向用户转述当前进度与下一步(等待文件、等待参数确认、失败可重跑等)。\n"
"5. 文件格式异常时先向用户说明并提问确认,再由用户决定如何处理。\n"
"6. 用户用自然语言给出参数(如「用 8 核、padj 用 0.01」)时,抽取为结构化参数后经确认步骤提交;"
"只提交白名单内的参数名,未知参数明确指出并追问(最多 3 次)。\n"
"7. 若用户消息是对上一轮卡片/提问的确认或选择,先复述用户的选择,确认无误再执行对应动作。\n"
)
def _env_enabled() -> bool:
"""门 1:QA_V1_ENABLE_BIOFLOW_AGENT 熔断开关。默认关(未设置 = 关)。"""
raw = os.environ.get(ENV_FUSE)
return raw is not None and str(raw).strip().lower() in {"1", "true", "yes", "on"}
def _intent_match(question: str, triggers: List[str]) -> bool:
"""门 3:意图预筛——问句是否命中任一触发词(大小写不敏感子串匹配)。"""
if not triggers:
return False
q = (question or "").lower()
return any(t and t.lower() in q for t in triggers)
def _registered_capabilities(capabilities: Optional[Dict[str, Any]]) -> List[CapabilityManifest]:
"""注册表 + 会话开关:返回所有已注册且会话开启的生信能力 manifest(不判意图)。"""
try:
registry = get_registry()
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: registry 访问失败,回退原 pipeline: %s", exc)
return []
session_caps = capabilities or {}
out: List[CapabilityManifest] = []
for name in BIOFLOW_CAPABILITY_NAMES:
try:
spec = registry.get(name)
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: registry.get(%s) 失败: %s", name, exc)
continue
if spec is None:
continue # 未注册
manifest = spec.manifest
if not bool(session_caps.get(name, manifest.session_default)):
continue # 会话开关关
out.append(manifest)
return out
def _candidate_capabilities(
question: str,
capabilities: Optional[Dict[str, Any]],
) -> List[CapabilityManifest]:
"""门 2 + 门 3:注册表 + 会话开关 + 意图预筛,返回命中的候选能力清单。"""
return [
m for m in _registered_capabilities(capabilities)
if _intent_match(question, m.nl_triggers)
]
def _build_tools(
manifests: List[CapabilityManifest],
) -> Tuple[List[Dict[str, Any]], Dict[str, CapabilityManifest]]:
"""经 MCP list_tools 发现各能力工具,构造 LLM 工具表 + ToolRunner manifests。
返回 (tools, tool_manifests):
- tools: OpenAI function schema 列表(真实 server 工具名/描述/入参 schema)
- tool_manifests: {真实工具名: manifest},manifest.name=能力名(共享连接),
mcp.tool=真实工具名
多能力同时命中时:专有工具名本就不同,各自保留;共享工具(两端点语义一致)
按名去重,只保留首个能力的一份,避免 LLM tools 重名 + 同名覆盖。
"""
dispatcher = get_dispatcher()
tools: List[Dict[str, Any]] = []
tool_manifests: Dict[str, CapabilityManifest] = {}
for cap_manifest in manifests:
url = str((cap_manifest.config.get("mcp") or {}).get("url") or "").strip()
if not url:
LOGGER.warning("bioflow agent: 能力 %s 缺 mcp.url,跳过", cap_manifest.name)
continue
try:
discovered = dispatcher.list_mcp_tools(cap_manifest)
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: 能力 %s MCP 工具发现失败: %s", cap_manifest.name, exc)
continue
for item in discovered:
tool_name = str(item.get("name") or "").strip()
if not tool_name:
continue
if tool_name in tool_manifests:
# 共享工具去重(见 S1 修复说明)
continue
input_schema = item.get("input_schema") or {}
if not isinstance(input_schema, dict):
input_schema = {"type": "object", "properties": {}}
tools.append(
{
"type": "function",
"function": {
"name": tool_name,
"description": str(item.get("description") or "")[:1000],
"parameters": input_schema,
},
}
)
tool_manifests[tool_name] = CapabilityManifest(
name=cap_manifest.name,
provider="mcp",
permission="read_only",
description=str(item.get("description") or ""),
config={"mcp": {"url": url, "tool": tool_name}},
)
return tools, tool_manifests
def _build_system(manifests: List[CapabilityManifest]) -> str:
"""基础系统提示 + 各能力对话规范(prompt_pack 注入)。"""
qa_root = Path(os.environ.get("GENE_FAMILY_ROOT", "/data/gene-family-mas-v2"))
parts = [_SYSTEM_BASE]
seen: set[str] = set()
for manifest in manifests:
file_name = str((manifest.config.get("prompt_pack") or {}).get("file") or "").strip()
if not file_name or file_name in seen:
continue
seen.add(file_name)
try:
text = PromptPackProvider.load(manifest, qa_root)
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: prompt_pack 加载失败 %s: %s", file_name, exc)
text = ""
if text:
parts.append(text)
return "\n\n".join(parts)
# ---------------------------------------------------------------------------
# S2 工具执行 / 转述 / 参数抽取 辅助
# ---------------------------------------------------------------------------
def _execute_tool(
tool_manifests: Dict[str, CapabilityManifest], tool_name: str, arguments: Dict[str, Any]
) -> str:
"""执行单个工具,返回结果文本(异常转文本,不炸主流程)。"""
manifest = tool_manifests.get(tool_name)
if manifest is None:
return f"工具 {tool_name} 未注册"
try:
return get_dispatcher().execute(manifest, arguments)
except Exception as exc: # noqa: BLE001
return f"工具执行失败: {type(exc).__name__}: {exc}"
def _paraphrase(llm_client: Any, user_text: str, system: Optional[str] = None) -> str:
"""把工具执行结果转述为自然语言(LLM 优先,失败回退原文,绝不抛异常)。"""
system = system or (
"你是 PlantAI 生信分析助手。把下面的工具执行结果转述为对用户的自然语言回复,"
"绝不提及工具名/参数名/内部实现;需要用户下一步操作时明确说明。"
)
try:
text = llm_client.complete(system, user_text, temperature=0.2, max_tokens=600)
if text and str(text).strip():
return str(text).strip()
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: paraphrase LLM 失败,回退原文: %s", exc)
return str(user_text or "").strip()
def _extract_params(
llm_client: Any, user_text: str, form_defaults: Dict[str, Any]
) -> Tuple[Dict[str, Any], List[str]]:
"""LLM 抽取用户自然语言中的阶段确认参数 → (已识别参数, 未识别参数列表)。
白名单 = form_defaults 的键;未识别参数(不在白名单)单独返回供追问(≤3 次)。
参照 extension_llm 的「LLM 出 JSON → 白名单净化」模式,LLM 失败/坏 JSON 返回空。
"""
whitelist = {str(k) for k in (form_defaults or {}).keys()}
if not whitelist:
return {}, []
system = (
"你是参数抽取助手。从用户的话中抽取阶段确认参数,只输出 JSON。\n"
'输出形态:{"params": {"<参数名>": <值>}, "unknown": ["<未识别参数名>"]}。\n'
f"允许的参数名(白名单):{', '.join(sorted(whitelist))}。\n"
"只抽取用户明确提到的参数;不在白名单里的参数名放进 unknown。"
)
try:
raw = llm_client.complete_json(system, user_text, max_tokens=500)
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: 参数抽取 LLM 失败: %s", exc)
return {}, []
if not isinstance(raw, dict):
return {}, []
params = raw.get("params") if isinstance(raw.get("params"), dict) else {}
known = {str(k): v for k, v in params.items() if str(k) in whitelist}
unknown_raw = raw.get("unknown") or []
if isinstance(unknown_raw, str):
unknown_raw = [unknown_raw]
unknown = [str(x).strip() for x in unknown_raw if isinstance(x, str) and str(x).strip()]
return known, unknown
def _try_parse_json(text: str) -> Optional[Any]:
if not text:
return None
try:
return json.loads(text)
except (json.JSONDecodeError, ValueError):
return None
def _extract_job_id(result_text: str) -> Optional[str]:
"""从 create_job 工具结果文本里提取 job_id(结果形如 {"ok":true,"data":{"id":...}})。"""
data = _try_parse_json(result_text)
if isinstance(data, dict):
inner = data.get("data") if isinstance(data.get("data"), dict) else data
jid = inner.get("id") or inner.get("job_id")
if jid:
return str(jid)
return None
def _query_job_status(job_id: str, tool_manifests: Dict[str, CapabilityManifest], llm_client: Any) -> str:
"""主动查 job_status 并转述(任务真相唯一权威)。"""
result = _execute_tool(tool_manifests, "job_status", {"job_id": job_id})
return _paraphrase(llm_client, f"查询任务状态的结果:\n{result}")
def _empty_audit() -> Dict[str, Any]:
return {"rounds": 0, "tool_calls": 0, "used_json_fallback": False, "tool_names": [], "errors": []}
def _make_payload(
question: str,
answer: str,
*,
audit: Optional[Dict[str, Any]] = None,
capabilities: Optional[List[str]] = None,
active_job_id: Optional[str] = None,
chain: Optional[str] = None,
) -> Dict[str, Any]:
payload: Dict[str, Any] = {
"success": True,
"version": "bioflow-agent-s2",
"question": question,
"answer": answer,
"sources": [],
"source_count": 0,
"citation_validation": {"valid": True, "cited_pmids": [], "unknown_pmids": []},
"bioflow": {
"handled": True,
"audit": audit or _empty_audit(),
"capabilities": capabilities or [],
},
}
if active_job_id:
payload["bioflow"]["active_job_id"] = str(active_job_id)
if chain:
payload["bioflow"]["chain"] = str(chain)
return payload
# ---------------------------------------------------------------------------
# card_result 回填处理(S2)
# ---------------------------------------------------------------------------
def _card_path_selection(
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
) -> Optional[Dict[str, Any]]:
"""路径卡确认 → create_*_job → 复述;带 reuse_archive 时先复用存档的阶段链/参数再建任务。"""
chain = str(payload.get("chain") or (bioflow or {}).get("chain") or "").strip()
stages = payload.get("stages")
reuse_name = str(payload.get("reuse_archive") or payload.get("reuse_name") or "").strip()
has_stages = isinstance(stages, list) and len(stages) > 0
if not has_stages and not reuse_name:
# 既无显式阶段链也无存档复用来源 → 无法建任务,回退原 pipeline
return None
cap_name = _chain_to_capability(chain)
if cap_name is None:
# 无法确定能力(chain 空/未知):不能默认走某条链,回退原 pipeline
return None
tool = "create_gene_family_job" if cap_name == "bioflow-gene-family" else "create_rnaseq_job"
if tool not in tool_manifests:
return None
args: Dict[str, Any] = {
"session_id": str((bioflow or {}).get("session_id") or ""),
"stages": [str(s) for s in stages] if isinstance(stages, list) and stages else [],
"title": str(payload.get("title") or ""),
"auto_continue": bool(payload.get("auto_continue", False)),
}
tool_names: List[str] = []
# 存档复用(FR-BIO-023):先 reuse_archive 取阶段链/参数/archive_id,再以其发起
if reuse_name and "reuse_archive" in tool_manifests:
tool_names.append("reuse_archive") # 实际先执行 reuse_archive(audit 顺序与实际一致)
reuse_text = _execute_tool(tool_manifests, "reuse_archive", {"name": reuse_name})
arch = _try_parse_json(reuse_text)
if isinstance(arch, dict) and arch.get("ok"):
data = arch.get("data") if isinstance(arch.get("data"), dict) else {}
if data.get("id"):
args["archive_id"] = str(data["id"])
if not args["stages"] and isinstance(data.get("stage_chain"), list):
args["stages"] = [str(s) for s in data["stage_chain"]]
if isinstance(data.get("params"), dict):
merged = dict(data["params"])
if isinstance(payload.get("params"), dict):
merged.update(payload["params"])
args["params"] = merged
elif not args["stages"]:
# 复用失败(非 ok)且无显式阶段链:不能空链建任务,回退原 pipeline
return None
tool_names.append(tool) # create_*_job 最后执行
if "params" not in args and isinstance(payload.get("params"), dict):
args["params"] = payload["params"]
result = _execute_tool(tool_manifests, tool, args)
job_id = _extract_job_id(result)
answer = _paraphrase(
llm_client,
f"用户确认了分析路径并创建任务({'基因家族' if cap_name == 'bioflow-gene-family' else '转录组'}链"
f"{',复用存档 ' + reuse_name if reuse_name else ''})。工具结果:\n{result}",
)
return _make_payload(
"确认路径并创建任务", answer,
audit={"rounds": 1, "tool_calls": len(tool_names), "used_json_fallback": False, "tool_names": tool_names, "errors": []},
capabilities=[cap_name] if cap_name else [],
active_job_id=job_id,
chain=chain or None,
)
def _card_form_confirm(
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
) -> Optional[Dict[str, Any]]:
"""参数卡确认 → confirm_stage(白名单校验)→ 复述;rerun:true 走 rerun_stage(改参数重跑)。"""
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
stage = str(payload.get("stage") or "").strip()
params = payload.get("params") if isinstance(payload.get("params"), dict) else {}
rerun = bool(payload.get("rerun", False))
if not job_id or not stage:
return None
tool = "rerun_stage" if rerun else "confirm_stage"
result = _execute_tool(tool_manifests, tool, {"job_id": job_id, "stage": stage, "params": params})
answer = _paraphrase(
llm_client,
f"用户{'重跑并确认' if rerun else '确认'}了阶段参数(任务 {job_id} 的 {stage})。工具结果:\n{result}",
)
return _make_payload(
f"{'重跑' if rerun else '确认'}阶段参数:{stage}", answer,
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": [tool], "errors": []},
active_job_id=job_id,
)
def _card_file_upload(
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
) -> Optional[Dict[str, Any]]:
"""文件卡 → 网络源 fetch_source_file / 服务器·局域网路径 register_job_files → 复述。"""
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
raw_files = payload.get("paths") or payload.get("files") or []
if isinstance(raw_files, str):
raw_files = [raw_files]
files = [str(f) for f in raw_files if str(f).strip()] if isinstance(raw_files, list) else []
if not job_id or not files:
return None
remote = [f for f in files if f.startswith(("http://", "https://", "oss://"))]
local = [f for f in files if f not in remote]
results: List[str] = []
tool_names: List[str] = []
for src in remote:
r = _execute_tool(tool_manifests, "fetch_source_file", {"source": src, "job_id": job_id})
results.append(r)
tool_names.append("fetch_source_file")
if local:
r = _execute_tool(tool_manifests, "register_job_files", {"job_id": job_id, "paths": local})
results.append(r)
tool_names.append("register_job_files")
answer = _paraphrase(
llm_client,
f"用户提供了文件({'远程 ' + str(len(remote)) + ' 个' if remote else ''}"
f"{'本地 ' + str(len(local)) + ' 个' if local else ''})。工具结果:\n" + "\n".join(results),
)
return _make_payload(
"登记任务文件", answer,
audit={"rounds": 1, "tool_calls": len(tool_names), "used_json_fallback": False, "tool_names": tool_names, "errors": []},
active_job_id=job_id,
)
def _card_archive_choice(
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
) -> Optional[Dict[str, Any]]:
"""存档卡 → archive_job 工具执行实际存档(或跳过)→ 复述。"""
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
do_archive = bool(payload.get("archive", False))
if not job_id:
return None
if do_archive:
name = str(payload.get("name") or "")
description = str(payload.get("description") or "")
result = _execute_tool(
tool_manifests, "archive_job",
{"job_id": job_id, "name": name, "description": description},
)
answer = _paraphrase(
llm_client,
f"用户选择把任务 {job_id} 存档(名称:{name or '未命名'})。工具结果:\n{result}",
)
return _make_payload(
"存档选择", answer,
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": ["archive_job"], "errors": []},
active_job_id=job_id,
)
answer = _paraphrase(
llm_client,
f"用户选择不存档任务 {job_id}。请用自然语言告知用户:已跳过存档,"
"本次分析结果仍可在报告/结果中查看。",
)
return _make_payload(
"存档选择", answer,
audit=_empty_audit(),
active_job_id=job_id,
)
def _card_extension_action(
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
) -> Optional[Dict[str, Any]]:
"""扩展操作卡(S4):action ∈ build|trial_run|validate|confirm|publish|run → 对应扩展工具。"""
ext_id = str(payload.get("ext_id") or "").strip()
action = str(payload.get("action") or "").strip()
if not ext_id or not action:
return None
tool = {
"build": "build_extension",
"trial_run": "trial_run_extension",
"validate": "validate_extension",
"confirm": "confirm_extension",
"publish": "publish_extension",
"run": "run_extension",
"repair": "repair_extension",
}.get(action)
if tool is None or tool not in tool_manifests:
return None
args: Dict[str, Any] = {"ext_id": ext_id}
if action == "confirm":
if isinstance(payload.get("nl_triggers"), list):
args["nl_triggers"] = payload["nl_triggers"]
if isinstance(payload.get("parameters"), list):
args["parameters"] = payload["parameters"]
if payload.get("runtime_image"):
args["runtime_image"] = str(payload["runtime_image"])
if action == "run":
if isinstance(payload.get("params"), dict):
args["params"] = payload["params"]
if isinstance(payload.get("files"), list):
args["files"] = payload["files"]
result = _execute_tool(tool_manifests, tool, args)
if action == "confirm" and "EXT_STATUS" in result:
# 状态降级话术:confirm 要求 validated/trial_passed 前置状态,
# 编辑修正若发生在校验前,明确提示先试运行/校验,不报 500。
answer = (
"该扩展当前状态还不能确认。请先完成 AI 转换与试运行、通过校验后,"
"再确认参数与触发词。"
)
else:
answer = _paraphrase(
llm_client,
f"用户对扩展 {ext_id} 执行了操作 {action}。工具结果:\n{result}",
)
return _make_payload(
f"扩展操作:{action}", answer,
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": [tool], "errors": []},
)
def _handle_card_result(
card_result: Dict[str, Any],
llm_client: Any,
bioflow: Optional[Dict[str, Any]],
tool_manifests: Dict[str, CapabilityManifest],
) -> Optional[Dict[str, Any]]:
card = str(card_result.get("card") or "").strip()
payload = card_result.get("payload") if isinstance(card_result.get("payload"), dict) else {}
ctx = bioflow or {}
if card == CARD_PATH_SELECTION:
return _card_path_selection(payload, llm_client, ctx, tool_manifests)
if card == CARD_FORM_CONFIRM:
return _card_form_confirm(payload, llm_client, ctx, tool_manifests)
if card == CARD_FILE_UPLOAD:
return _card_file_upload(payload, llm_client, ctx, tool_manifests)
if card == CARD_ARCHIVE_CHOICE:
return _card_archive_choice(payload, llm_client, ctx, tool_manifests)
if card == CARD_EXTENSION_ACTION:
return _card_extension_action(payload, llm_client, ctx, tool_manifests)
return None
# ---------------------------------------------------------------------------
# 前门路由
# ---------------------------------------------------------------------------
def try_bioflow_agent(
question: str,
*,
llm_client: Any,
history: Optional[List[Dict[str, str]]] = None,
capabilities: Optional[Dict[str, Any]] = None,
card_result: Optional[Dict[str, Any]] = None,
bioflow: Optional[Dict[str, Any]] = None,
progress_callback: Optional[Any] = None,
) -> Optional[Dict[str, Any]]:
"""前门路由入口。命中生信能力则返回答案 payload,否则返回 None(走原 pipeline)。
- card_result:非空时走卡片回填协议(直接路由到对应工具,不做意图预筛)。
- 任何异常都不上抛——记日志后返回 None,保证金标路径永不因本模块失败。
"""
try:
return _try_bioflow_agent_impl(
question,
llm_client=llm_client,
history=history,
capabilities=capabilities,
card_result=card_result,
bioflow=bioflow,
progress_callback=progress_callback,
)
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: 执行失败,回退原 pipeline: %s", exc)
return None
def _try_bioflow_agent_impl(
question: str,
*,
llm_client: Any,
history: Optional[List[Dict[str, str]]],
capabilities: Optional[Dict[str, Any]],
card_result: Optional[Dict[str, Any]],
bioflow: Optional[Dict[str, Any]],
progress_callback: Optional[Any],
) -> Optional[Dict[str, Any]]:
# 门 1:会话开关(熔断)
if not _env_enabled():
return None
# card_result 卡片操作优先:用户已在卡片上明确选择,不做意图预筛,直接路由
if isinstance(card_result, dict) and str(card_result.get("card") or "").strip() in CARD_TYPES:
return _route_card_result(card_result, llm_client, capabilities, bioflow, progress_callback)
# NL 路径:门 2 + 门 3
manifests = _candidate_capabilities(question, capabilities)
if not manifests:
return None
tools, tool_manifests = _build_tools(manifests)
if not tools:
LOGGER.warning("bioflow agent: 意图命中但 MCP 工具发现为空,回退原 pipeline")
return None
if progress_callback is not None:
try:
progress_callback(
{
"stage": "bioflow_agent_started",
"capabilities": [m.name for m in manifests],
"tool_count": len(tools),
}
)
except Exception: # noqa: BLE001
pass
system = _build_system(manifests)
runner = ToolRunner(make_llm_complete(llm_client), max_rounds=MAX_ROUNDS)
content, audit = runner.run(system, question, tools, tool_manifests, history=history)
answer = (content or "").strip()
if not answer:
return None
return _make_payload(
question,
answer,
audit=audit,
capabilities=[m.name for m in manifests],
active_job_id=(bioflow or {}).get("active_job_id"),
chain=(bioflow or {}).get("chain"),
)
def _route_card_result(
card_result: Dict[str, Any],
llm_client: Any,
capabilities: Optional[Dict[str, Any]],
bioflow: Optional[Dict[str, Any]],
progress_callback: Optional[Any],
) -> Optional[Dict[str, Any]]:
"""卡片操作路由:确定相关能力 → 构建工具 → 分发到对应 card handler。"""
card = str(card_result.get("card") or "").strip()
payload = card_result.get("payload") if isinstance(card_result.get("payload"), dict) else {}
chain = str(payload.get("chain") or (bioflow or {}).get("chain") or "").strip()
if card == CARD_EXTENSION_ACTION:
# 扩展操作只依赖 bioflow-extension 能力,避免对三个能力各做一次 list_tools
cap_name = "bioflow-extension"
else:
cap_name = _chain_to_capability(chain)
registered = _registered_capabilities(capabilities)
if cap_name:
manifests = [m for m in registered if m.name == cap_name]
else:
manifests = registered
if not manifests:
return None
tools, tool_manifests = _build_tools(manifests)
if not tools:
LOGGER.warning("bioflow agent: card_result 但 MCP 工具发现为空,回退原 pipeline")
return None
if progress_callback is not None:
try:
progress_callback({"stage": "bioflow_card", "card": card_result.get("card")})
except Exception: # noqa: BLE001
pass
return _handle_card_result(card_result, llm_client, bioflow, tool_manifests)
@@ -128,8 +128,6 @@ class InstallManager:
"allowed_llm_phases": manifest.allowed_llm_phases,
"session_default": manifest.session_default,
}
if manifest.nl_triggers:
raw["nl_triggers"] = manifest.nl_triggers
if manifest.schema:
raw["schema"] = manifest.schema
for section in ("http", "mcp", "python", "prompt_pack"):
@@ -37,7 +37,6 @@ class CapabilityManifest:
session_default: bool = False
schema: Dict[str, Any] = field(default_factory=dict)
config: Dict[str, Any] = field(default_factory=dict) # provider 专属段(http:/mcp:/python:/prompt_pack: 归并)
nl_triggers: List[str] = field(default_factory=list) # 意图预筛触发词(S1,生信能力路由用)
def to_tool_spec(self) -> Dict[str, Any]:
"""转 OpenAI function schema(tools 参数形态)。"""
@@ -111,19 +110,6 @@ def _parse_entry(raw: Any, index: int, load_warnings: List[str]) -> CapabilityMa
if isinstance(raw.get(section), dict):
config[section] = raw[section]
config["session_default"] = bool(raw.get("session_default", False))
# nl_triggers(S1):意图预筛触发词,只收字符串、去空、去重、限长
nl_triggers_raw = raw.get("nl_triggers") or []
if isinstance(nl_triggers_raw, str):
nl_triggers_raw = [nl_triggers_raw]
nl_triggers: List[str] = []
if isinstance(nl_triggers_raw, list):
for trigger in nl_triggers_raw:
if not isinstance(trigger, str):
continue # 触发词只收字符串,非字符串(数字等)丢弃
cleaned = _clean_text(trigger, 64)
if cleaned and cleaned not in nl_triggers:
nl_triggers.append(cleaned)
nl_triggers = nl_triggers[:64]
manifest = CapabilityManifest(
name=name,
provider=provider,
@@ -133,7 +119,6 @@ def _parse_entry(raw: Any, index: int, load_warnings: List[str]) -> CapabilityMa
session_default=bool(raw.get("session_default", False)),
schema=schema,
config=config,
nl_triggers=nl_triggers,
)
# 大小上限:单条目序列化后 ≤8KB
import json
@@ -130,7 +130,6 @@ class _McpConnection:
self._thread: Optional[threading.Thread] = None
self._session: Any = None
self._server_tools: List[str] = []
self._server_tool_schemas: List[Dict[str, Any]] = []
self._started_at = 0.0
self._last_used = 0.0
self._restarts = 0
@@ -201,18 +200,6 @@ class _McpConnection:
raise
self._ctx = ctx
self._server_tools = [t.name for t in tools_result.tools]
self._server_tool_schemas = [
{
"name": t.name,
"description": (getattr(t, "description", None) or ""),
"input_schema": (
getattr(t, "inputSchema", None)
or getattr(t, "input_schema", None)
or {}
),
}
for t in tools_result.tools
]
self._session = session
self._started_at = time.time()
@@ -238,7 +225,6 @@ class _McpConnection:
ctx = getattr(self, "_ctx", None)
self._ctx = None
self._server_tools = []
self._server_tool_schemas = []
if session is not None:
try:
await session.__aexit__(None, None, None)
@@ -288,14 +274,6 @@ class _McpConnection:
return []
return list(self._server_tools)
def list_tool_schemas(self) -> List[Dict[str, Any]]:
"""返回 MCP server 全量工具 schema(name/description/input_schema)。连接失败返回空。"""
try:
self._ensure_session()
except ToolExecutionError:
return []
return list(self._server_tool_schemas)
def call(self, tool_name: str, arguments: Dict[str, Any]) -> str:
self._ensure_session()
try:
@@ -359,11 +337,6 @@ class McpToolProvider:
return tools + [f"<缺失声明工具 {tool_name}>"]
return tools
def list_tool_schemas(self, manifest: CapabilityManifest) -> List[Dict[str, Any]]:
"""返回该能力 MCP server 的全量工具 schema(供前门 agent 构建 LLM 工具表)。"""
conn = self._connection(manifest)
return conn.list_tool_schemas()
def reap_idle(self) -> int:
reaped = 0
with self._lock:
@@ -475,10 +448,6 @@ class ToolDispatcher:
return PythonToolProvider.execute(manifest, arguments)
raise ToolExecutionError(f"工具 {manifest.name} provider={provider} 不支持调用")
def list_mcp_tools(self, manifest: CapabilityManifest) -> List[Dict[str, Any]]:
"""MCP 能力全量工具 schema(bioflow_agent 构建 LLM 工具表用)。"""
return self._mcp.list_tool_schemas(manifest)
def reap_idle(self) -> int:
return self._mcp.reap_idle()
@@ -1,174 +0,0 @@
"""test_bioflow_agent.py —— bioflow_agent 前门三重门 + nl_triggers 解析单测。
覆盖 S1 验收要点:
- 熔断开关(QA_V1_ENABLE_BIOFLOW_AGENT)默认关 → 返回 None(走原 pipeline)
- 意图预筛(nl_triggers)各自评估,未命中返回 None
- 注册表未注册 / 会话开关关 → 返回 None
- 命中后短路返回自然语言 payload(bioflow.handled=True)
- 只装一个能力时,另一个能力的触发语不响应(隔离)
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.manifests import parse_capabilities_text # noqa: E402
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
BIOFLOW_YAML = """
capabilities:
- name: bioflow-gene-family
provider: mcp
permission: read_only
description: 基因家族分析
session_default: true
nl_triggers: [基因家族, 家族成员, 亚家族, 建树]
mcp:
url: http://127.0.0.1:8098/mcp/gene-family/
prompt_pack:
file: bioflow_gene_family_agent_rules.md
- name: bioflow-rnaseq
provider: mcp
permission: read_only
description: 转录组分析
session_default: true
nl_triggers: [转录组, RNA-seq, 差异表达, 富集]
mcp:
url: http://127.0.0.1:8098/mcp/rnaseq/
prompt_pack:
file: bioflow_rnaseq_agent_rules.md
"""
def _make_registry(tmp_path: Path) -> CapabilityRegistry:
path = tmp_path / "capabilities.yaml"
path.write_text(BIOFLOW_YAML, encoding="utf-8")
registry = CapabilityRegistry(path)
registry.load()
return registry
class FakeLLMClient:
def __init__(self, content: str = "已为您规划分析路径。") -> None:
self.content = content
def complete_raw_messages(self, system, messages, tools=None, **kwargs):
return {"content": self.content, "tool_calls": [], "finish_reason": "stop"}
class TestNlTriggers:
def test_parse_nl_triggers(self):
manifests, warnings = parse_capabilities_text(BIOFLOW_YAML)
assert warnings == []
gf = next(m for m in manifests if m.name == "bioflow-gene-family")
assert gf.nl_triggers == ["基因家族", "家族成员", "亚家族", "建树"]
assert gf.config["mcp"]["url"].endswith("/mcp/gene-family/")
assert gf.config["prompt_pack"]["file"] == "bioflow_gene_family_agent_rules.md"
def test_nl_triggers_dedupe_and_clean(self):
text = """
capabilities:
- name: t
provider: mcp
nl_triggers: [基因家族, 基因家族, "", " 亚家族 ", 123]
"""
manifests, _ = parse_capabilities_text(text)
assert manifests[0].nl_triggers == ["基因家族", "亚家族"]
class TestIntentMatch:
def test_match(self):
assert ba._intent_match("帮我做水稻SPL基因家族分析", ["基因家族"]) is True
assert ba._intent_match("转录组差异表达分析", ["转录组", "RNA-seq"]) is True
assert ba._intent_match("什么是光合作用", ["基因家族", "转录组"]) is False
class TestGateLogic:
def test_fuse_off_returns_none(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict("os.environ", {}, clear=False):
ba.os.environ.pop(ba.ENV_FUSE, None)
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
def test_no_intent_match_returns_none(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"光合作用的电子传递链是什么", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
def test_not_registered_returns_none(self, tmp_path):
empty = CapabilityRegistry(tmp_path / "empty.yaml")
empty.load()
with mock.patch.object(ba, "get_registry", return_value=empty):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
def test_session_switch_off_returns_none(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"帮我做基因家族分析",
llm_client=FakeLLMClient(),
capabilities={"bioflow-gene-family": False},
)
assert result is None
def test_isolation_only_installed_responds(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
caps = ba._candidate_capabilities("帮我做基因家族分析", None)
assert [m.name for m in caps] == ["bioflow-gene-family"]
caps = ba._candidate_capabilities("转录组差异表达", None)
assert [m.name for m in caps] == ["bioflow-rnaseq"]
def test_full_flow_short_circuits(self, tmp_path):
registry = _make_registry(tmp_path)
fake_tools = [
{
"type": "function",
"function": {
"name": "plan_gene_family",
"description": "规划基因家族路径",
"parameters": {"type": "object", "properties": {}},
},
}
]
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {"plan_gene_family": mock.ANY})):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is not None
assert result["success"] is True
assert result["answer"] == "已为您规划分析路径。"
assert result["bioflow"]["handled"] is True
assert result["bioflow"]["capabilities"] == ["bioflow-gene-family"]
def test_exception_falls_back_to_none(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.object(ba, "_candidate_capabilities", side_effect=RuntimeError("boom")):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
@@ -1,255 +0,0 @@
"""test_bioflow_agent_s2.py —— S2 对话式编排闭环单测。
覆盖:card_result 回填协议(四类卡片)、参数抽取(白名单)、多轮记忆注入、
session↔job 绑定上下文(active_job_id/chain)、job_status 主动查询。
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest # noqa: E402
from rag.qa_v1_1_73.tool_runner import ToolRunner # noqa: E402
def _manifest(name="bioflow-gene-family", url="http://x/mcp/gene-family/"):
return CapabilityManifest(name=name, provider="mcp", config={"mcp": {"url": url, "tool": name}})
def _tool_manifests():
"""覆盖四类卡片用到的工具 + job_status。"""
names = ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
"register_job_files", "fetch_source_file", "job_status")
out = {}
for t in names:
out[t] = CapabilityManifest(
name="bioflow-gene-family", provider="mcp",
config={"mcp": {"url": "http://x/mcp/gene-family/", "tool": t}},
)
return out
class FakeDispatcher:
def __init__(self, results=None):
self.results = results or {}
self.calls = []
def execute(self, manifest, arguments):
tool = manifest.config.get("mcp", {}).get("tool")
self.calls.append((tool, arguments))
return self.results.get(tool, '{"ok": true, "data": {"id": "job-1"}}')
class FakeLLM:
def __init__(self, complete_text="已收到,已为您处理。", json_payload=None):
self.complete_text = complete_text
self.json_payload = json_payload or {}
def complete(self, system, user, **kw):
return self.complete_text
def complete_json(self, system, user, **kw):
return self.json_payload
def complete_raw_messages(self, system, messages, tools=None, **kw):
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
# ---------------------------------------------------------------------------
# 参数抽取
# ---------------------------------------------------------------------------
class TestExtractParams:
def test_extract_known_and_unknown(self):
llm = FakeLLM(json_payload={"params": {"threads": 8, "padj": 0.01}, "unknown": ["bogus"]})
form_defaults = {"threads": {"default": 4}, "padj": {"default": 0.05}}
known, unknown = ba._extract_params(llm, "用 8 核,padj 用 0.01", form_defaults)
assert known == {"threads": 8, "padj": 0.01}
assert unknown == ["bogus"]
def test_non_whitelist_param_filtered(self):
llm = FakeLLM(json_payload={"params": {"evil": 1, "threads": 4}, "unknown": []})
form_defaults = {"threads": {"default": 4}}
known, unknown = ba._extract_params(llm, "x", form_defaults)
assert known == {"threads": 4}
def test_bad_json_returns_empty(self):
llm = FakeLLM(json_payload="not-a-dict")
form_defaults = {"threads": {}}
known, unknown = ba._extract_params(llm, "x", form_defaults)
assert known == {} and unknown == []
def test_empty_whitelist(self):
llm = FakeLLM()
known, unknown = ba._extract_params(llm, "x", {})
assert known == {} and unknown == []
# ---------------------------------------------------------------------------
# card_result 回填协议
# ---------------------------------------------------------------------------
class TestCardResult:
def test_path_selection_creates_job_and_extracts_job_id(self):
dispatcher = FakeDispatcher()
llm = FakeLLM()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "gene_family", "stages": ["identify", "tree"], "title": "水稻SPL"}},
llm, {"session_id": "s1"}, _tool_manifests(),
)
assert out is not None
assert out["bioflow"]["active_job_id"] == "job-1"
assert out["bioflow"]["chain"] == "gene_family"
assert dispatcher.calls[0][0] == "create_gene_family_job"
assert dispatcher.calls[0][1]["stages"] == ["identify", "tree"]
assert dispatcher.calls[0][1]["session_id"] == "s1"
def test_path_selection_missing_stages_returns_none(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "path_selection", "payload": {"chain": "gene_family"}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is None
def test_form_confirm_calls_confirm_stage(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "form_confirm",
"payload": {"job_id": "job-9", "stage": "identify", "params": {"evalue": 1e-5}}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "confirm_stage"
assert dispatcher.calls[0][1] == {"job_id": "job-9", "stage": "identify", "params": {"evalue": 1e-5}}
def test_form_confirm_uses_active_job_id_fallback(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
ba._handle_card_result(
{"card": "form_confirm", "payload": {"stage": "identify", "params": {}}},
FakeLLM(), {"active_job_id": "job-7"}, _tool_manifests(),
)
assert dispatcher.calls[0][1]["job_id"] == "job-7"
def test_file_upload_splits_remote_and_local(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "file_upload",
"payload": {"job_id": "j1", "paths": ["https://x/a.fa", "/data/b.fa"]}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
tools = [c[0] for c in dispatcher.calls]
assert "fetch_source_file" in tools and "register_job_files" in tools
def test_archive_choice_paraphrases(self):
out = ba._handle_card_result(
{"card": "archive_choice", "payload": {"job_id": "j1", "archive": True}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert out["answer"] == "已收到,已为您处理。"
def test_unknown_card_returns_none(self):
out = ba._handle_card_result({"card": "bogus", "payload": {}}, FakeLLM(), {}, _tool_manifests())
assert out is None
def test_path_selection_reuse_failure_no_stages_returns_none(self):
"""bug ②:复用失败且无显式 stages → 返回 None(不空链建任务)。"""
dispatcher = FakeDispatcher({"reuse_archive": '{"ok": false, "code": "ARCHIVE_NOT_FOUND"}'})
tm = _tool_manifests()
tm["reuse_archive"] = CapabilityManifest(
name="bioflow-gene-family", provider="mcp",
config={"mcp": {"url": "http://x", "tool": "reuse_archive"}},
)
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "gene_family", "reuse_archive": "不存在的存档"}},
FakeLLM(), {}, tm,
)
assert out is None
# 只执行了 reuse_archive,未调用 create_gene_family_job
assert [c[0] for c in dispatcher.calls] == ["reuse_archive"]
# ---------------------------------------------------------------------------
# 前门路由(card_result / bioflow 上下文)
# ---------------------------------------------------------------------------
class TestRouteCardResult:
def test_try_bioflow_agent_card_result_short_circuits(self):
registry_manifest = _manifest()
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_registered_capabilities", return_value=[registry_manifest]):
with mock.patch.object(ba, "_build_tools", return_value=([{"type": "function", "function": {"name": "confirm_stage", "description": "", "parameters": {"type": "object", "properties": {}}}}], {"confirm_stage": registry_manifest})):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba.try_bioflow_agent(
"确认", llm_client=FakeLLM(),
card_result={"card": "form_confirm", "payload": {"job_id": "j1", "stage": "s", "params": {}}},
)
assert out is not None
assert out["bioflow"]["handled"] is True
def test_try_bioflow_agent_fuse_off_ignores_card_result(self):
with mock.patch.dict("os.environ", {}, clear=False):
ba.os.environ.pop(ba.ENV_FUSE, None)
out = ba.try_bioflow_agent(
"确认", llm_client=FakeLLM(),
card_result={"card": "form_confirm", "payload": {}},
)
assert out is None
# ---------------------------------------------------------------------------
# 多轮记忆注入(ToolRunner.run history)
# ---------------------------------------------------------------------------
class TestMultiTurnHistory:
def test_history_injected_into_messages(self):
class RecordingLLM:
def __init__(self):
self.captured_messages = None
def complete_raw_messages(self, system, messages, tools=None, **kw):
self.captured_messages = list(messages)
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
from rag.qa_v1_1_73.tool_runner import make_llm_complete
llm = RecordingLLM()
runner = ToolRunner(make_llm_complete(llm), max_rounds=1)
history = [{"role": "user", "content": "上一轮问题"}, {"role": "assistant", "content": "上一轮答案"}]
runner.run("sys", "当前问题", [], {}, history=history)
msgs = llm.captured_messages
assert msgs[0] == {"role": "user", "content": "上一轮问题"}
assert msgs[1] == {"role": "assistant", "content": "上一轮答案"}
assert msgs[-1] == {"role": "user", "content": "当前问题"}
def test_history_filters_bad_roles(self):
class RecordingLLM:
def __init__(self):
self.captured_messages = None
def complete_raw_messages(self, system, messages, tools=None, **kw):
self.captured_messages = list(messages)
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
from rag.qa_v1_1_73.tool_runner import make_llm_complete
llm = RecordingLLM()
runner = ToolRunner(make_llm_complete(llm), max_rounds=1)
history = [{"role": "system", "content": "evil"}, {"role": "user", "content": ""}, {"role": "user", "content": "ok"}]
runner.run("sys", "q", [], {}, history=history)
roles = [m["role"] for m in llm.captured_messages]
assert "system" not in roles
assert llm.captured_messages[-1] == {"role": "user", "content": "q"}
@@ -1,511 +0,0 @@
"""test_bioflow_agent_s2_supplement.py —— S2 覆盖清单补强测试。
补强 dev 的 test_bioflow_agent_s2.py 未覆盖的分支:
- _chain_to_capability / _extract_job_id / _execute_tool / _paraphrase / _query_job_status
- card 四类 handler 缺必需字段回退 None、files 键、archive=False、非 dict payload
- _route_card_result:chain 过滤 / 未注册能力→None / tools 空→None
- NL 路径注入 bioflow 上下文(active_job_id/chain)
- system prompt 硬规则在位(追问 ≤3 / 不可代确认 / 工具名不外泄)
- ToolRunner.run(history) 边界(non-dict 条目 / history=None / 空内容剔除)
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest # noqa: E402
from rag.qa_v1_1_73.tool_runner import ToolRunner, make_llm_complete # noqa: E402
def _manifest(name="bioflow-gene-family", url="http://x/mcp/gene-family/"):
return CapabilityManifest(name=name, provider="mcp", config={"mcp": {"url": url, "tool": name}})
def _tool_manifests():
names = ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
"register_job_files", "fetch_source_file", "job_status",
"rerun_stage", "reuse_archive", "archive_job")
out = {}
for t in names:
out[t] = CapabilityManifest(
name="bioflow-gene-family", provider="mcp",
config={"mcp": {"url": "http://x/mcp/gene-family/", "tool": t}},
)
return out
class FakeLLM:
def __init__(self, complete_text="已收到。", json_payload=None, complete_error=None):
self.complete_text = complete_text
self.json_payload = json_payload or {}
self.complete_error = complete_error
def complete(self, system, user, **kw):
if self.complete_error:
raise self.complete_error
return self.complete_text
def complete_json(self, system, user, **kw):
return self.json_payload
def complete_raw_messages(self, system, messages, tools=None, **kw):
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
class FakeDispatcher:
def __init__(self, results=None):
self.results = results or {}
self.calls = []
def execute(self, manifest, arguments):
tool = manifest.config.get("mcp", {}).get("tool")
self.calls.append((tool, arguments))
return self.results.get(tool, '{"ok": true, "data": {"id": "job-1"}}')
# ===========================================================================
# 辅助函数
# ===========================================================================
class TestChainToCapability:
def test_known(self):
assert ba._chain_to_capability("gene_family") == "bioflow-gene-family"
assert ba._chain_to_capability("rnaseq") == "bioflow-rnaseq"
def test_unknown_and_empty(self):
assert ba._chain_to_capability("wgs") is None
assert ba._chain_to_capability("") is None
assert ba._chain_to_capability(None) is None
def test_case_and_whitespace(self):
assert ba._chain_to_capability(" Gene_Family ") == "bioflow-gene-family"
class TestExtractJobId:
def test_data_id(self):
assert ba._extract_job_id('{"ok": true, "data": {"id": "job-9"}}') == "job-9"
def test_data_job_id(self):
assert ba._extract_job_id('{"ok": true, "data": {"job_id": "job-8"}}') == "job-8"
def test_flat_id(self):
assert ba._extract_job_id('{"ok": true, "id": "job-7"}') == "job-7"
def test_non_json_and_no_id(self):
assert ba._extract_job_id("not json") is None
assert ba._extract_job_id('{"ok": false}') is None
assert ba._extract_job_id("") is None
class TestExecuteTool:
def test_unknown_tool(self):
out = ba._execute_tool(_tool_manifests(), "ghost", {})
assert "未注册" in out
def test_tool_exception_becomes_text(self):
class Boom:
def execute(self, manifest, arguments):
raise RuntimeError("boom")
with mock.patch.object(ba, "get_dispatcher", return_value=Boom()):
out = ba._execute_tool(_tool_manifests(), "job_status", {"job_id": "j"})
assert "工具执行失败" in out
class TestParaphrase:
def test_llm_success(self):
out = ba._paraphrase(FakeLLM(complete_text="已转述。"), "原文")
assert out == "已转述。"
def test_llm_exception_falls_back_to_raw(self):
out = ba._paraphrase(FakeLLM(complete_error=RuntimeError("boom")), "原文")
assert out == "原文"
def test_llm_empty_falls_back_to_raw(self):
out = ba._paraphrase(FakeLLM(complete_text=" "), "原文")
assert out == "原文"
class TestQueryJobStatus:
def test_calls_job_status_and_paraphrases(self):
with mock.patch.object(ba, "_execute_tool", return_value='{"ok": true, "data": {"status": "running"}}'):
out = ba._query_job_status("j1", _tool_manifests(), FakeLLM(complete_text="任务进行中。"))
assert out == "任务进行中。"
# ===========================================================================
# card 四类 handler 边界(缺必需字段 / files 键 / archive=False / 非 dict payload)
# ===========================================================================
class TestCardHandlersEdge:
def test_form_confirm_missing_job_and_stage_returns_none(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "form_confirm", "payload": {"params": {}}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is None
def test_file_upload_missing_job_returns_none(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "file_upload", "payload": {"paths": ["/data/a.fa"]}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is None
def test_file_upload_missing_files_returns_none(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "file_upload", "payload": {"job_id": "j1"}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is None
def test_file_upload_uses_files_key(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "file_upload",
"payload": {"job_id": "j1", "files": ["https://x/a.fa"]}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "fetch_source_file"
def test_file_upload_string_files_coerced(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "file_upload",
"payload": {"job_id": "j1", "files": "https://x/a.fa"}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "fetch_source_file"
def test_archive_choice_missing_job_returns_none(self):
out = ba._handle_card_result(
{"card": "archive_choice", "payload": {"archive": True}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is None
def test_archive_choice_skip(self):
out = ba._handle_card_result(
{"card": "archive_choice", "payload": {"job_id": "j1", "archive": False}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert out["bioflow"]["audit"]["tool_calls"] == 0 # 存档不调工具(走 REST)
def test_path_selection_rnaseq_chain(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "rnaseq", "stages": ["rnaseq_qc"]}},
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "create_rnaseq_job"
def test_card_non_dict_payload_treated_empty(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "form_confirm", "payload": "not-a-dict"},
FakeLLM(), {"active_job_id": "j1"}, _tool_manifests(),
)
assert out is None # payload 非 dict → 无 stage/job_id → None
def test_card_result_with_non_dict_is_ignored(self):
# try_bioflow_agent 对非 dict 的 card_result 不做 card 路由,走 NL 意图
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_candidate_capabilities", return_value=[]):
out = ba.try_bioflow_agent("确认", llm_client=FakeLLM(), card_result="bogus")
assert out is None
# ===========================================================================
# _route_card_result 边界
# ===========================================================================
class TestRouteCardResultEdge:
def test_chain_filters_to_capability(self):
gf = _manifest("bioflow-gene-family")
rn = _manifest("bioflow-rnaseq", "http://x/mcp/rnaseq/")
captured = {}
def fake_build(manifests):
captured["manifests"] = [m.name for m in manifests]
return ([], {})
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf, rn]):
with mock.patch.object(ba, "_build_tools", side_effect=fake_build):
ba._route_card_result(
{"card": "form_confirm", "payload": {"chain": "rnaseq"}},
FakeLLM(), None, None, None,
)
assert captured["manifests"] == ["bioflow-rnaseq"] # chain 过滤到 rnaseq 能力
def test_chain_to_unregistered_capability_returns_none(self):
gf = _manifest("bioflow-gene-family")
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
out = ba._route_card_result(
{"card": "form_confirm", "payload": {"chain": "rnaseq"}},
FakeLLM(), None, None, None,
)
assert out is None # rnaseq 未注册 → manifests 空 → None
def test_tools_empty_returns_none(self):
gf = _manifest("bioflow-gene-family")
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
with mock.patch.object(ba, "_build_tools", return_value=([], {})):
out = ba._route_card_result(
{"card": "form_confirm", "payload": {}},
FakeLLM(), None, None, None,
)
assert out is None
def test_progress_callback_invoked(self):
gf = _manifest("bioflow-gene-family")
calls = []
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
with mock.patch.object(ba, "_handle_card_result", return_value=None):
ba._route_card_result(
{"card": "form_confirm", "payload": {}},
FakeLLM(), None, None, lambda p: calls.append(p),
)
assert calls and calls[0]["stage"] == "bioflow_card"
assert calls[0]["card"] == "form_confirm"
def test_progress_callback_exception_swallowed(self):
gf = _manifest("bioflow-gene-family")
def boom(p):
raise RuntimeError("callback boom")
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
with mock.patch.object(ba, "_handle_card_result", return_value={"ok": True}):
out = ba._route_card_result(
{"card": "form_confirm", "payload": {}},
FakeLLM(), None, None, boom,
)
assert out == {"ok": True} # 回调异常被吞,不影响主流程
# ===========================================================================
# NL 路径:bioflow 上下文注入 / 卡片与 NL 语义等价
# ===========================================================================
class TestNLPathBioflowContext:
def test_nl_path_injects_bioflow_context(self, tmp_path):
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry
from rag.qa_v1_1_73.capabilities.manifests import parse_capabilities_text
yaml = """
capabilities:
- name: bioflow-gene-family
provider: mcp
permission: read_only
session_default: true
nl_triggers: [基因家族]
mcp:
url: http://127.0.0.1:8098/mcp/gene-family/
"""
(tmp_path / "c.yaml").write_text(yaml, encoding="utf-8")
registry = CapabilityRegistry(tmp_path / "c.yaml")
registry.load()
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
"description": "x",
"parameters": {"type": "object", "properties": {}}}}]
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
out = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLM(),
bioflow={"session_id": "s1", "active_job_id": "job-9", "chain": "gene_family"},
)
assert out is not None
assert out["bioflow"]["active_job_id"] == "job-9"
assert out["bioflow"]["chain"] == "gene_family"
def test_card_and_nl_share_tool_chain(self):
"""卡片操作与纯 NL 服务端语义等价:卡片 handler 只调 MCP 工具,不直连控制面。"""
# 卡片 handler 用到的工具名 = MCP 工具(job_status/confirm_stage/create_* 等)
tm = _tool_manifests()
# 四类卡片触发的工具都在 tool_manifests(即 MCP 工具表)里
for tool in ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
"register_job_files", "fetch_source_file", "job_status"):
assert tool in tm
class TestExtractParamsEdge:
def test_llm_exception_returns_empty(self):
class RaisingLLM(FakeLLM):
def complete_json(self, system, user, **kw):
raise RuntimeError("boom")
known, unknown = ba._extract_params(RaisingLLM(), "x", {"threads": {}})
assert known == {} and unknown == []
def test_unknown_as_string_coerced_to_list(self):
llm = FakeLLM(json_payload={"params": {"threads": 8}, "unknown": "bogus_param"})
known, unknown = ba._extract_params(llm, "x", {"threads": {}})
assert known == {"threads": 8}
assert unknown == ["bogus_param"]
def test_path_selection_with_params(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "gene_family", "stages": ["identify"],
"params": {"prefix": "Ft.SPL"}}},
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
)
assert dispatcher.calls[0][1]["params"] == {"prefix": "Ft.SPL"}
def test_path_selection_create_tool_missing_returns_none(self):
tm = _tool_manifests()
tm.pop("create_gene_family_job")
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "gene_family", "stages": ["identify"]}},
FakeLLM(), {"session_id": "s1"}, tm,
)
assert out is None
# ===========================================================================
# system prompt 硬规则在位
# ===========================================================================
class TestSystemPromptRules:
def test_param_retry_rule(self):
assert "最多 3 次" in ba._SYSTEM_BASE
def test_human_boundary_rule(self):
assert "不可跳过" in ba._SYSTEM_BASE
assert "不可代用户确认" in ba._SYSTEM_BASE
def test_tool_name_no_leak_rule(self):
assert "绝不向用户泄露工具名" in ba._SYSTEM_BASE
# ===========================================================================
# ToolRunner.run(history) 边界
# ===========================================================================
class TestHistoryEdge:
def _run_with_history(self, history):
captured = {}
class RecordingLLM:
def complete_raw_messages(self, system, messages, tools=None, **kw):
captured["messages"] = list(messages)
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
runner = ToolRunner(make_llm_complete(RecordingLLM()), max_rounds=1)
runner.run("sys", "当前", [], {}, history=history)
return captured["messages"]
def test_none_history(self):
msgs = self._run_with_history(None)
assert msgs == [{"role": "user", "content": "当前"}]
def test_non_dict_items_skipped(self):
msgs = self._run_with_history(["not-a-dict", {"role": "user", "content": "ok"}])
assert msgs == [{"role": "user", "content": "ok"}, {"role": "user", "content": "当前"}]
def test_empty_content_filtered(self):
msgs = self._run_with_history([{"role": "user", "content": " "}])
assert msgs == [{"role": "user", "content": "当前"}]
# ===========================================================================
# S3:card_result rerun / reuse_archive / archive_job 语义
# ===========================================================================
class TestS3CardSemantics:
def test_form_confirm_rerun_true_uses_rerun_stage(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "form_confirm",
"payload": {"job_id": "j1", "stage": "identify",
"params": {"evalue": 1e-5}, "rerun": True}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "rerun_stage"
assert out["bioflow"]["audit"]["tool_names"] == ["rerun_stage"]
def test_form_confirm_rerun_false_uses_confirm_stage(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
ba._handle_card_result(
{"card": "form_confirm",
"payload": {"job_id": "j1", "stage": "identify", "params": {}}},
FakeLLM(), {}, _tool_manifests(),
)
assert dispatcher.calls[0][0] == "confirm_stage"
def test_path_selection_reuse_archive(self):
dispatcher = FakeDispatcher({
"reuse_archive": '{"ok": true, "data": {"id": "arch-1", '
'"stage_chain": ["identify", "phylogeny"], '
'"params": {"identify": {"prefix": "Ft"}}}}',
})
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "gene_family", "reuse_archive": "SPL 鉴定"}},
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
)
assert out is not None
tools = [c[0] for c in dispatcher.calls]
assert tools[0] == "reuse_archive"
assert tools[1] == "create_gene_family_job"
create_args = dispatcher.calls[1][1]
assert create_args["archive_id"] == "arch-1"
assert create_args["stages"] == ["identify", "phylogeny"]
assert create_args["params"] == {"identify": {"prefix": "Ft"}}
assert out["bioflow"]["audit"]["tool_names"] == ["reuse_archive", "create_gene_family_job"]
def test_archive_choice_archive_true_calls_archive_job(self):
dispatcher = FakeDispatcher({"archive_job": '{"ok": true, "data": {"id": "a1"}}'})
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "archive_choice",
"payload": {"job_id": "j1", "archive": True,
"name": "SPL 存档", "description": "d"}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "archive_job"
assert dispatcher.calls[0][1] == {"job_id": "j1", "name": "SPL 存档", "description": "d"}
assert out["bioflow"]["audit"]["tool_names"] == ["archive_job"]
def test_archive_choice_archive_false_no_tool_call(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "archive_choice",
"payload": {"job_id": "j1", "archive": False}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls == [] # 跳过存档不发工具请求
assert out["bioflow"]["audit"]["tool_calls"] == 0
@@ -1,206 +0,0 @@
"""test_bioflow_agent_s4_supplement.py —— S4 覆盖清单补强测试。
覆盖:
- _card_extension_action 六种 action 分发到对应扩展工具
- 未知 action / 缺 ext_id/action 回退 None
- confirm 携带 nl_triggers/parameters/runtime_image;EXT_STATUS 降级话术(确定性)
- run 携带 params/files
- 真实 configs/capabilities.yaml 三条目(含 bioflow-extension)可解析
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest, parse_capabilities_file # noqa: E402
EXT_TOOLS = (
"create_extension", "extension_status", "build_extension", "trial_run_extension",
"validate_extension", "confirm_extension", "publish_extension", "run_extension",
"repair_extension",
)
def _ext_tool_manifests():
out = {}
for t in EXT_TOOLS:
out[t] = CapabilityManifest(
name="bioflow-extension", provider="mcp",
config={"mcp": {"url": "http://127.0.0.1:8098/mcp/extension/", "tool": t}},
)
return out
class FakeLLM:
def complete(self, system, user, **kw):
return "已收到。"
def complete_json(self, system, user, **kw):
return {}
def complete_raw_messages(self, system, messages, tools=None, **kw):
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
class FakeDispatcher:
def __init__(self, results=None):
self.results = results or {}
self.calls = []
def execute(self, manifest, arguments):
tool = manifest.config.get("mcp", {}).get("tool")
self.calls.append((tool, arguments))
return self.results.get(tool, '{"ok": true, "data": {"ext_id": "e1"}}')
# ===========================================================================
# _card_extension_action 分发
# ===========================================================================
class TestExtensionAction:
@pytest.mark.parametrize("action,expected_tool", [
("build", "build_extension"),
("trial_run", "trial_run_extension"),
("validate", "validate_extension"),
("confirm", "confirm_extension"),
("publish", "publish_extension"),
("run", "run_extension"),
("repair", "repair_extension"),
])
def test_action_dispatches_to_tool(self, action, expected_tool):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1", "action": action}},
FakeLLM(), {}, _ext_tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == expected_tool
assert dispatcher.calls[0][1] == {"ext_id": "e1"}
assert out["bioflow"]["audit"]["tool_names"] == [expected_tool]
def test_unknown_action_returns_none(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "bogus"}},
FakeLLM(), {}, _ext_tool_manifests(),
)
assert out is None
def test_missing_ext_id_or_action_returns_none(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
assert ba._handle_card_result(
{"card": "extension_action", "payload": {"action": "build"}},
FakeLLM(), {}, _ext_tool_manifests(),
) is None
assert ba._handle_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1"}},
FakeLLM(), {}, _ext_tool_manifests(),
) is None
def test_confirm_passes_triggers_params_image(self):
dispatcher = FakeDispatcher()
payload = {"ext_id": "e1", "action": "confirm",
"nl_triggers": ["扩展", "触发词"], "parameters": [{"name": "x"}],
"runtime_image": "img"}
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
ba._handle_card_result({"card": "extension_action", "payload": payload},
FakeLLM(), {}, _ext_tool_manifests())
args = dispatcher.calls[0][1]
assert args == {"ext_id": "e1", "nl_triggers": ["扩展", "触发词"],
"parameters": [{"name": "x"}], "runtime_image": "img"}
def test_confirm_status_insufficient_uses_deterministic_script(self):
# confirm 返回 EXT_STATUS(状态不足)→ 确定性降级话术(不报 500、不经 LLM)
dispatcher = FakeDispatcher({
"confirm_extension": '{"ok": false, "code": "EXT_STATUS", "message": "状态不可确认"}',
})
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "confirm"}},
FakeLLM(), {}, _ext_tool_manifests(),
)
assert out is not None
assert "还不能确认" in out["answer"]
assert "先完成" in out["answer"] # 确定性话术(非 LLM 转述)
def test_run_passes_params_and_files(self):
dispatcher = FakeDispatcher()
payload = {"ext_id": "e1", "action": "run",
"params": {"cpu": 8}, "files": ["/data/a.fa"]}
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
ba._handle_card_result({"card": "extension_action", "payload": payload},
FakeLLM(), {}, _ext_tool_manifests())
assert dispatcher.calls[0][1] == {"ext_id": "e1", "params": {"cpu": 8}, "files": ["/data/a.fa"]}
def test_extension_tool_not_in_manifests_returns_none(self):
# tool_manifests 里没有扩展工具 → 回退 None(能力未安装/未发现工具)
empty = CapabilityManifest(name="bioflow-gene-family", provider="mcp",
config={"mcp": {"url": "http://x", "tool": "job_status"}})
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "build"}},
FakeLLM(), {}, {"job_status": empty},
)
assert out is None
# ===========================================================================
# 真实 capabilities.yaml 三条目可解析
# ===========================================================================
class TestCapabilitiesThreeEntries:
def _real_yaml(self) -> Path:
return Path(__file__).resolve().parents[5] / "agent" / "qa" / "configs" / "capabilities.yaml"
def test_three_bioflow_entries(self):
manifests, warnings = parse_capabilities_file(self._real_yaml())
by_name = {m.name: m for m in manifests}
assert {"bioflow-gene-family", "bioflow-rnaseq", "bioflow-extension"} <= set(by_name)
ext = by_name["bioflow-extension"]
assert ext.provider == "mcp"
assert ext.permission == "read_only"
assert ext.nl_triggers
assert ext.config["mcp"]["url"].endswith("/mcp/extension/")
assert ext.config["prompt_pack"]["file"] == "bioflow_extension_agent_rules.md"
def test_bioflow_extension_in_capability_names(self):
assert "bioflow-extension" in ba.BIOFLOW_CAPABILITY_NAMES
assert ba.CARD_EXTENSION_ACTION == "extension_action"
assert ba.CARD_EXTENSION_ACTION in ba.CARD_TYPES
# ===========================================================================
# S5:_route_card_result 对 extension_action 只发现 extension 能力(一次 list_tools)
# ===========================================================================
class TestRouteExtensionOptimization:
def test_extension_action_discovers_only_extension_capability(self):
gf = CapabilityManifest(name="bioflow-gene-family", provider="mcp",
config={"mcp": {"url": "http://x/gene-family/"}})
rn = CapabilityManifest(name="bioflow-rnaseq", provider="mcp",
config={"mcp": {"url": "http://x/rnaseq/"}})
ext = CapabilityManifest(name="bioflow-extension", provider="mcp",
config={"mcp": {"url": "http://x/extension/"}})
captured = {}
def fake_build(manifests):
captured["names"] = [m.name for m in manifests]
return ([], {})
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf, rn, ext]):
with mock.patch.object(ba, "_build_tools", side_effect=fake_build):
ba._route_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "build"}},
FakeLLM(), None, None, None,
)
# 只把 bioflow-extension manifest 交给 _build_tools(避免对三能力各 list_tools 一次)
assert captured["names"] == ["bioflow-extension"]
@@ -1,116 +0,0 @@
"""test_bioflow_agent_s6_config.py —— S6 多配置回归(QA 单元层)。
模拟四种部署配置,断言候选能力集合 + 意图预筛结果正确(真实 E2E/金标 20 题依赖部署,
此处以 monkeypatch/env 在单元层覆盖漏判安全语义)。
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
THREE_YAML = """
capabilities:
- name: bioflow-gene-family
provider: mcp
permission: read_only
session_default: true
nl_triggers: [基因家族, 亚家族]
mcp: {url: "http://x/gene-family/"}
- name: bioflow-rnaseq
provider: mcp
permission: read_only
session_default: true
nl_triggers: [转录组, 差异表达]
mcp: {url: "http://x/rnaseq/"}
- name: bioflow-extension
provider: mcp
permission: read_only
session_default: true
nl_triggers: [自定义扩展]
mcp: {url: "http://x/extension/"}
"""
def _registry(tmp_path, yaml_text=THREE_YAML):
path = tmp_path / "caps.yaml"
path.write_text(yaml_text, encoding="utf-8")
reg = CapabilityRegistry(path)
reg.load()
return reg
class FakeLLM:
def complete_raw_messages(self, *a, **kw):
return {"content": "x", "tool_calls": [], "finish_reason": "stop"}
# ===========================================================================
# 配置 1:全关(QA_V1_ENABLE_BIOFLOW_AGENT=0 + 三能力全卸载)
# ===========================================================================
class TestConfigAllOff:
def test_fuse_off_returns_none(self, tmp_path):
reg = _registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=reg):
with mock.patch.dict(ba.os.environ, {}, clear=False):
ba.os.environ.pop(ba.ENV_FUSE, None)
out = ba.try_bioflow_agent("帮我做基因家族分析", llm_client=FakeLLM())
assert out is None # 走原 pipeline
def test_all_uninstalled_no_candidate(self, tmp_path):
reg = _registry(tmp_path, "capabilities: []\n") # 三能力全卸载
with mock.patch.object(ba, "get_registry", return_value=reg):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
assert ba._candidate_capabilities("帮我做基因家族分析", None) == []
# ===========================================================================
# 配置 2:只开基因家族
# ===========================================================================
class TestConfigGeneFamilyOnly:
def test_gene_family_hit_rnaseq_miss(self, tmp_path):
reg = _registry(tmp_path)
caps = {"bioflow-gene-family": True, "bioflow-rnaseq": False, "bioflow-extension": False}
with mock.patch.object(ba, "get_registry", return_value=reg):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
names = [m.name for m in ba._candidate_capabilities("帮我做水稻 SPL 基因家族分析", caps)]
assert names == ["bioflow-gene-family"]
assert ba._candidate_capabilities("帮我做转录组差异表达分析", caps) == []
# ===========================================================================
# 配置 3:只开转录组
# ===========================================================================
class TestConfigRnaseqOnly:
def test_rnaseq_hit_gene_family_miss(self, tmp_path):
reg = _registry(tmp_path)
caps = {"bioflow-gene-family": False, "bioflow-rnaseq": True, "bioflow-extension": False}
with mock.patch.object(ba, "get_registry", return_value=reg):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
names = [m.name for m in ba._candidate_capabilities("转录组差异表达分析", caps)]
assert names == ["bioflow-rnaseq"]
assert ba._candidate_capabilities("帮我做水稻 SPL 基因家族分析", caps) == []
# ===========================================================================
# 配置 4:全开但非生信提问
# ===========================================================================
class TestConfigAllOnNonBioflow:
def test_non_bioflow_question_no_candidate(self, tmp_path):
reg = _registry(tmp_path)
caps = {"bioflow-gene-family": True, "bioflow-rnaseq": True, "bioflow-extension": True}
with mock.patch.object(ba, "get_registry", return_value=reg):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
assert ba._candidate_capabilities("光合作用的电子传递链是什么", caps) == []
assert ba._candidate_capabilities("什么是转录因子", caps) == []
@@ -1,496 +0,0 @@
"""test_bioflow_agent_supplement.py —— S1 覆盖清单补强测试。
补强 dev 的 test_bioflow_agent.py 未覆盖的分支(对照 S1 覆盖清单):
- 熔断开关值解析(1/true/yes/on 与 0/false/off/空/缺失)
- nl_triggers 同时命中 → 两个候选(交 LLM 依 description 选)
- session_default 回退(True/False)/ registry 访问异常 / registry.get 异常
- _build_tools:MCP 工具发现(空 url 跳过 / 发现异常 / 共享连接按能力名 / 无名跳过 / bad schema 兜底)
- _build_system:prompt_pack 注入 / 去重 / 加载异常兜底
- try_bioflow_agent:tools 空→None、answer 空→None、progress_callback 触发与吞异常
- ToolRunner max_rounds=5 / bioflow_agent.MAX_ROUNDS 常量
- make_llm_complete 包装(正常 / 异常)
- 真实 configs/capabilities.yaml 两条目可解析
- bioflow_agent 不 import pipeline/answer_composer(红线静态守卫)
"""
from __future__ import annotations
import os
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.manifests import ( # noqa: E402
CapabilityManifest,
parse_capabilities_file,
)
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
BIOFLOW_YAML = """
capabilities:
- name: bioflow-gene-family
provider: mcp
permission: read_only
description: 基因家族分析
session_default: true
nl_triggers: [基因家族, 家族成员, 亚家族, 建树]
mcp:
url: http://127.0.0.1:8098/mcp/gene-family/
prompt_pack:
file: bioflow_gene_family_agent_rules.md
- name: bioflow-rnaseq
provider: mcp
permission: read_only
description: 转录组分析
session_default: true
nl_triggers: [转录组, RNA-seq, 差异表达, 富集]
mcp:
url: http://127.0.0.1:8098/mcp/rnaseq/
prompt_pack:
file: bioflow_rnaseq_agent_rules.md
"""
def _make_registry(tmp_path: Path, yaml_text: str = BIOFLOW_YAML) -> CapabilityRegistry:
path = tmp_path / "capabilities.yaml"
path.write_text(yaml_text, encoding="utf-8")
registry = CapabilityRegistry(path)
registry.load()
return registry
def _gf_manifest(**overrides) -> CapabilityManifest:
cfg = {
"mcp": {"url": "http://127.0.0.1:8098/mcp/gene-family/"},
"prompt_pack": {"file": "bioflow_gene_family_agent_rules.md"},
}
cfg.update(overrides.pop("config", {}))
return CapabilityManifest(
name="bioflow-gene-family", provider="mcp", permission="read_only",
description="基因家族分析", config=cfg, nl_triggers=["基因家族"], **overrides,
)
class FakeLLMClient:
def __init__(self, content: str = "已为您规划分析路径。") -> None:
self.content = content
def complete_raw_messages(self, system, messages, tools=None, **kwargs):
return {"content": self.content, "tool_calls": [], "finish_reason": "stop"}
# ===========================================================================
# 门 1:熔断开关值解析
# ===========================================================================
class TestEnvFuse:
@pytest.mark.parametrize("value", ["1", "true", "TRUE", "yes", "on", " On ", "YES"])
def test_truthy(self, value):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: value}):
assert ba._env_enabled() is True
@pytest.mark.parametrize("value", ["0", "false", "off", "", "no", " "])
def test_falsy(self, value):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: value}):
assert ba._env_enabled() is False
def test_missing_defaults_off(self):
with mock.patch.dict(ba.os.environ, {}, clear=False):
ba.os.environ.pop(ba.ENV_FUSE, None)
assert ba._env_enabled() is False
# ===========================================================================
# 门 3:nl_triggers 意图预筛
# ===========================================================================
class TestIntentMatchEdge:
def test_empty_triggers_never_match(self):
assert ba._intent_match("做基因家族分析", []) is False
assert ba._intent_match("做基因家族分析", [""]) is False
def test_case_insensitive_substring(self):
assert ba._intent_match("RNA-SEQ 分析", ["rna-seq"]) is True
assert ba._intent_match("ka/ks 选择压力", ["Ka/Ks"]) is True
# ===========================================================================
# 门 2 + 门 3:候选能力(registry 异常 / session_default 回退 / 同时命中)
# ===========================================================================
class TestCandidateCapabilities:
def test_registry_access_exception_returns_empty(self):
with mock.patch.object(ba, "get_registry", side_effect=RuntimeError("boom")):
assert ba._candidate_capabilities("做基因家族", None) == []
def test_registry_get_exception_skips(self):
class BadRegistry:
def get(self, name):
raise RuntimeError("boom")
with mock.patch.object(ba, "get_registry", return_value=BadRegistry()):
assert ba._candidate_capabilities("做基因家族", None) == []
def test_both_triggers_match_returns_both(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
caps = ba._candidate_capabilities("做基因家族和转录组差异表达分析", None)
assert [m.name for m in caps] == ["bioflow-gene-family", "bioflow-rnaseq"]
def test_session_default_false_skips(self, tmp_path):
yaml = BIOFLOW_YAML.replace("session_default: true", "session_default: false")
registry = _make_registry(tmp_path, yaml)
with mock.patch.object(ba, "get_registry", return_value=registry):
# 无会话开关时回退 session_default=false → 不候选
assert ba._candidate_capabilities("做基因家族分析", None) == []
# 显式会话开关 true → 候选
caps = ba._candidate_capabilities("做基因家族分析", {"bioflow-gene-family": True})
assert [m.name for m in caps] == ["bioflow-gene-family"]
def test_session_switch_true_overrides_default_false(self, tmp_path):
yaml = BIOFLOW_YAML.replace("session_default: true", "session_default: false", 1)
registry = _make_registry(tmp_path, yaml)
with mock.patch.object(ba, "get_registry", return_value=registry):
caps = ba._candidate_capabilities("做基因家族分析", {"bioflow-gene-family": True})
assert [m.name for m in caps] == ["bioflow-gene-family"]
# ===========================================================================
# _build_tools:MCP 工具发现
# ===========================================================================
class _FakeDispatcher:
def __init__(self, tools_by_name=None, exc=None):
self.tools_by_name = tools_by_name or {}
self.exc = exc
def list_mcp_tools(self, manifest):
if self.exc:
raise self.exc
return self.tools_by_name.get(manifest.name, [])
class TestBuildTools:
def test_empty_url_skipped(self):
manifest = CapabilityManifest(
name="x", provider="mcp", config={"mcp": {"url": ""}}
)
with mock.patch.object(ba, "get_dispatcher", return_value=_FakeDispatcher()):
tools, tool_manifests = ba._build_tools([manifest])
assert tools == []
assert tool_manifests == {}
def test_discovery_exception_skipped(self):
manifest = _gf_manifest()
with mock.patch.object(
ba, "get_dispatcher",
return_value=_FakeDispatcher(exc=RuntimeError("mcp down")),
):
tools, tool_manifests = ba._build_tools([manifest])
assert tools == []
assert tool_manifests == {}
def test_builds_schema_and_shared_connection(self):
dispatcher = _FakeDispatcher({
"bioflow-gene-family": [
{"name": "plan_gene_family", "description": "规划路径",
"input_schema": {"type": "object", "properties": {"message": {"type": "string"}}}},
]
})
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
tools, tool_manifests = ba._build_tools([_gf_manifest()])
assert len(tools) == 1
assert tools[0]["function"]["name"] == "plan_gene_family"
assert tools[0]["function"]["description"] == "规划路径"
# 共享连接:manifest.name 用能力名,mcp.tool 指向真实 server 工具名
tm = tool_manifests["plan_gene_family"]
assert tm.name == "bioflow-gene-family"
assert tm.config["mcp"]["tool"] == "plan_gene_family"
assert tm.config["mcp"]["url"].endswith("/mcp/gene-family/")
def test_skips_nameless_and_bad_schema(self):
dispatcher = _FakeDispatcher({
"bioflow-gene-family": [
{"name": "", "description": "无名", "input_schema": {}},
{"name": "job_status", "description": "查状态", "input_schema": "not-a-dict"},
]
})
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
tools, tool_manifests = ba._build_tools([_gf_manifest()])
assert [t["function"]["name"] for t in tools] == ["job_status"]
assert tools[0]["function"]["parameters"] == {"type": "object", "properties": {}}
def test_shared_tool_deduped_across_capabilities(self):
"""两能力同时命中:共享工具去重(只保留首个能力),专有工具各自保留。
覆盖 bug ① 修复:job_status 等 12 个共享工具两端点语义一致,按名去重后
LLM tools 列表无重名,且 job_status 路由到首个能力(gene-family)。
"""
dispatcher = _FakeDispatcher({
"bioflow-gene-family": [
{"name": "plan_gene_family", "description": "规划基因家族", "input_schema": {}},
{"name": "job_status", "description": "查状态", "input_schema": {}},
{"name": "confirm_stage", "description": "确认参数", "input_schema": {}},
],
"bioflow-rnaseq": [
{"name": "plan_rnaseq", "description": "规划转录组", "input_schema": {}},
{"name": "job_status", "description": "查状态", "input_schema": {}},
{"name": "confirm_stage", "description": "确认参数", "input_schema": {}},
],
})
rn = CapabilityManifest(
name="bioflow-rnaseq", provider="mcp", permission="read_only",
description="转录组分析", nl_triggers=["转录组"],
config={"mcp": {"url": "http://127.0.0.1:8098/mcp/rnaseq/"}},
)
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
tools, tool_manifests = ba._build_tools([_gf_manifest(), rn])
names = [t["function"]["name"] for t in tools]
# 专有工具各保留 + 共享工具只保留一份,且无重名
assert names == ["plan_gene_family", "job_status", "confirm_stage", "plan_rnaseq"]
assert len(names) == len(set(names))
# 共享工具路由到首个能力(gene-family)
assert tool_manifests["job_status"].name == "bioflow-gene-family"
assert tool_manifests["job_status"].config["mcp"]["url"].endswith("/mcp/gene-family/")
assert tool_manifests["confirm_stage"].name == "bioflow-gene-family"
# 专有工具各自路由到自身能力
assert tool_manifests["plan_rnaseq"].name == "bioflow-rnaseq"
assert tool_manifests["plan_rnaseq"].config["mcp"]["url"].endswith("/mcp/rnaseq/")
assert tool_manifests["plan_gene_family"].name == "bioflow-gene-family"
# ===========================================================================
# _build_system:prompt_pack 注入
# ===========================================================================
class TestBuildSystem:
def test_appends_prompt_pack(self):
manifest = _gf_manifest()
with mock.patch.object(ba.PromptPackProvider, "load", return_value="基因家族规则正文"):
system = ba._build_system([manifest])
assert ba._SYSTEM_BASE in system
assert "基因家族规则正文" in system
def test_dedup_same_file_once(self):
m1 = _gf_manifest()
m2 = _gf_manifest() # 同 file
calls = []
with mock.patch.object(ba.PromptPackProvider, "load", side_effect=lambda m, root: calls.append(m) or "RULES"):
system = ba._build_system([m1, m2])
assert len(calls) == 1 # 去重后只加载一次
assert system.count("RULES") == 1
def test_empty_file_skipped(self):
manifest = CapabilityManifest(name="x", provider="mcp", config={"prompt_pack": {"file": ""}})
system = ba._build_system([manifest])
assert system == ba._SYSTEM_BASE
def test_load_exception_falls_back(self):
manifest = _gf_manifest()
with mock.patch.object(ba.PromptPackProvider, "load", side_effect=RuntimeError("boom")):
system = ba._build_system([manifest])
assert ba._SYSTEM_BASE in system # 不炸,只用基础提示
# ===========================================================================
# try_bioflow_agent:tools 空 / answer 空 / progress_callback
# ===========================================================================
class TestFullFlowEdge:
def test_tools_empty_returns_none(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_build_tools", return_value=([], {})):
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
def test_answer_empty_returns_none(self, tmp_path):
registry = _make_registry(tmp_path)
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
"description": "x",
"parameters": {"type": "object", "properties": {}}}}]
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
FakeRunner.return_value.run.return_value = ("", {"rounds": 1})
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
def test_progress_callback_invoked(self, tmp_path):
registry = _make_registry(tmp_path)
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
"description": "x",
"parameters": {"type": "object", "properties": {}}}}]
calls = []
def cb(payload):
calls.append(payload)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(),
capabilities=None, progress_callback=cb,
)
assert result is not None
assert calls and calls[0]["stage"] == "bioflow_agent_started"
assert calls[0]["capabilities"] == ["bioflow-gene-family"]
assert calls[0]["tool_count"] == 1
def test_progress_callback_exception_swallowed(self, tmp_path):
registry = _make_registry(tmp_path)
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
"description": "x",
"parameters": {"type": "object", "properties": {}}}}]
def cb(payload):
raise RuntimeError("callback boom")
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(),
capabilities=None, progress_callback=cb,
)
assert result is not None # 回调异常被吞,不影响主流程
assert result["answer"] == "已答复"
# ===========================================================================
# ToolRunner max_rounds=5 / MAX_ROUNDS 常量
# ===========================================================================
class TestMaxRounds:
def test_max_rounds_constant_is_5(self):
assert ba.MAX_ROUNDS == 5
def test_tool_runner_respects_max_rounds_5(self):
from rag.qa_v1_1_73.tool_runner import ToolRunner
class LLM:
def __call__(self, system, messages, tools):
return "", [{"id": "c", "name": "echo", "arguments": "{}"}], None
class Dispatcher:
def execute(self, manifest, arguments):
return "ok"
manifest = CapabilityManifest(name="echo", provider="python")
tools = [{"type": "function", "function": {"name": "echo", "description": "e",
"parameters": {"type": "object", "properties": {}}}}]
runner = ToolRunner(LLM(), dispatcher=Dispatcher(), max_rounds=5)
content, audit = runner.run("sys", "user", tools, {"echo": manifest})
assert audit["rounds"] == 5
assert audit["tool_calls"] == 5
# ===========================================================================
# make_llm_complete 包装
# ===========================================================================
class TestMakeLLMComplete:
def test_normal(self):
from rag.qa_v1_1_73.tool_runner import make_llm_complete
class Client:
def complete_raw_messages(self, system, messages, tools=None, **kw):
return {"content": "hi", "tool_calls": [{"name": "x"}], "finish_reason": "stop"}
complete = make_llm_complete(Client())
content, tool_calls, error = complete("sys", [{"role": "user", "content": "q"}], [])
assert content == "hi"
assert tool_calls == [{"name": "x"}]
assert error is None
def test_exception_becomes_error(self):
from rag.qa_v1_1_73.tool_runner import make_llm_complete
class Client:
def complete_raw_messages(self, *a, **kw):
raise RuntimeError("llm down")
complete = make_llm_complete(Client())
content, tool_calls, error = complete("sys", [], [])
assert content == ""
assert tool_calls == []
assert "RuntimeError" in error
# ===========================================================================
# 真实 configs/capabilities.yaml 两条目可解析
# ===========================================================================
class TestRealCapabilitiesYaml:
def _real_yaml(self) -> Path:
repo_root = Path(__file__).resolve().parents[5]
return repo_root / "agent" / "qa" / "configs" / "capabilities.yaml"
def test_both_bioflow_entries_parse(self):
manifests, warnings = parse_capabilities_file(self._real_yaml())
by_name = {m.name: m for m in manifests}
assert "bioflow-gene-family" in by_name
assert "bioflow-rnaseq" in by_name
gf = by_name["bioflow-gene-family"]
assert gf.provider == "mcp"
assert gf.permission == "read_only"
assert gf.nl_triggers
assert gf.config["mcp"]["url"].endswith("/mcp/gene-family/")
assert gf.config["prompt_pack"]["file"] == "bioflow_gene_family_agent_rules.md"
rn = by_name["bioflow-rnaseq"]
assert rn.permission == "read_only"
assert rn.nl_triggers
assert rn.config["mcp"]["url"].endswith("/mcp/rnaseq/")
assert rn.config["prompt_pack"]["file"] == "bioflow_rnaseq_agent_rules.md"
def test_nl_triggers_cover_expected_words(self):
manifests, _ = parse_capabilities_file(self._real_yaml())
gf = next(m for m in manifests if m.name == "bioflow-gene-family")
rn = next(m for m in manifests if m.name == "bioflow-rnaseq")
# 关键触发词在位(意图预筛依赖)
assert "基因家族" in gf.nl_triggers
assert "转录组" in rn.nl_triggers
# ===========================================================================
# 红线静态守卫:bioflow_agent 不依赖 pipeline / answer_composer
# ===========================================================================
class TestRedLineNoPipelineDep:
def test_no_pipeline_or_answer_composer_import(self):
import ast
src = Path(ba.__file__).read_text(encoding="utf-8")
tree = ast.parse(src)
imported = []
for node in tree.body:
if isinstance(node, ast.Import):
imported += [a.name for a in node.names]
elif isinstance(node, ast.ImportFrom):
imported.append(node.module or "")
assert not any("pipeline" in m or "answer_composer" in m for m in imported)
def test_never_raises(self, tmp_path):
# 任何异常都返回 None/正常 payload,绝不外抛
with mock.patch.object(ba, "get_registry", side_effect=RuntimeError("boom")):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent("做基因家族", llm_client=FakeLLMClient())
assert result is None
+1 -14
View File
@@ -51,13 +51,9 @@ class ToolRunner:
user: str,
tools: List[Dict[str, Any]],
manifests: Dict[str, CapabilityManifest],
history: Optional[List[Dict[str, str]]] = None,
) -> Tuple[str, Dict[str, Any]]:
"""执行工具调用循环。返回 (content, audit)。
history:多轮记忆(S2),List[{role: user|assistant, content}],旧→新,
不含当前这轮提问;注入为 messages 前置轮次(作为上下文,不参与工具结果回填)。
audit: {"rounds": n, "tool_calls": k, "used_json_fallback": bool,
"tool_names": [...], "errors": [...]}
"""
@@ -68,16 +64,7 @@ class ToolRunner:
"tool_names": [],
"errors": [],
}
messages: List[Dict[str, Any]] = []
for h in history or []:
if not isinstance(h, dict):
continue
role = str(h.get("role") or "").strip()
content = str(h.get("content") or "").strip()
if role not in {"user", "assistant"} or not content:
continue
messages.append({"role": role, "content": content})
messages.append({"role": "user", "content": user})
messages: List[Dict[str, Any]] = [{"role": "user", "content": user}]
system_text = system
for round_index in range(1, self.max_rounds + 1):
audit["rounds"] = round_index
+9 -30
View File
@@ -2,8 +2,7 @@
- 独立部署树 bioflow_service/(FR-BIO-028:独立进程 8098 + 独立 SQLite)。
- lifespan:init_db(幂等迁移)→ init_context(kernel/catalog/planner/runner)
→ recover(服务重启恢复 in-flight 续跑,FR-BIO-031)→ MCP session manager
(anyio task group,streamable-http 处理请求必需)→ shutdown 清理。
→ recover(服务重启恢复 in-flight 续跑,FR-BIO-031)→ shutdown 清理。
- 端点:health(/api/bioflow/health,前端探活 FR-BIO-025)。
- 不依赖 BFF/生产服务(8090/8091 等绝不探测/连接)。
"""
@@ -18,18 +17,11 @@ from fastapi.middleware.cors import CORSMiddleware
from .config import get_settings
from .context import context, init_context, shutdown_context
from .db import close_db, init_db
from .mcp_server import extension_mcp, gene_family_mcp, rnaseq_mcp
from .routers import bioarchives, bioflow, biojobs, extension
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# 三个 MCP streamable-http app(调用即创建 session_manager,供 lifespan 进入 run())。
# 端点地址:/mcp/gene-family、/mcp/rnaseq、/mcp/extension(streamable_http_path="/" + 挂载前缀)。
_gene_family_mcp_app = gene_family_mcp.streamable_http_app(streamable_http_path="/")
_rnaseq_mcp_app = rnaseq_mcp.streamable_http_app(streamable_http_path="/")
_extension_mcp_app = extension_mcp.streamable_http_app(streamable_http_path="/")
def create_app() -> FastAPI:
settings = get_settings()
@@ -68,21 +60,14 @@ def create_app() -> FastAPI:
logger.info("main: 重启恢复 %d 个 in-flight 任务续跑", len(recovered))
except Exception:
logger.exception("main: 重启恢复扫描失败(任务不重跑,等待后续操作)")
# MCP session manager 的 task group 必须在服务运行期内保持活跃
# (streamable-http 处理请求时要求已初始化)。Starlette 不会自动运行
# 子 Mount 的 lifespan,故在此显式进入三个 MCP 的 run()。
async with gene_family_mcp.session_manager.run():
async with rnaseq_mcp.session_manager.run():
async with extension_mcp.session_manager.run():
try:
yield
finally:
await shutdown_context()
try:
await close_db()
except Exception:
pass
try:
yield
finally:
await shutdown_context()
try:
await close_db()
except Exception:
pass
app.router.lifespan_context = _lifespan
@@ -90,12 +75,6 @@ def create_app() -> FastAPI:
app.include_router(biojobs.router)
app.include_router(bioarchives.router)
app.include_router(extension.router)
# 生信 MCP 能力外挂(mcp 2.x:MCPServer.streamable_http_app)。
# 三个独立 streamable-http 端点,各自装卸(REST 面不受影响)。
app.mount("/mcp/gene-family", _gene_family_mcp_app)
app.mount("/mcp/rnaseq", _rnaseq_mcp_app)
app.mount("/mcp/extension", _extension_mcp_app)
return app
-864
View File
@@ -1,864 +0,0 @@
"""app/mcp_server.py —— 生信 MCP server(两个 streamable-http 端点)。
- `/mcp/gene-family`:基因家族能力单元(plan/create/ask + 共享控制组 + 文件组)
- `/mcp/rnaseq`:转录组能力单元(plan/create/generate_samples_yaml/ask + 共享控制组 + 文件组)
设计约束(实施方案 S0):
- 工具只承诺「已受理」,执行进度/日志由 SSE 事件总线直推,不走 MCP。
- 每个能力单元只接受本链阶段(create 跨链拒绝 CROSS_CHAIN)。
- 人工确认边界(subfamily_confirm / rnaseq_de 实验设计确认)在 job_status 的
pending_actions 里标 requires_human=true,工具不提供绕过。
- 白名单沿用 runner._allowed_param_names(单一事实源)。
- 工具名不外泄由 QA agent(S1)负责,本模块只负责工具语义。
mcp 2.x 注意:MCPServer(非 FastMCP,来自 mcp.server.mcpserver);挂载用
`mcp.streamable_http_app()`。本文件位于 app/ 层(非 services/),可用 pydantic(mcp
自身也依赖 pydantic 构造 schema)。
"""
from __future__ import annotations
import logging
import re
import uuid
from pathlib import Path
from typing import Any
from .config import get_settings
from .context import load_record
from .services.bioflow import extensions as ext_mod
from .services.bioflow import store
from .services.bioflow.archive import archive_from_job, normalize_archive_name
from .services.bioflow.catalog import GENE_FAMILY_CHAIN
from .services.bioflow.file_transform import (
inspect_file as _ft_inspect_file,
transform_file as _ft_transform_file,
validate_format,
)
from .services.bioflow.reports import locate_report
from .services.bioflow.runner import RunnerError, ensure_samples_yaml, llm_sample_fallback
from .services.bioflow.source_fetcher import FetchError, fetch_source
logger = logging.getLogger(__name__)
# ---- 能力单元阶段白名单 ----
GENE_FAMILY_STAGES = set(GENE_FAMILY_CHAIN)
RNASEQ_STAGES = {
"rnaseq_qc", "rnaseq_align", "rnaseq_quant", "rnaseq_sample_qc", "rnaseq_de",
"rnaseq_wgcna", "rnaseq_go_enrich", "rnaseq_kegg_enrich", "rnaseq_enrich",
"rnaseq_fgsea", "rnaseq_candidate_family", "rnaseq_report",
"annotation", "expression",
}
# 人工确认边界:这些阶段 waiting_confirm 时 requires_human=true,工具/QA 不可代确认
HUMAN_CONFIRM_STAGES = {"subfamily_confirm", "rnaseq_de"}
# ---------------------------------------------------------------------------
# context 访问(动态读模块全局,测试 monkeypatch 兼容)
# ---------------------------------------------------------------------------
def _ctx():
from . import context as context_mod
return context_mod.context
def _job_detail(record: Any) -> dict[str, Any]:
stages = []
for stage in record.stage_chain:
st = record._stage_states.get(stage, {})
stages.append({
"stage": stage,
"status": st.get("status", "waiting"),
"params": st.get("params") or {},
"attempt": int(st.get("attempt") or 0),
"manifest_path": st.get("manifest_path"),
"log_tail": st.get("log_tail") or "",
"error": st.get("error"),
})
return {
"id": record.id,
"session_id": record.session_id,
"title": record.title,
"status": record.status,
"stages": stages,
"params": record.params,
"auto_continue": record.auto_continue,
"archive_id": record.archive_id,
"progress": record.progress(),
}
# ---------------------------------------------------------------------------
# pending_actions 派生(job_status 工具返回给 QA agent 的下一步动作)
# ---------------------------------------------------------------------------
def derive_pending_actions(record: Any, runner: Any) -> list[dict[str, Any]]:
"""由任务/阶段状态派生待办动作(QA agent 据此引导用户,人工边界不可代确认)。
按阶段链顺序找第一个「卡住」的环节,返回动作清单(通常 0-1 条)。
"""
if record.status == "paused":
return [{"action": "resume", "stage": None, "requires_human": False,
"message": "任务已暂停,可恢复"}]
if record.status == "done":
return [{"action": "archive", "stage": None, "requires_human": False,
"message": "任务已完成,可存档复用"}]
if record.status == "archived":
return [{"action": "none", "stage": None, "requires_human": False,
"message": "任务已存档"}]
for stage in record.stage_chain:
status = record._stage_states.get(stage, {}).get("status", "waiting")
if status == "await_files":
return [{"action": "provide_files", "stage": stage, "requires_human": False,
"message": f"阶段 {stage} 等待输入文件"}]
if status == "waiting_confirm":
human = stage in HUMAN_CONFIRM_STAGES
return [{"action": "confirm", "stage": stage, "requires_human": human,
"form_defaults": runner._form_defaults(stage, record),
"message": (f"阶段 {stage} 需人工确认(不可代确认)" if human
else f"阶段 {stage} 等待参数确认")}]
if status == "failed":
return [{"action": "rerun_or_skip", "stage": stage, "requires_human": False,
"message": f"阶段 {stage} 失败,可重跑或跳过"}]
# waiting/running/queued/done/skipped → 继续
return []
# ---------------------------------------------------------------------------
# 工具实现(命名 async 函数,带类型提示,供两个 MCPServer 注册)
# ---------------------------------------------------------------------------
async def _plan(message: str, allowed_stages: set[str]) -> dict[str, Any]:
from .services.bioflow.planner import PlanCandidate, PlannerError, PlanResult
ctx = _ctx()
if ctx.planner is None:
return {"ok": False, "code": "NOT_READY", "message": "规划器未就绪"}
try:
plan = await ctx.planner.plan(message)
except PlannerError as exc:
fallback = [s for s in ("identify", "rnaseq_qc") if s in allowed_stages and ctx.catalog.has(s)]
plan = PlanResult(candidates=[PlanCandidate(purpose="内置默认流程", stages=fallback)],
source="llm_only", message=exc.message)
candidates = []
for c in plan.candidates:
stages = [s for s in c.stages if s in allowed_stages]
if stages:
candidates.append({"purpose": c.purpose, "stages": stages})
return {
"ok": True,
"candidates": candidates,
"source": plan.source,
"reuse_archive": plan.reuse_archive_id,
"reuse_name": plan.reuse_name,
"fallback": plan.message or None,
}
async def _create_job(session_id: str, title: str, stages: list[str],
params: dict[str, Any] | None, auto_continue: bool,
archive_id: str | None, allowed_stages: set[str]) -> dict[str, Any]:
ctx = _ctx()
unknown = [s for s in stages if s not in allowed_stages]
if unknown:
return {"ok": False, "code": "CROSS_CHAIN",
"message": f"阶段不属于该能力单元: {', '.join(unknown)}"}
try:
record = await ctx.runner.create_job(
session_id, title or "生信任务", stages, params or {},
auto_continue=auto_continue, archive_id=archive_id,
)
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
ctx.records[record.id] = record
return {"ok": True, "data": _job_detail(record)}
async def _job_status(job_id: str) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
detail = _job_detail(record)
detail["pending_actions"] = derive_pending_actions(record, ctx.runner)
return {"ok": True, "data": detail}
async def _confirm_stage(job_id: str, stage: str, params: dict[str, Any] | None) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
if stage not in record.stage_chain:
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
try:
await ctx.runner.confirm_stage(record, stage, params or {})
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
"status": ctx.runner.stage_status(record, stage)}}
async def _rerun_stage(job_id: str, stage: str, params: dict[str, Any] | None) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
if stage not in record.stage_chain:
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
try:
await ctx.runner.rerun_stage(record, stage, params)
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
"status": ctx.runner.stage_status(record, stage)}}
async def _skip_stage(job_id: str, stage: str) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
if stage not in record.stage_chain:
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
try:
await ctx.runner.skip_stage(record, stage)
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
"status": ctx.runner.stage_status(record, stage)}}
async def _pause_job(job_id: str) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
try:
await ctx.runner.pause_job(record)
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
return {"ok": True, "data": {"job_id": job_id, "status": record.status}}
async def _resume_job(job_id: str) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
try:
await ctx.runner.resume_job(record)
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
return {"ok": True, "data": {"job_id": job_id, "status": record.status}}
async def _archive_job(job_id: str, name: str, description: str,
include_intermediate: bool) -> dict[str, Any]:
"""任务存档:打包任务目录 + 落 bio_archives 行(复用 archive_from_job + store.insert_archive)。"""
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
archive_name = normalize_archive_name(name)
existing = await store.find_archive_by_name(ctx.db, archive_name)
if existing is not None:
return {"ok": False, "code": "ARCHIVE_EXISTS", "message": f"归档名已存在: {archive_name}"}
try:
report_path = locate_report(record.workspace_dir)
result = await archive_from_job(
{
"id": record.id, "session_id": record.session_id,
"title": record.title, "stage_chain": record.stage_chain,
"params": record.params,
},
record.job_dir,
get_settings().archives_dir_abs,
name=archive_name,
description=description or "",
include_intermediate=bool(include_intermediate),
report_artifact=str(report_path) if report_path else None,
)
await store.insert_archive(ctx.db, {
"id": result.id,
"job_id": record.id,
"name": archive_name,
"description": description or "",
"stage_chain": record.stage_chain,
"params": record.params,
"report_artifact": result.report_artifact,
"disk_path": result.disk_path,
"size_bytes": result.size_bytes,
})
except Exception as exc: # noqa: BLE001
return {"ok": False, "code": "ARCHIVE_FAILED",
"message": f"存档失败: {type(exc).__name__}: {exc}"}
ctx.events.publish("job_archived", {"job_id": record.id, "archive_id": result.id})
return {"ok": True, "data": {
"id": result.id,
"job_id": record.id,
"name": archive_name,
"description": description or "",
"disk_path": result.disk_path,
"size_bytes": result.size_bytes,
"report_artifact": result.report_artifact,
}}
# ---------------------------------------------------------------------------
# 扩展能力单元(第三个能力单元 bioflow-extension):复用 extensions.py 状态机
# ---------------------------------------------------------------------------
_EXT_VIEW_KEYS = (
"ext_id", "display_name", "status", "command_template", "original_command",
"runtime_image", "requirements", "parameters", "input_roles", "nl_triggers",
"analysis_type", "golden", "repair_count", "trial", "validation", "last_build",
"created_at", "updated_at",
)
def _ext_view(ext: dict[str, Any]) -> dict[str, Any]:
"""扩展精简视图(剔除 source_files 等超长字段,供 MCP 工具结果返回)。"""
return {k: ext.get(k) for k in _EXT_VIEW_KEYS if k in ext}
def _file_items(files: Any) -> list[tuple[str, bytes]]:
"""把 MCP 入参 [{path, content|data}] 转为 [(rel_path, bytes)]。"""
out: list[tuple[str, bytes]] = []
if not isinstance(files, list):
return out
for f in files:
if not isinstance(f, dict):
continue
path = str(f.get("path") or f.get("name") or "").strip()
if not path:
continue
content = f.get("content")
if content is None:
content = f.get("data") or ""
if isinstance(content, (bytes, bytearray)):
blob = bytes(content)
else:
blob = str(content).encode("utf-8")
out.append((path, blob))
return out
def _ext_err(exc: Exception) -> str:
if isinstance(exc, ext_mod.ExtensionNotFoundError):
return "NOT_FOUND"
if isinstance(exc, ext_mod.ExtensionStatusError):
return "EXT_STATUS"
if isinstance(exc, ext_mod.ExtensionBuildError):
return "EXT_BUILD"
if isinstance(exc, ext_mod.ExtensionRunError):
return "EXT_RUN"
return "EXT_ERROR"
async def _create_extension(display_name: str, command: str, runtime_image: str,
code_files: Any, data_files: Any) -> dict[str, Any]:
code = _file_items(code_files)
if not code:
return {"ok": False, "code": "VALIDATION", "message": "扩展代码包不能为空"}
try:
ext = ext_mod.get_store().create_extension(
display_name=display_name, command=command, runtime_image=runtime_image or "",
code_files=code, data_files=_file_items(data_files),
)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _extension_status(ext_id: str) -> dict[str, Any]:
try:
ext = ext_mod.get_store().load_extension(ext_id)
except ext_mod.ExtensionNotFoundError:
return {"ok": False, "code": "NOT_FOUND", "message": "扩展不存在"}
return {"ok": True, "data": _ext_view(ext) if ext else None}
async def _build_extension(ext_id: str) -> dict[str, Any]:
try:
ext = await ext_mod.get_store().build_extension(ext_id)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _trial_run_extension(ext_id: str) -> dict[str, Any]:
try:
ext = await ext_mod.get_store().trial_run(ext_id)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _validate_extension(ext_id: str) -> dict[str, Any]:
try:
ext = ext_mod.get_store().validate(ext_id)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _confirm_extension(ext_id: str, nl_triggers: Any, parameters: Any,
runtime_image: str) -> dict[str, Any]:
triggers = [str(x).strip() for x in nl_triggers if str(x).strip()] if isinstance(nl_triggers, list) else None
params = parameters if isinstance(parameters, list) else None
try:
ext = ext_mod.get_store().confirm(
ext_id, nl_triggers=triggers, parameters=params,
runtime_image=runtime_image or None,
)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _publish_extension(ext_id: str) -> dict[str, Any]:
try:
ext = ext_mod.get_store().publish(ext_id)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _run_extension(ext_id: str, params: Any, files: Any) -> dict[str, Any]:
try:
result = await ext_mod.get_store().run(
ext_id,
params=params if isinstance(params, dict) else None,
files=[str(f) for f in files] if isinstance(files, list) else None,
)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": result}
async def _repair_extension(ext_id: str) -> dict[str, Any]:
try:
ext = await ext_mod.get_store().repair(ext_id)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
# ---- 文件登记(复刻 biojobs._register_path 核心,无 FastAPI 依赖)----
def _blocked_reason(path: Path, settings: Any) -> str | None:
s = str(path)
for prefix in settings.sensitive_path_prefixes:
if s == prefix or s.startswith(prefix + "/"):
return f"敏感路径前缀 {prefix}"
for pat in settings.sensitive_filename_patterns:
if re.match(pat, path.name):
return f"敏感文件名 {path.name}"
return None
async def _register_job_files(job_id: str, paths: list[str]) -> dict[str, Any]:
"""登记服务器/局域网路径(单文件或目录展开)到任务 input。"""
ctx = _ctx()
settings = get_settings()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
input_dir = record.input_dir
input_dir.mkdir(parents=True, exist_ok=True)
items: list[dict[str, Any]] = []
registered: list[str] = []
for raw in paths:
p = Path(raw).expanduser().resolve()
if not p.exists():
return {"ok": False, "code": "PATH_NOT_EXIST", "message": f"路径不存在: {raw}"}
blocked = _blocked_reason(p, settings)
if blocked:
return {"ok": False, "code": "PATH_NOT_ALLOWED",
"message": f"路径不允许: {raw}({blocked})"}
files: list[Path] = []
if p.is_file():
files = [p]
else:
ext_set = settings.upload_ext_set
for f in sorted(p.rglob("*")):
if f.is_file() and (not ext_set or f.suffix.lower() in ext_set) and not _blocked_reason(f, settings):
files.append(f)
if not files:
return {"ok": False, "code": "VALIDATION", "message": f"目录内无可登记文件: {raw}"}
for f in files:
link = input_dir / f.name
try:
if not link.exists():
link.symlink_to(f)
except OSError as exc: # noqa: BLE001
logger.warning("mcp register symlink 失败: %s -> %s (%s)", f, link, exc)
items.append({"file_id": f"{record.id}:{f.name}", "name": f.name,
"size": f.stat().st_size, "path": str(f), "source": "server_path"})
registered.append(str(f))
# rnaseq 系:samples.yaml 生成(描述文件优先 + 实况分析;规则不可归纳走 LLM 兜底)
try:
source_roots = [str(Path(x).parent) for x in paths if Path(x).exists()]
extra_mounts = ensure_samples_yaml(input_dir, source_roots=[Path(r) for r in source_roots])
if not extra_mounts:
try:
await llm_sample_fallback(input_dir, [Path(r) for r in source_roots])
except Exception as exc: # noqa: BLE001
logger.warning("samples.yaml LLM 兜底失败: %s", exc)
if extra_mounts:
record.registered_server_paths = list(
dict.fromkeys((getattr(record, "registered_server_paths", None) or [])
+ [str(Path(m)) for m in extra_mounts]))
except Exception as exc: # noqa: BLE001
logger.warning("samples.yaml 分析失败: %s", exc)
if registered:
import json
record.registered_server_paths = list(
dict.fromkeys((getattr(record, "registered_server_paths", None) or []) + registered)
)
(record.job_dir / "server_registered.json").write_text(
json.dumps({"registered_server_paths": record.registered_server_paths},
ensure_ascii=False), encoding="utf-8")
# 自动续跑(await_inputs 任务重推;非等待态 no-op)
await ctx.runner.retry_after_files(record)
return {"ok": True, "data": {"items": items, "total": len(items)}}
async def _ask_question(question: str) -> dict[str, Any]:
ctx = _ctx()
hits: list[dict[str, Any]] = []
if ctx.flow_store is not None:
try:
hits = ctx.flow_store.search(question, top_k=8).get("items") or []
except Exception: # noqa: BLE001
hits = []
citations = []
for h in hits:
citations.append({
"source": h.get("source") or h.get("doc_kind") or "",
"title_path": h.get("title_path") or "",
"path": h.get("path") or "",
"text": str(h.get("text") or "")[:1200],
})
return {"ok": True, "citations": citations, "degraded": not citations}
async def _list_archives(page: int, page_size: int) -> dict[str, Any]:
ctx = _ctx()
items, total = await store.list_archives(ctx.db, page=page, page_size=page_size)
return {"ok": True, "items": items, "total": total, "page": page, "page_size": page_size}
async def _reuse_archive(name: str) -> dict[str, Any]:
ctx = _ctx()
arch = await store.find_archive_by_name(ctx.db, (name or "").strip())
if arch is None:
return {"ok": False, "code": "ARCHIVE_NOT_FOUND", "message": f"未找到匹配存档: {name}"}
return {"ok": True, "data": {
"id": arch["id"], "name": arch["name"], "stage_chain": arch["stage_chain"],
"params": arch["params"], "description": arch["description"],
}}
async def _fetch_source_file(job_id: str | None, source: str,
credentials: dict[str, Any] | None) -> dict[str, Any]:
settings = get_settings()
dest_dir: Path
if job_id:
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
dest_dir = record.input_dir
else:
dest_dir = Path(settings.jobs_dir_abs) / "_incoming" / uuid.uuid4().hex[:8]
try:
result = await fetch_source(source, dest_dir, credentials=credentials)
except FetchError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
result["job_id"] = job_id
return result
async def _inspect_file(path: str) -> dict[str, Any]:
return _ft_inspect_file(path)
async def _transform_file(src_path: str, target_format: str,
column_map: dict[str, str] | None,
dest_path: str | None) -> dict[str, Any]:
from .services.bioflow.file_transform import TransformError
from .services.bioflow.llm import complete_json
async def llm_adapter(user: str, system: str) -> Any:
return await complete_json(system, user, temperature=0.2, max_tokens=2000)
try:
return await _ft_transform_file(
src_path, target_format, column_map=column_map, dest_path=dest_path, llm=llm_adapter,
)
except TransformError as exc:
return {"ok": False, "code": exc.code, "message": exc.message, "details": exc.details}
async def _generate_samples_yaml(job_id: str, sample_description: str) -> dict[str, Any]:
import yaml
from .services.bioflow.llm import complete_json
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
system = (
"你是 RNA-seq 样本配置助手。根据用户的自然语言描述,输出 samples.yaml 结构 JSON:"
'{"samples": {"样本名": {"r1": "fastq路径", "r2": "fastq路径", "condition": "...", ...}}}。'
"只输出 JSON。"
)
try:
raw = await complete_json(system, sample_description, temperature=0.2, max_tokens=2000)
samples = raw.get("samples") if isinstance(raw, dict) else None
if not isinstance(samples, dict) or not samples:
return {"ok": False, "code": "EMPTY_SAMPLES", "message": "未能从描述解析出样本"}
except Exception as exc: # noqa: BLE001
return {"ok": False, "code": "SAMPLES_GEN_FAILED", "message": f"样本配置生成失败: {exc}"}
record.input_dir.mkdir(parents=True, exist_ok=True)
out = record.input_dir / "samples.yaml"
out.write_text(yaml.safe_dump({"samples": samples}, allow_unicode=True), encoding="utf-8")
v = validate_format(out, "samples.yaml")
if not v.get("ok"):
out.unlink(missing_ok=True)
return {"ok": False, "code": "VALIDATION_FAILED", "message": f"样本配置校验失败: {v.get('reason')}"}
return {"ok": True, "data": {"path": str(out), "samples": samples, "validated": v}}
# ---------------------------------------------------------------------------
# 能力单元工具(命名 async,带类型提示 + 中文描述)
# ---------------------------------------------------------------------------
async def plan_gene_family(message: str) -> dict[str, Any]:
"""根据用户自然语言需求规划基因家族分析候选路径(仅基因家族链阶段)。"""
return await _plan(message, GENE_FAMILY_STAGES)
async def create_gene_family_job(session_id: str, stages: list[str], title: str = "",
params: dict[str, Any] | None = None,
auto_continue: bool = False,
archive_id: str | None = None) -> dict[str, Any]:
"""按确认的阶段链创建基因家族任务(stages 只能是基因家族链阶段,跨链拒绝)。"""
return await _create_job(session_id, title, stages, params, auto_continue, archive_id, GENE_FAMILY_STAGES)
async def ask_gene_family_question(question: str) -> dict[str, Any]:
"""回答基因家族流程/参数/生物学意义类问题(流程知识库检索,带引用)。"""
return await _ask_question(question)
async def plan_rnaseq(message: str) -> dict[str, Any]:
"""根据用户自然语言需求规划转录组分析候选路径(仅转录组链阶段)。"""
return await _plan(message, RNASEQ_STAGES)
async def create_rnaseq_job(session_id: str, stages: list[str], title: str = "",
params: dict[str, Any] | None = None,
auto_continue: bool = False,
archive_id: str | None = None) -> dict[str, Any]:
"""按确认的阶段链创建转录组任务(stages 只能是转录组链阶段,跨链拒绝)。"""
return await _create_job(session_id, title, stages, params, auto_continue, archive_id, RNASEQ_STAGES)
async def generate_samples_yaml(job_id: str, sample_description: str) -> dict[str, Any]:
"""把用户的自然语言样本描述(样本名/对照处理组/重复/单双端等)解析为 samples.yaml 写入任务 input。"""
return await _generate_samples_yaml(job_id, sample_description)
async def ask_rnaseq_question(question: str) -> dict[str, Any]:
"""回答转录组流程/参数/实验设计类问题(流程知识库检索,带引用)。"""
return await _ask_question(question)
async def job_status(job_id: str) -> dict[str, Any]:
"""查询生信任务状态:任务/各阶段状态 + 下一步待办动作(pending_actions)。卡住或需确认时用本工具查真相。"""
return await _job_status(job_id)
async def confirm_stage(job_id: str, stage: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
"""确认某阶段参数并执行。仅 waiting_confirm 状态可确认;参数名须在白名单内(未知参数被拒)。人工确认边界阶段(subfamily_confirm/rnaseq_de)不可代用户确认。"""
return await _confirm_stage(job_id, stage, params)
async def register_job_files(job_id: str, paths: list[str]) -> dict[str, Any]:
"""把服务器/局域网路径(单文件或目录)登记为任务输入文件。目录按扩展名白名单展开。"""
return await _register_job_files(job_id, paths)
async def rerun_stage(job_id: str, stage: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
"""重跑失败阶段(带参数则先等确认)。仅 failed 状态可重跑。"""
return await _rerun_stage(job_id, stage, params)
async def skip_stage(job_id: str, stage: str) -> dict[str, Any]:
"""跳过失败阶段并推进后续。仅 failed 状态可跳过。"""
return await _skip_stage(job_id, stage)
async def pause_job(job_id: str) -> dict[str, Any]:
"""暂停运行中的任务(running 阶段会立即 kill 容器)。"""
return await _pause_job(job_id)
async def resume_job(job_id: str) -> dict[str, Any]:
"""恢复已暂停的任务(从下一阶段继续,确认点不跳过)。"""
return await _resume_job(job_id)
async def list_archives(page: int = 1, page_size: int = 20) -> dict[str, Any]:
"""列出历史存档任务(供复用参考)。"""
return await _list_archives(page, page_size)
async def reuse_archive(name: str) -> dict[str, Any]:
"""按名称匹配存档,返回其阶段链+参数(供复用发起)。"""
return await _reuse_archive(name)
async def archive_job(job_id: str, name: str, description: str = "",
include_intermediate: bool = False) -> dict[str, Any]:
"""把已完成任务存档(打包结果/日志/事件 + 落档登记);重名拒绝(ARCHIVE_EXISTS)。"""
return await _archive_job(job_id, name, description, include_intermediate)
# ---------------------------------------------------------------------------
# 扩展能力单元工具(第三个能力单元 bioflow-extension)
# ---------------------------------------------------------------------------
async def create_extension(display_name: str, command: str, runtime_image: str = "",
code_files: list[dict[str, Any]] | None = None,
data_files: list[dict[str, Any]] | None = None) -> dict[str, Any]:
"""创建扩展草稿(draft):上传代码文件清单(每项 {path, content 文本}) + 运行命令 + 显示名。"""
return await _create_extension(display_name, command, runtime_image, code_files, data_files)
async def extension_status(ext_id: str) -> dict[str, Any]:
"""查询扩展详情与状态(draft/build/trial/validated/confirming/published)。"""
return await _extension_status(ext_id)
async def build_extension(ext_id: str) -> dict[str, Any]:
"""AI 转换扩展(原始命令→command_template+参数+dependencies);LLM 失败自动回退规则版。"""
return await _build_extension(ext_id)
async def trial_run_extension(ext_id: str) -> dict[str, Any]:
"""试运行扩展(docker 执行;未接入执行器时返回 EXT_RUN 503 语义)。"""
return await _trial_run_extension(ext_id)
async def validate_extension(ext_id: str) -> dict[str, Any]:
"""校验扩展(金标/试运行校验)。仅 built/trial_passed 可校验。"""
return await _validate_extension(ext_id)
async def confirm_extension(ext_id: str, nl_triggers: list[str] | None = None,
parameters: list[dict[str, Any]] | None = None,
runtime_image: str = "") -> dict[str, Any]:
"""确认扩展参数/触发词(用户可编辑 nl_triggers);仅 validated/trial_passed 可确认。"""
return await _confirm_extension(ext_id, nl_triggers, parameters, runtime_image)
async def publish_extension(ext_id: str) -> dict[str, Any]:
"""发布扩展(仅 published 状态可被自然语言触发运行)。"""
return await _publish_extension(ext_id)
async def run_extension(ext_id: str, params: dict[str, Any] | None = None,
files: list[str] | None = None) -> dict[str, Any]:
"""正式运行已发布扩展(同步返回 run_result)。"""
return await _run_extension(ext_id, params, files)
async def repair_extension(ext_id: str) -> dict[str, Any]:
"""AI 修复扩展(试运行失败上下文→LLM 修复 command_template/依赖;LLM 失败回退规则版)。"""
return await _repair_extension(ext_id)
async def fetch_source_file(source: str, job_id: str | None = None,
credentials: dict[str, Any] | None = None) -> dict[str, Any]:
"""从 web(http/https 流式)、OSS(oss:// 或预签名 URL)或局域网路径拉取源文件到任务 input;下载前检查磁盘余量并记录 provenance。"""
return await _fetch_source_file(job_id, source, credentials)
async def inspect_file(path: str) -> dict[str, Any]:
"""探测文件格式(扩展名仅参考):返回预览/列结构/大小/格式猜测/乱码检测,供判断是否需要预处理。"""
return await _inspect_file(path)
async def transform_file(src_path: str, target_format: str,
column_map: dict[str, str] | None = None,
dest_path: str | None = None) -> dict[str, Any]:
"""把非标准文件(xlsx/csv/tsv/txt)转换为 fasta/gff3/tsv/samples.yaml;LLM 定列映射规则+脚本执行,结果过格式验证器。"""
return await _transform_file(src_path, target_format, column_map, dest_path)
# ---------------------------------------------------------------------------
# 构建三个 MCPServer(基因家族 / 转录组 / 扩展)
# ---------------------------------------------------------------------------
def build_mcp_servers() -> tuple[Any, Any, Any]:
from mcp.server.mcpserver import MCPServer
gf = MCPServer(
"bioflow-gene-family",
title="PlantAI 基因家族分析",
description="基因家族分析能力单元:路径规划/建任务/确认参数/查状态/文件获取与预处理。",
)
rn = MCPServer(
"bioflow-rnaseq",
title="PlantAI 转录组分析",
description="转录组分析能力单元:路径规划/样本配置生成/建任务/确认参数/查状态/文件获取与预处理。",
)
ext = MCPServer(
"bioflow-extension",
title="PlantAI 用户自定义扩展分析",
description="用户自定义扩展分析能力单元:创建/转换/试运行/校验/确认/发布/运行。",
)
gf.tool(name="plan_gene_family", description="根据用户自然语言需求规划基因家族分析候选路径(仅基因家族链阶段)。返回 2-3 条候选路径供用户确认。")(plan_gene_family)
gf.tool(name="create_gene_family_job", description="按确认的阶段链创建基因家族任务。stages 只能是基因家族链阶段(跨链拒绝)。")(create_gene_family_job)
gf.tool(name="ask_gene_family_question", description="回答基因家族流程/参数/生物学意义类问题(流程知识库检索,带引用)。")(ask_gene_family_question)
rn.tool(name="plan_rnaseq", description="根据用户自然语言需求规划转录组分析候选路径(仅转录组链阶段)。返回候选路径供用户确认。")(plan_rnaseq)
rn.tool(name="create_rnaseq_job", description="按确认的阶段链创建转录组任务。stages 只能是转录组链阶段(跨链拒绝)。")(create_rnaseq_job)
rn.tool(name="generate_samples_yaml", description="把用户的自然语言样本描述解析为 samples.yaml 写入任务 input。")(generate_samples_yaml)
rn.tool(name="ask_rnaseq_question", description="回答转录组流程/参数/实验设计类问题(流程知识库检索,带引用)。")(ask_rnaseq_question)
ext.tool(name="create_extension", description="创建扩展草稿(draft):上传代码文件清单(每项 {path, content 文本}) + 运行命令 + 显示名。")(create_extension)
ext.tool(name="extension_status", description="查询扩展详情与状态(draft/build/trial/validated/confirming/published)。")(extension_status)
ext.tool(name="build_extension", description="AI 转换扩展(原始命令→command_template+参数+依赖);LLM 失败自动回退规则版。")(build_extension)
ext.tool(name="trial_run_extension", description="试运行扩展(docker 执行;未接入执行器时返回 EXT_RUN)。")(trial_run_extension)
ext.tool(name="validate_extension", description="校验扩展(金标/试运行校验)。仅 built/trial_passed 可校验。")(validate_extension)
ext.tool(name="confirm_extension", description="确认扩展参数/触发词(用户可编辑 nl_triggers);仅 validated/trial_passed 可确认。")(confirm_extension)
ext.tool(name="publish_extension", description="发布扩展(仅 published 状态可被自然语言触发运行)。")(publish_extension)
ext.tool(name="run_extension", description="正式运行已发布扩展(同步返回 run_result)。")(run_extension)
ext.tool(name="repair_extension", description="AI 修复扩展(试运行失败上下文→LLM 修复;LLM 失败回退规则版)。")(repair_extension)
shared = [
job_status, confirm_stage, register_job_files, rerun_stage, skip_stage,
pause_job, resume_job, list_archives, reuse_archive, archive_job,
fetch_source_file, inspect_file, transform_file,
]
for mcp in (gf, rn):
for fn in shared:
mcp.tool()(fn)
return gf, rn, ext
gene_family_mcp, rnaseq_mcp, extension_mcp = build_mcp_servers()
@@ -1,438 +0,0 @@
"""app/services/bioflow/file_transform.py —— 非标准文件预处理(探测 + 转换 + 校验)。
FR-BIO-014 文件医生闭环:
- inspect_file:内容嗅探(扩展名仅参考),返回结构化摘要(预览行/列结构/大小/
格式猜测/乱码检测/与目标阶段期望格式的差异)。
- transform_file:LLM 抽取列映射/清洗规则 + 规则脚本执行,把 xlsx/csv/tsv/txt 转换
为 fasta/gff3/tsv/samples.yaml;转换结果过格式验证器硬校验,失败返回原因供追问
循环(≤3 次由上层 QA 控制)。
服务层纪律:纯 dataclass/dict/stdlib + yaml/openpyxl,无 pydantic。
LLM 可注入(默认走 .llm.complete_json),测试注入 fake。
"""
from __future__ import annotations
import csv
import io
import json
import logging
import re
from pathlib import Path
from typing import Any, Awaitable, Callable
logger = logging.getLogger(__name__)
PREVIEW_LINES = 10
PREVIEW_BYTES = 4000
# 支持的目标格式
TARGET_FORMATS = ("fasta", "gff3", "tsv", "samples.yaml")
# 乱码检测:非 ASCII 可打印且非 CJK 的异常字节比例阈值
GARBLED_RATIO = 0.30
class TransformError(Exception):
def __init__(self, code: str, message: str, details: dict[str, Any] | None = None) -> None:
super().__init__(message)
self.code = code
self.message = message
self.details = details or {}
# ---------------------------------------------------------------------------
# 探测
# ---------------------------------------------------------------------------
def _guess_format(name: str, head: bytes) -> str:
lower = name.lower()
if lower.endswith((".fa", ".fasta", ".faa", ".fna", ".pep", ".cds")):
return "fasta"
if lower.endswith((".gff", ".gff3")):
return "gff3"
if lower.endswith((".gtf",)):
return "gtf"
if lower.endswith((".vcf", ".vcf.gz")):
return "vcf"
if lower.endswith((".tsv",)):
return "tsv"
if lower.endswith((".csv",)):
return "csv"
if lower.endswith((".xlsx", ".xls")):
return "xlsx"
if lower.endswith((".yaml", ".yml")):
return "yaml"
# 内容嗅探兜底
text = head.decode("utf-8", errors="replace")
if text.lstrip().startswith(">"):
return "fasta"
if "\t" in text and len(text.splitlines()[0].split("\t")) >= 3:
return "tsv"
return "txt"
def _read_text(path: Path) -> str:
raw = path.read_bytes()
for enc in ("utf-8", "gb18030", "latin-1"):
try:
return raw.decode(enc)
except UnicodeDecodeError:
continue
return raw.decode("utf-8", errors="replace")
def _garbled_ratio(text: str) -> float:
"""异常字符比例:控制字符 + 替换符 + 私有区(不含常规空白)。"""
if not text:
return 0.0
bad = 0
for ch in text:
o = ord(ch)
if o == 0xFFFD or (o < 32 and ch not in "\t\n\r") or (0xE000 <= o <= 0xF8FF):
bad += 1
return bad / len(text)
def _preview(text: str) -> list[str]:
return text.splitlines()[:PREVIEW_LINES]
def _columns(path: Path, fmt: str) -> list[str]:
"""表格列名提取(tsv/csv/xlsx)。"""
try:
if fmt == "xlsx":
import openpyxl
wb = openpyxl.load_workbook(path, read_only=True, data_only=True)
try:
ws = wb.active
row = next(ws.iter_rows(min_row=1, max_row=1, values_only=True), None)
return [str(c or "").strip() for c in (row or [])]
finally:
wb.close()
if fmt in ("tsv", "csv"):
text = _read_text(path)
reader = csv.reader(io.StringIO(text), delimiter="\t" if fmt == "tsv" else ",")
first = next(reader, None)
return [str(c or "").strip() for c in (first or [])]
except Exception as exc: # noqa: BLE001 - 列提取失败降级空
logger.warning("file_transform: 列提取失败 %s: %s", path, exc)
return []
def inspect_file(path: str | Path, *, preview_lines: int = PREVIEW_LINES) -> dict[str, Any]:
"""内容嗅探摘要(扩展名仅参考;失败返回 ok=False + 原因,不抛)。"""
p = Path(path)
if not p.is_file():
return {"ok": False, "code": "NOT_FOUND", "message": f"文件不存在: {path}"}
try:
size = p.stat().st_size
head = p.read_bytes()[:PREVIEW_BYTES]
fmt = _guess_format(p.name, head)
text = _read_text(p) if size <= 4 * 1024 * 1024 else _read_text_preview(p)
return {
"ok": True,
"path": str(p),
"name": p.name,
"ext": p.suffix.lower() or "",
"size": size,
"format": fmt,
"preview": _preview(text)[:preview_lines],
"columns": _columns(p, fmt),
"garbled": _garbled_ratio(text),
"line_count": text.count("\n") + 1 if text else 0,
}
except OSError as exc:
return {"ok": False, "code": "READ_ERROR", "message": f"文件读取失败: {exc}"}
def _read_text_preview(path: Path) -> str:
"""大文件只读头部做预览/乱码检测,避免整读内存。"""
with path.open("rb") as fh:
head = fh.read(PREVIEW_BYTES)
for enc in ("utf-8", "gb18030", "latin-1"):
try:
return head.decode(enc)
except UnicodeDecodeError:
continue
return head.decode("utf-8", errors="replace")
# ---------------------------------------------------------------------------
# 校验器
# ---------------------------------------------------------------------------
def validate_fasta(path: str | Path) -> dict[str, Any]:
p = Path(path)
if not p.is_file():
return {"ok": False, "reason": "文件不存在"}
text = _read_text(p)
records = re.findall(r"^>.*$", text, flags=re.M)
if not records:
return {"ok": False, "reason": "无 FASTA 记录(缺 '>' 头行)"}
# 序列非空:头行之间至少存在非空白序列行
body = re.sub(r"^>.*$", "", text, flags=re.M)
if not body.strip():
return {"ok": False, "reason": "FASTA 记录缺少序列"}
return {"ok": True, "records": len(records)}
def validate_gff3(path: str | Path) -> dict[str, Any]:
p = Path(path)
if not p.is_file():
return {"ok": False, "reason": "文件不存在"}
bad = 0
total = 0
for line in _read_text(p).splitlines():
if not line.strip() or line.startswith("#"):
continue
total += 1
cols = line.split("\t")
if len(cols) != 9:
bad += 1
continue
try:
int(cols[3])
int(cols[4])
except ValueError:
bad += 1
continue
if cols[6] not in ("+", "-", "."):
bad += 1
if total == 0:
return {"ok": False, "reason": "无 GFF3 数据行"}
if bad:
return {"ok": False, "reason": f"{bad}/{total} 行列数或字段非法"}
return {"ok": True, "records": total}
def validate_tsv(path: str | Path) -> dict[str, Any]:
p = Path(path)
if not p.is_file():
return {"ok": False, "reason": "文件不存在"}
widths: set[int] = set()
for line in _read_text(p).splitlines():
if not line.strip():
continue
widths.add(len(line.split("\t")))
if len(widths) > 1:
return {"ok": False, "reason": f"列数不一致: {sorted(widths)}"}
if not widths:
return {"ok": False, "reason": "空文件"}
return {"ok": True, "columns": widths.pop()}
def validate_samples_yaml(path: str | Path) -> dict[str, Any]:
import yaml
p = Path(path)
if not p.is_file():
return {"ok": False, "reason": "文件不存在"}
try:
raw = yaml.safe_load(_read_text(p))
except Exception as exc: # noqa: BLE001
return {"ok": False, "reason": f"YAML 解析失败: {exc}"}
samples = (raw or {}).get("samples") if isinstance(raw, dict) else None
if not isinstance(samples, dict) or not samples:
return {"ok": False, "reason": "缺 samples 映射或为空"}
for name, entry in samples.items():
if not isinstance(entry, dict) or not (entry.get("r1") or entry.get("r2")):
return {"ok": False, "reason": f"样本 {name} 缺 r1/r2"}
return {"ok": True, "samples": len(samples)}
_VALIDATORS = {
"fasta": validate_fasta,
"gff3": validate_gff3,
"tsv": validate_tsv,
"samples.yaml": validate_samples_yaml,
}
def validate_format(path: str | Path, target_format: str) -> dict[str, Any]:
fn = _VALIDATORS.get(target_format)
if fn is None:
return {"ok": False, "reason": f"不支持的目标格式: {target_format}"}
return fn(path)
# ---------------------------------------------------------------------------
# 转换
# ---------------------------------------------------------------------------
LLMFn = Callable[[str, str], Awaitable[Any]]
def _read_table(path: Path) -> tuple[list[str], list[dict[str, str]]]:
"""读表格(tsv/csv/xlsx)→ (表头, 行 dict 列表)。"""
fmt = _guess_format(path.name, path.read_bytes()[:64])
if fmt == "xlsx":
import openpyxl
wb = openpyxl.load_workbook(path, read_only=True, data_only=True)
try:
ws = wb.active
rows = [list(r) for r in ws.iter_rows(values_only=True)]
finally:
wb.close()
if not rows:
return [], []
headers = [str(c or "").strip() for c in rows[0]]
return headers, [dict(zip(headers, ["" if v is None else str(v).strip() for v in r])) for r in rows[1:]]
text = _read_text(path)
delimiter = "\t" if fmt == "tsv" else ","
reader = csv.reader(io.StringIO(text), delimiter=delimiter)
rows = list(reader)
if not rows:
return [], []
headers = [str(c or "").strip() for c in rows[0]]
return headers, [dict(zip(headers, ["" if v is None else str(v).strip() for v in r])) for r in rows[1:]]
def _default_column_map(headers: list[str], target: str) -> dict[str, str]:
"""按表头名启发式推断列映射(名称/序列/基因名等)。"""
m: dict[str, str] = {}
for h in headers:
lh = h.lower()
if target == "fasta":
if not m.get("seq") and any(k in lh for k in ("seq", "sequence", "序列", "蛋白", "pep", "cds")):
m["seq"] = h
elif not m.get("name") and any(k in lh for k in ("name", "id", "gene", "名称", "基因", "基因名", "accession")):
m["name"] = h
elif target == "samples.yaml":
if "r1" in lh or "read1" in lh or "forward" in lh:
m.setdefault("r1", h)
elif "r2" in lh or "read2" in lh or "reverse" in lh:
m.setdefault("r2", h)
elif "run" in lh or "sample" in lh or "样本" in lh or "name" in lh:
m.setdefault("name", h)
return m
async def _llm_column_map(headers: list[str], target: str, llm: LLMFn, sample_rows: list[dict[str, str]]) -> dict[str, str]:
"""LLM 抽取列映射(启发式不充分时)。返回 {目标字段: 列名}。"""
import json as _json
prompt = (
f"你是生信数据整理助手。下表头 {_json.dumps(headers, ensure_ascii=False)} "
f"需转换为 {target} 格式。样本行:{_json.dumps(sample_rows[:3], ensure_ascii=False)[:2000]}。"
"请输出列映射 JSON,键为目标字段名,值为表头列名。"
)
try:
result = await llm(prompt, "只输出 JSON 对象,如 {\"name\":\"基因名\",\"seq\":\"序列\"}。")
if isinstance(result, dict):
return {str(k): str(v) for k, v in result.items() if str(v) in headers}
except Exception as exc: # noqa: BLE001
logger.warning("file_transform: LLM 列映射失败 %s", exc)
return {}
def _write_fasta(rows: list[dict[str, str]], name_col: str, seq_col: str, dest: Path) -> int:
n = 0
with dest.open("w", encoding="utf-8") as fh:
for r in rows:
name = r.get(name_col) or f"seq{n + 1}"
seq = (r.get(seq_col) or "").replace("\n", "").replace("\r", "")
if not seq:
continue
fh.write(f">{name}\n{seq}\n")
n += 1
return n
def _write_tsv(rows: list[dict[str, str]], cols: list[str], dest: Path) -> int:
n = 0
with dest.open("w", encoding="utf-8", newline="") as fh:
w = csv.writer(fh, delimiter="\t")
w.writerow(cols)
for r in rows:
w.writerow([r.get(c, "") for c in cols])
n += 1
return n
def _write_samples_yaml(rows: list[dict[str, str]], name_col: str, r1_col: str, r2_col: str, dest: Path) -> int:
import yaml
samples: dict[str, dict[str, str]] = {}
for r in rows:
name = r.get(name_col) or ""
r1 = r.get(r1_col) or ""
r2 = r.get(r2_col) or ""
if not name and not r1 and not r2:
continue
entry: dict[str, str] = {}
if r1:
entry["r1"] = r1
if r2:
entry["r2"] = r2
# 额外列作为 meta(run/condition/group 等)
for k, v in r.items():
if v and k not in (name_col, r1_col, r2_col):
entry[k] = v
samples[name or Path(r1).stem] = entry
dest.write_text(yaml.safe_dump({"samples": samples}, allow_unicode=True), encoding="utf-8")
return len(samples)
async def transform_file(
src_path: str | Path,
target_format: str,
*,
column_map: dict[str, str] | None = None,
dest_path: str | Path | None = None,
llm: LLMFn | None = None,
) -> dict[str, Any]:
"""非标准文件转换(LLM 列映射兜底 + 规则执行 + 格式校验)。
column_map:用户/QA 确认的 {目标字段: 源列名}(可部分),缺列经启发式/LLM 补齐。
返回 ok=True + path/size/validated/records,或 ok=False + code/message(供追问)。
"""
src = Path(src_path)
if not src.is_file():
raise TransformError("NOT_FOUND", f"源文件不存在: {src_path}")
if target_format not in TARGET_FORMATS:
raise TransformError("BAD_FORMAT", f"不支持的目标格式: {target_format}")
dest = Path(dest_path) if dest_path else src.with_suffix("").with_name(
f"{src.stem}.{target_format.replace('.', '_')}"
)
headers, rows = _read_table(src)
if not headers:
raise TransformError("EMPTY_TABLE", "表格为空或无表头,无法转换")
cm = dict(column_map or {})
cm = {**cm, **_default_column_map(headers, target_format)}
if target_format == "fasta":
if "seq" not in cm or "name" not in cm:
if llm is not None:
inferred = await _llm_column_map(headers, target_format, llm, rows)
cm = {**cm, **inferred}
if "seq" not in cm or "name" not in cm:
raise TransformError("MISSING_COLUMNS", "无法确定序列列/名称列,请确认列映射")
n = _write_fasta(rows, cm["name"], cm["seq"], dest)
elif target_format == "tsv":
cols = [cm.get("name")] if cm.get("name") else []
cols = [c for c in (cols + [h for h in headers if h not in cols]) if c]
n = _write_tsv(rows, cols, dest)
elif target_format == "samples.yaml":
if "r1" not in cm and "r2" not in cm:
if llm is not None:
inferred = await _llm_column_map(headers, target_format, llm, rows)
cm = {**cm, **inferred}
if "r1" not in cm and "r2" not in cm:
raise TransformError("MISSING_COLUMNS", "无法确定 fastq 路径列(r1/r2),请确认列映射")
n = _write_samples_yaml(rows, cm.get("name", ""), cm.get("r1", ""), cm.get("r2", ""), dest)
else: # gff3:表格式 GFF 需严格 9 列映射,LLM 定规则后由上层脚本执行
raise TransformError("UNSUPPORTED", f"{target_format} 需 9 列映射,请经 LLM 定规则后转换")
validated = validate_format(dest, target_format)
if not validated.get("ok"):
# 校验失败:删除产物,返回原因供追问
dest.unlink(missing_ok=True)
raise TransformError("VALIDATION_FAILED", f"转换结果校验失败: {validated.get('reason')}", validated)
return {
"ok": True,
"path": str(dest),
"name": dest.name,
"size": dest.stat().st_size,
"format": target_format,
"records": n,
"validated": validated,
}
@@ -1,271 +0,0 @@
"""app/services/bioflow/source_fetcher.py —— 源文件拉取(web/OSS/局域网)+ provenance。
三类来源(FR-BIO-013 网络源扩展):
- web:http/https。httpx 流式下载,不设大小上限,超时 300s,重定向跟随。
- OSS:`oss://bucket/key` 或预签名 URL。无 OSS SDK 依赖:`oss://` 经可配 endpoint
(BIOFLOW_OSS_ENDPOINT)转 http 下载;预签名 URL 直接走 http 通道。
- 局域网:本机已挂载/可达路径直读(不复制,登记原路径);或 `[user@]host:/abs/path`
经 scp 拉取(密钥认证优先;密码需本机有 sshpass)。凭据仅单次内存使用,不落盘。
- 下载前检查磁盘剩余空间(Content-Length 已知时硬校验;否则保底阈值 512MB)。
- 每次拉取记录 provenance(来源/哈希/时间/kind)到返回值并落 sidecar .provenance.json,
不记明文密码。
服务层纪律:纯 dataclass/dict/stdlib + httpx,无 pydantic。
"""
from __future__ import annotations
import asyncio
import hashlib
import json
import logging
import os
import shutil
import urllib.parse
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import httpx
logger = logging.getLogger(__name__)
ALLOWED_SCHEMES = ("http", "https")
DEFAULT_TIMEOUT = 300.0
MIN_FREE_FALLBACK = 512 * 1024 * 1024 # 未知大小时保底磁盘余量检查(512MB)
CHUNK = 1024 * 1024
class FetchError(Exception):
"""源文件拉取错误(带错误码;MCP 工具层转自然语言提示)。"""
def __init__(self, code: str, message: str) -> None:
super().__init__(message)
self.code = code
self.message = message
def _now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def _sha256(path: Path) -> str:
h = hashlib.sha256()
with path.open("rb") as fh:
for block in iter(lambda: fh.read(CHUNK), b""):
h.update(block)
return h.hexdigest()
def _safe_name(name: str) -> str:
"""URL 解码后取文件名(去路径分隔符),空则占位。"""
name = urllib.parse.unquote(name or "")
base = Path(name).name or "download"
return base
def _name_from_url(url: str) -> str:
return _safe_name(urllib.parse.urlparse(url).path.rsplit("/", 1)[-1])
def check_disk_space(dest_dir: str | Path, needed: int) -> tuple[bool, int]:
"""检查目标目录所在盘剩余空间是否 ≥ needed 字节。返回 (够, 剩余字节)。"""
dest_dir = Path(dest_dir)
dest_dir.mkdir(parents=True, exist_ok=True)
free = shutil.disk_usage(dest_dir).free
return (free >= needed, free)
def _write_provenance(dest: Path, source: str, kind: str, sha256: str, extra: dict[str, Any] | None = None) -> dict[str, Any]:
"""落 provenance sidecar({dest}.provenance.json),返回 provenance dict。"""
prov: dict[str, Any] = {
"source": source,
"kind": kind,
"sha256": sha256,
"fetched_at": _now_iso(),
}
if extra:
prov.update(extra)
sidecar = dest.parent / f"{dest.name}.provenance.json"
try:
sidecar.write_text(json.dumps(prov, ensure_ascii=False, indent=2), encoding="utf-8")
except OSError: # noqa: BLE001 - provenance 落盘失败不阻断拉取
logger.warning("source_fetcher: provenance 落盘失败 %s", sidecar)
return prov
# ---- web ----
async def fetch_web(url: str, dest_dir: str | Path, *, timeout: float = DEFAULT_TIMEOUT) -> dict[str, Any]:
"""http/https 流式下载(不设大小上限;Content-Length 已知时下载前查磁盘余量)。"""
scheme = urllib.parse.urlparse(url).scheme.lower()
if scheme not in ALLOWED_SCHEMES:
raise FetchError("BAD_SCHEME", f"仅支持 http/https 协议: {url}")
dest_dir = Path(dest_dir)
dest_dir.mkdir(parents=True, exist_ok=True)
name = _name_from_url(url)
dest = dest_dir / name
async with httpx.AsyncClient(follow_redirects=True) as client:
try:
async with client.stream("GET", url, timeout=httpx.Timeout(connect=30.0, read=timeout, write=60.0, pool=10.0)) as resp:
if resp.status_code >= 400:
raise FetchError("HTTP_ERROR", f"下载失败 HTTP {resp.status_code}: {url}")
total = int(resp.headers.get("content-length") or 0)
if total:
ok, free = check_disk_space(dest_dir, total)
if not ok:
raise FetchError("DISK_FULL", f"磁盘空间不足:需 {total} 字节,剩余 {free} 字节")
written = 0
with dest.open("wb") as fh:
async for chunk in resp.aiter_bytes(CHUNK):
fh.write(chunk)
written += len(chunk)
if total and written != total:
dest.unlink(missing_ok=True)
raise FetchError("TRUNCATED", f"下载不完整:期望 {total} 字节,实收 {written} 字节")
except FetchError:
raise
except httpx.HTTPError as exc:
dest.unlink(missing_ok=True)
raise FetchError("NETWORK_ERROR", f"网络错误: {exc}") from exc
size = dest.stat().st_size
sha = _sha256(dest)
prov = _write_provenance(dest, url, "web", sha)
return {"ok": True, "path": str(dest), "name": name, "size": size, "kind": "web", "provenance": prov}
# ---- OSS ----
def _resolve_oss_url(spec: str) -> str:
"""oss://bucket/key → http 下载 URL(public-read 或预签名;无 SDK,需 endpoint 配置)。
环境键:BIOFLOW_OSS_ENDPOINT(如 oss-cn-hangzhou.aliyuncs.com,或内网 endpoint)。
预签名 URL(含 aliyuncs.com 的 http/https)原样返回,走 web 通道。
"""
if not spec.startswith("oss://"):
return spec # 已是 http/https 预签名 URL
rest = spec[len("oss://"):]
bucket, _, key = rest.partition("/")
endpoint = (os.environ.get("BIOFLOW_OSS_ENDPOINT") or "").strip().rstrip("/")
if not bucket or not key:
raise FetchError("BAD_OSS", f"OSS 地址格式非法(应为 oss://bucket/key): {spec}")
if not endpoint:
raise FetchError("OSS_UNCONFIGURED", "未配置 BIOFLOW_OSS_ENDPOINT,无法解析 oss:// 地址;请提供预签名 URL")
# 内网/公网 endpoint 统一拼 http(https 由 endpoint 决定;公网 OSS 建议 https)
scheme = "https" if "aliyuncs.com" in endpoint else "http"
return f"{scheme}://{bucket}.{endpoint}/{key}"
async def fetch_oss(spec: str, dest_dir: str | Path, *, timeout: float = DEFAULT_TIMEOUT) -> dict[str, Any]:
url = _resolve_oss_url(spec)
result = await fetch_web(url, dest_dir, timeout=timeout)
result["kind"] = "oss"
# 重写 provenance kind(sidecar 已由 fetch_web 写为 web;此处修正)
if result.get("provenance"):
result["provenance"]["kind"] = "oss"
result["provenance"]["oss_spec"] = spec
try:
sidecar = Path(result["path"]).parent / f"{Path(result['path']).name}.provenance.json"
sidecar.write_text(json.dumps(result["provenance"], ensure_ascii=False, indent=2), encoding="utf-8")
except OSError:
pass
return result
# ---- 局域网 ----
def _parse_lan_remote(spec: str) -> tuple[str, str] | None:
"""解析 `[user@]host:/abs/path` 远端形态;非远端返回 None。"""
if ":" not in spec:
return None
# 排除协议(http:// / oss:// 含 ://,已在上层分流)
if "://" in spec:
return None
host_part, _, path_part = spec.partition(":")
if not path_part.startswith("/"):
return None
return host_part, path_part
async def _scp_fetch(host_part: str, remote_path: str, dest_dir: Path, credentials: dict[str, Any] | None) -> dict[str, Any]:
"""scp 拉取远端文件(密钥认证;密码需本机 sshpass)。"""
creds = credentials or {}
user = str(creds.get("user") or "").strip()
password = str(creds.get("password") or "").strip()
port = str(creds.get("port") or "22").strip()
target = f"{user}@{host_part}" if user else host_part
name = _safe_name(remote_path.rsplit("/", 1)[-1])
dest_dir.mkdir(parents=True, exist_ok=True)
cmd: list[str] = []
if password:
if shutil.which("sshpass") is None:
raise FetchError("SSHPASS_MISSING", "本机未安装 sshpass,无法使用密码拉取;请改用密钥认证或提供已挂载路径")
cmd += ["sshpass", "-p", password]
cmd += ["scp", "-P", port, "-o", "StrictHostKeyChecking=accept-new", "-o", "ConnectTimeout=30",
f"{target}:{remote_path}", str(dest_dir / name)]
try:
proc = await asyncio.create_subprocess_exec(
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
_out, err = await asyncio.wait_for(proc.communicate(), timeout=DEFAULT_TIMEOUT)
except asyncio.TimeoutError as exc:
raise FetchError("LAN_TIMEOUT", f"局域网拉取超时: {target}:{remote_path}") from exc
except OSError as exc:
raise FetchError("LAN_ERROR", f"局域网拉取失败: {exc}") from exc
if proc.returncode != 0:
raise FetchError("LAN_ERROR", f"scp 失败: {(err or b'').decode(errors='replace')[:500]}")
dest = dest_dir / name
if not dest.is_file():
raise FetchError("LAN_ERROR", f"scp 未产出文件: {dest}")
size = dest.stat().st_size
sha = _sha256(dest)
prov = _write_provenance(dest, f"{target}:{remote_path}", "lan", sha)
return {"ok": True, "path": str(dest), "name": name, "size": size, "kind": "lan", "provenance": prov}
async def fetch_lan(spec: str, dest_dir: str | Path, *, credentials: dict[str, Any] | None = None) -> dict[str, Any]:
"""局域网路径:本机已挂载/可达直读(返回原路径,不复制);远端走 scp。"""
dest_dir = Path(dest_dir)
remote = _parse_lan_remote(spec)
if remote is not None:
return await _scp_fetch(remote[0], remote[1], dest_dir, credentials)
# 本机路径:直读(存在即成功;不复制,登记语义由 register_job_files 处理)
p = Path(spec).expanduser()
if not p.exists():
raise FetchError("PATH_NOT_EXIST", f"局域网路径不存在或不可达: {spec}")
if p.is_dir():
raise FetchError("IS_DIR", f"路径是目录(仅支持单文件拉取): {spec}")
sha = _sha256(p)
prov = _write_provenance(p, str(p), "lan", sha)
return {"ok": True, "path": str(p), "name": p.name, "size": p.stat().st_size, "kind": "lan", "provenance": prov}
# ---- 总入口 ----
async def fetch_source(
source: str,
dest_dir: str | Path,
*,
credentials: dict[str, Any] | None = None,
timeout: float = DEFAULT_TIMEOUT,
) -> dict[str, Any]:
"""按来源形态分流拉取(web/OSS/局域网/本机路径)。
credentials(可选,仅局域网 scp 用):{"user": ..., "password": ..., "port": ...}。
凭据不落盘、不进入 provenance。
"""
source = (source or "").strip()
if not source:
raise FetchError("EMPTY_SOURCE", "源地址为空")
lower = source.lower()
# 协议白名单:含 "://" 的 URL 只允许 http/https/oss(其余如 ftp/file/sftp 拒绝)
if "://" in source and not lower.startswith(("http://", "https://", "oss://")):
raise FetchError("BAD_SCHEME", f"不支持的协议(仅 http/https/oss): {source}")
if lower.startswith(("http://", "https://")):
if lower.startswith("oss://") or "aliyuncs.com" in lower:
return await fetch_oss(source, dest_dir, timeout=timeout)
return await fetch_web(source, dest_dir, timeout=timeout)
if lower.startswith("oss://"):
return await fetch_oss(source, dest_dir, timeout=timeout)
return await fetch_lan(source, dest_dir, credentials=credentials)
@@ -1,403 +0,0 @@
"""tests/test_file_transform.py —— 非标准文件预处理(探测 + 转换 + 校验)。
分层标记 -m bioflow_unit。覆盖(S0 验收):
- inspect_file:fasta/tsv 探测、乱码检测、列结构提取。
- transform_file:xlsx→fasta(显式列映射)、csv→fasta(启发式列映射)、
samples.yaml 转换 + 校验。
- 校验失败 → TransformError VALIDATION_FAILED(追问循环依据)。
- 缺列映射且无 LLM → MISSING_COLUMNS。
"""
from __future__ import annotations
import csv
import asyncio
from pathlib import Path
import pytest
from app.services.bioflow.file_transform import (
TransformError,
_garbled_ratio,
inspect_file,
transform_file,
validate_fasta,
validate_format,
validate_gff3,
validate_samples_yaml,
validate_tsv,
)
def _run(coro):
return asyncio.new_event_loop().run_until_complete(coro)
# ---------------------------------------------------------------------------
# inspect_file 探测
# ---------------------------------------------------------------------------
def test_inspect_file_fasta(tmp_path):
f = tmp_path / "proteins.fa"
f.write_text(">SPL1\nMKTA\n>SPL2\nMKTB\n")
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "fasta"
assert r["size"] > 0
def test_inspect_file_tsv_columns(tmp_path):
f = tmp_path / "samples.tsv"
f.write_text("run\tfastq_1\tfastq_2\nSRR1\ta_1.fq\ta_2.fq\n")
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "tsv"
assert r["columns"] == ["run", "fastq_1", "fastq_2"]
def test_inspect_file_garbled(tmp_path):
f = tmp_path / "messy.txt"
f.write_bytes(b"\x00\x01\x02\x03\xff\xfe garbage \x00\x00")
r = inspect_file(f)
assert r["ok"] is True
assert r["garbled"] > 0.3
def test_garbled_ratio_clean():
assert _garbled_ratio("ATCGATCG\n>seq\n") < 0.01
# ---------------------------------------------------------------------------
# transform_file
# ---------------------------------------------------------------------------
def test_transform_xlsx_to_fasta(tmp_path):
import openpyxl
src = tmp_path / "spl_list.xlsx"
wb = openpyxl.Workbook()
ws = wb.active
ws.append(["基因名", "序列", "备注"])
ws.append(["SPL1", "ATGGCTA", "水稻"])
ws.append(["SPL2", "ATGGCTB", "水稻"])
wb.save(src)
r = _run(transform_file(src, "fasta", column_map={"name": "基因名", "seq": "序列"}))
assert r["ok"] is True
assert r["records"] == 2
assert r["validated"]["ok"] is True
text = Path(r["path"]).read_text()
assert ">SPL1" in text and "ATGGCTA" in text
def test_transform_csv_to_fasta_heuristic(tmp_path):
src = tmp_path / "genes.csv"
src.write_text("gene_id,protein_sequence\nG1,MKTA\nG2,MKTB\n")
r = _run(transform_file(src, "fasta"))
assert r["ok"] is True
assert r["records"] == 2
def test_transform_csv_to_samples_yaml(tmp_path):
src = tmp_path / "design.csv"
src.write_text("sample,fastq_1,fastq_2,condition\nA,a_1.fq,a_2.fq,ctrl\nB,b_1.fq,b_2.fq,treated\n")
r = _run(transform_file(src, "samples.yaml", column_map={"name": "sample", "r1": "fastq_1", "r2": "fastq_2"}))
assert r["ok"] is True
assert r["records"] == 2
assert r["validated"]["ok"] is True
import yaml
samples = yaml.safe_load(Path(r["path"]).read_text())["samples"]
assert "A" in samples and samples["A"]["r1"] == "a_1.fq"
def test_transform_validation_failed(tmp_path):
# seq 列全空 → 产物无序列 → 校验失败
src = tmp_path / "bad.csv"
src.write_text("name,seq\nG1,\nG2,\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta", column_map={"name": "name", "seq": "seq"}))
assert ei.value.code == "VALIDATION_FAILED"
def test_transform_missing_columns(tmp_path):
# 无列映射、无 LLM、表头无启发式命中 → MISSING_COLUMNS
src = tmp_path / "unknown.csv"
src.write_text("col_a,col_b\n1,2\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta"))
assert ei.value.code == "MISSING_COLUMNS"
# ---------------------------------------------------------------------------
# 校验器
# ---------------------------------------------------------------------------
def test_validate_fasta(tmp_path):
f = tmp_path / "ok.fa"
f.write_text(">a\nACGT\n")
assert validate_fasta(f)["ok"] is True
def test_validate_fasta_empty(tmp_path):
f = tmp_path / "bad.fa"
f.write_text("no records here\n")
assert validate_fasta(f)["ok"] is False
def test_validate_tsv_inconsistent(tmp_path):
f = tmp_path / "bad.tsv"
f.write_text("a\tb\tc\n1\t2\n")
assert validate_tsv(f)["ok"] is False
def test_validate_samples_yaml(tmp_path):
f = tmp_path / "s.yaml"
f.write_text("samples:\n A:\n r1: a.fq\n r2: a2.fq\n")
assert validate_samples_yaml(f)["ok"] is True
# ===========================================================================
# S0 补强(1)inspect_file:NOT_FOUND / xlsx 列 / csv 格式 / 大文件 / 内容嗅探
# ===========================================================================
def test_inspect_file_not_found():
r = inspect_file("/no/such/file.fa")
assert r["ok"] is False
assert r["code"] == "NOT_FOUND"
def test_inspect_file_xlsx_columns(tmp_path):
import openpyxl
f = tmp_path / "spl.xlsx"
wb = openpyxl.Workbook()
ws = wb.active
ws.append(["基因名", "序列"])
ws.append(["SPL1", "MKTA"])
wb.save(f)
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "xlsx"
assert r["columns"] == ["基因名", "序列"]
def test_inspect_file_csv_format(tmp_path):
f = tmp_path / "genes.csv"
f.write_text("id,seq\nG1,MKTA\n")
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "csv"
assert r["columns"] == ["id", "seq"]
def test_inspect_file_large_uses_preview(tmp_path):
f = tmp_path / "big.fa"
f.write_bytes(b">seq\nACGT\n" + b"A" * (4 * 1024 * 1024))
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "fasta"
def test_guess_format_content_sniffing_fasta(tmp_path):
from app.services.bioflow.file_transform import _guess_format
assert _guess_format("unknown.dat", b">seq1\nACGT\n") == "fasta"
def test_guess_format_content_sniffing_tsv(tmp_path):
from app.services.bioflow.file_transform import _guess_format
assert _guess_format("unknown.dat", b"a\tb\tc\n1\t2\t3\n") == "tsv"
def test_garbled_ratio_empty():
assert _garbled_ratio("") == 0.0
# ===========================================================================
# S0 补强(2)transform_file:NOT_FOUND / BAD_FORMAT / EMPTY_TABLE / tsv / gff3 /
# samples.yaml 启发式 / LLM 兜底 / LLM 异常兜底
# ===========================================================================
def test_transform_not_found(tmp_path):
with pytest.raises(TransformError) as ei:
_run(transform_file(tmp_path / "nope.csv", "fasta"))
assert ei.value.code == "NOT_FOUND"
def test_transform_bad_format(tmp_path):
src = tmp_path / "x.csv"
src.write_text("a,b\n1,2\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "bogus"))
assert ei.value.code == "BAD_FORMAT"
def test_transform_empty_table(tmp_path):
src = tmp_path / "empty.csv"
src.write_text("")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta"))
assert ei.value.code == "EMPTY_TABLE"
def test_transform_to_tsv(tmp_path):
src = tmp_path / "genes.csv"
src.write_text("gene_id,protein_sequence\nG1,MKTA\nG2,MKTB\n")
r = _run(transform_file(src, "tsv"))
assert r["ok"] is True
assert r["records"] == 2
assert r["validated"]["ok"] is True
def test_transform_to_gff3_unsupported(tmp_path):
src = tmp_path / "x.csv"
src.write_text("a,b\n1,2\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "gff3"))
assert ei.value.code == "UNSUPPORTED"
def test_transform_samples_yaml_heuristic(tmp_path):
src = tmp_path / "design.csv"
src.write_text("name,r1,r2,condition\nA,a_1.fq,a_2.fq,ctrl\nB,b_1.fq,b_2.fq,trt\n")
r = _run(transform_file(src, "samples.yaml"))
assert r["ok"] is True
assert r["records"] == 2
def test_transform_samples_yaml_missing_columns(tmp_path):
src = tmp_path / "unknown.csv"
src.write_text("col_a,col_b\n1,2\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "samples.yaml"))
assert ei.value.code == "MISSING_COLUMNS"
def test_transform_fasta_llm_column_map(tmp_path):
src = tmp_path / "candidates.csv"
src.write_text("col_a,col_b\nSPL1,ATGGCTA\nSPL2,ATGGCTB\n")
async def fake_llm(user, system):
return {"name": "col_a", "seq": "col_b"}
r = _run(transform_file(src, "fasta", llm=fake_llm))
assert r["ok"] is True
assert r["records"] == 2
def test_transform_fasta_llm_exception_falls_back_missing(tmp_path):
src = tmp_path / "candidates.csv"
src.write_text("col_a,col_b\nSPL1,ATGGCTA\n")
async def fake_llm(user, system):
raise RuntimeError("llm down")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta", llm=fake_llm))
assert ei.value.code == "MISSING_COLUMNS"
# ===========================================================================
# S0 补强(3)校验器:gff3 各失败态 / samples.yaml 缺 r1r2 / YAML 解析失败 /
# tsv 空文件 / validate_format 未知格式
# ===========================================================================
def test_validate_gff3_ok(tmp_path):
f = tmp_path / "ok.gff3"
f.write_text("chr1\tsrc\tgene\t1\t100\t.\t+\t.\tID=x\n")
assert validate_gff3(f)["ok"] is True
def test_validate_gff3_no_data(tmp_path):
f = tmp_path / "empty.gff3"
f.write_text("# just a comment\n")
assert validate_gff3(f)["ok"] is False
def test_validate_gff3_bad_columns(tmp_path):
f = tmp_path / "bad.gff3"
f.write_text("chr1\tsrc\tgene\t1\t100\n") # 5 列 ≠ 9
assert validate_gff3(f)["ok"] is False
def test_validate_gff3_bad_strand(tmp_path):
f = tmp_path / "bad.gff3"
f.write_text("chr1\tsrc\tgene\t1\t100\t.\tx\t.\tID=x\n") # 第 7 列非法
assert validate_gff3(f)["ok"] is False
def test_validate_gff3_bad_int(tmp_path):
f = tmp_path / "bad.gff3"
f.write_text("chr1\tsrc\tgene\tabc\t100\t.\t+\t.\tID=x\n") # 起止非整数
assert validate_gff3(f)["ok"] is False
def test_validate_fasta_no_sequence(tmp_path):
f = tmp_path / "bad.fa"
f.write_text(">a\n")
assert validate_fasta(f)["ok"] is False
def test_validate_samples_yaml_missing_r1r2(tmp_path):
f = tmp_path / "s.yaml"
f.write_text("samples:\n A:\n condition: ctrl\n")
assert validate_samples_yaml(f)["ok"] is False
def test_validate_samples_yaml_parse_fail(tmp_path):
f = tmp_path / "s.yaml"
f.write_text("samples: [unclosed\n")
assert validate_samples_yaml(f)["ok"] is False
def test_validate_tsv_empty(tmp_path):
f = tmp_path / "empty.tsv"
f.write_text("")
assert validate_tsv(f)["ok"] is False
def test_validate_format_unknown(tmp_path):
f = tmp_path / "x.txt"
f.write_text("x")
r = validate_format(f, "bogus")
assert r["ok"] is False
assert "不支持" in r["reason"]
# ===========================================================================
# S0 补强(4)校验器缺失文件 / 格式猜测扩展名 / samples 非 dict / 空行跳过
# ===========================================================================
def test_validators_missing_file(tmp_path):
missing = tmp_path / "nope"
assert validate_fasta(missing)["ok"] is False
assert validate_gff3(missing)["ok"] is False
assert validate_tsv(missing)["ok"] is False
assert validate_samples_yaml(missing)["ok"] is False
def test_guess_format_all_extensions():
from app.services.bioflow.file_transform import _guess_format
assert _guess_format("a.gff3", b"") == "gff3"
assert _guess_format("a.gff", b"") == "gff3"
assert _guess_format("a.gtf", b"") == "gtf"
assert _guess_format("a.vcf", b"") == "vcf"
assert _guess_format("a.vcf.gz", b"") == "vcf"
assert _guess_format("a.yaml", b"") == "yaml"
assert _guess_format("a.yml", b"") == "yaml"
assert _guess_format("a.xls", b"") == "xlsx"
assert _guess_format("a.faa", b"") == "fasta"
assert _guess_format("a.fna", b"") == "fasta"
assert _guess_format("a.pep", b"") == "fasta"
assert _guess_format("a.cds", b"") == "fasta"
assert _guess_format("a.dat", b"no marker") == "txt"
def test_validate_samples_yaml_not_dict(tmp_path):
f = tmp_path / "s.yaml"
f.write_text("samples: just_a_string\n")
assert validate_samples_yaml(f)["ok"] is False
def test_transform_samples_yaml_skips_empty_rows(tmp_path):
src = tmp_path / "d.csv"
src.write_text("name,r1,r2\n,,\nA,a.fq,a2.fq\n")
r = _run(transform_file(src, "samples.yaml"))
assert r["ok"] is True
assert r["records"] == 1
File diff suppressed because it is too large Load Diff
@@ -1,455 +0,0 @@
"""tests/test_source_fetcher.py —— 源文件拉取(web/OSS/局域网)+ 磁盘余量 + provenance。
分层标记 -m bioflow_unit。覆盖(S0 验收):
- web http 流式下载(本地 http.server 实测 httpx 通道)+ provenance 落盘。
- 协议白名单(ftp:// 拒绝 BAD_SCHEME)。
- oss:// 无 endpoint 拒绝 / 预签名 URL 直通 web / endpoint 解析。
- 局域网本机路径直读;远端 host:/path 解析。
- 磁盘余量不足 DISK_FULL(check_disk_space 纯函数)。
"""
from __future__ import annotations
import asyncio
import functools
import http.server
import threading
from pathlib import Path
import pytest
from app.services.bioflow.source_fetcher import (
FetchError,
_parse_lan_remote,
_resolve_oss_url,
check_disk_space,
fetch_lan,
fetch_source,
fetch_web,
)
def _run(coro):
return asyncio.new_event_loop().run_until_complete(coro)
# ---------------------------------------------------------------------------
# 本地 http 服务器(测 httpx 通道,不依赖外网)
# ---------------------------------------------------------------------------
@pytest.fixture()
def http_url(tmp_path):
content = b">seq1\nACGTACGT\n>seq2\nTGCATGCA\n"
(tmp_path / "test.fa").write_bytes(content)
handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=str(tmp_path))
server = http.server.HTTPServer(("127.0.0.1", 0), handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
yield f"http://127.0.0.1:{server.server_address[1]}/test.fa"
server.shutdown()
def test_fetch_web_download_and_provenance(http_url, tmp_path):
out = tmp_path / "out"
r = _run(fetch_web(http_url, out))
assert r["ok"] is True
assert r["name"] == "test.fa"
assert r["size"] > 0
assert r["kind"] == "web"
assert r["provenance"]["source"] == http_url
assert len(r["provenance"]["sha256"]) == 64
assert (out / "test.fa").read_text().startswith(">seq1")
# sidecar provenance
assert (out / "test.fa.provenance.json").is_file()
def test_fetch_web_http_error(http_url, tmp_path):
with pytest.raises(FetchError) as ei:
_run(fetch_web(http_url + "/missing", tmp_path / "out"))
assert ei.value.code == "HTTP_ERROR"
def test_fetch_source_bad_scheme(tmp_path):
with pytest.raises(FetchError) as ei:
_run(fetch_source("ftp://host/x.fa", tmp_path))
assert ei.value.code == "BAD_SCHEME"
# ---------------------------------------------------------------------------
# OSS
# ---------------------------------------------------------------------------
def test_oss_url_no_endpoint_rejected():
with pytest.raises(FetchError) as ei:
_resolve_oss_url("oss://bucket/key.fa")
assert ei.value.code == "OSS_UNCONFIGURED"
def test_oss_url_endpoint_resolve(monkeypatch):
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-cn-hangzhou.aliyuncs.com")
url = _resolve_oss_url("oss://seq-bucket/data/x.fa")
assert url == "https://seq-bucket.oss-cn-hangzhou.aliyuncs.com/data/x.fa"
def test_oss_presigned_passthrough():
url = _resolve_oss_url("https://bucket.oss-cn-hangzhou.aliyuncs.com/x.fa?sig=abc")
assert url.startswith("https://")
# ---------------------------------------------------------------------------
# 局域网 / 本机路径
# ---------------------------------------------------------------------------
def test_parse_lan_remote():
assert _parse_lan_remote("192.168.1.5:/data/x.fa") == ("192.168.1.5", "/data/x.fa")
assert _parse_lan_remote("xzq@192.168.1.5:/data/x.fa") == ("xzq@192.168.1.5", "/data/x.fa")
assert _parse_lan_remote("/local/path/x.fa") is None
assert _parse_lan_remote("http://x/y") is None
def test_fetch_lan_local_path(tmp_path):
f = tmp_path / "genome.fa"
f.write_text(">g1\nACGT\n")
r = _run(fetch_lan(str(f), tmp_path / "out"))
assert r["ok"] is True
assert r["kind"] == "lan"
assert r["path"] == str(f)
assert r["provenance"]["sha256"]
def test_fetch_lan_missing_path(tmp_path):
with pytest.raises(FetchError) as ei:
_run(fetch_lan("/no/such/path.fa", tmp_path))
assert ei.value.code == "PATH_NOT_EXIST"
# ---------------------------------------------------------------------------
# 磁盘余量
# ---------------------------------------------------------------------------
def test_check_disk_space_insufficient(tmp_path):
ok, free = check_disk_space(tmp_path, 10 ** 18) # 1EB,必不足
assert ok is False
assert free < 10 ** 18
def test_check_disk_space_sufficient(tmp_path):
ok, _free = check_disk_space(tmp_path, 1) # 1 字节
assert ok is True
# ===========================================================================
# S0 补强(1)web:BAD_SCHEME / 磁盘满 DISK_FULL / 断点 TRUNCATED / NETWORK_ERROR
# ===========================================================================
def test_fetch_web_bad_scheme(tmp_path):
with pytest.raises(FetchError) as ei:
_run(fetch_web("ftp://host/x.fa", tmp_path / "out"))
assert ei.value.code == "BAD_SCHEME"
def test_fetch_web_disk_full(http_url, tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
monkeypatch.setattr(sf, "check_disk_space", lambda dest_dir, needed: (False, 0))
with pytest.raises(FetchError) as ei:
_run(fetch_web(http_url, tmp_path / "out"))
assert ei.value.code == "DISK_FULL"
def test_fetch_web_truncated(tmp_path, monkeypatch):
"""断点/不完整下载:aiter_bytes 实收字节 < Content-Length → TRUNCATED。
用 fake httpx 确定性触发(真实服务器下 httpx 会因短读抛 HTTPError)。
"""
import types as _types
import httpx as real_httpx
from app.services.bioflow import source_fetcher as sf
class FakeResp:
status_code = 200
headers = {"content-length": "100"}
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
async def aiter_bytes(self, chunk_size):
yield b"short"
class FakeStream:
async def __aenter__(self):
return FakeResp()
async def __aexit__(self, *a):
return False
class FakeClient:
def __init__(self, follow_redirects=True):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
def stream(self, method, url, timeout=None):
return FakeStream()
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
AsyncClient=FakeClient, Timeout=real_httpx.Timeout, HTTPError=real_httpx.HTTPError,
))
with pytest.raises(FetchError) as ei:
_run(fetch_web("http://x/y.fa", tmp_path / "out"))
assert ei.value.code == "TRUNCATED"
def test_fetch_web_network_error(tmp_path, monkeypatch):
"""网络错误:httpx.HTTPError → NETWORK_ERROR(fake 确定性触发)。"""
import types as _types
import httpx as real_httpx
from app.services.bioflow import source_fetcher as sf
class FailingStream:
async def __aenter__(self):
raise real_httpx.HTTPError("connection reset")
async def __aexit__(self, *a):
return False
class FailingClient:
def __init__(self, follow_redirects=True):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
def stream(self, method, url, timeout=None):
return FailingStream()
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
AsyncClient=FailingClient, Timeout=real_httpx.Timeout, HTTPError=real_httpx.HTTPError,
))
with pytest.raises(FetchError) as ei:
_run(fetch_web("http://x/y.fa", tmp_path / "out"))
assert ei.value.code == "NETWORK_ERROR"
# ===========================================================================
# S0 补强(2)OSS:BAD_OSS / endpoint 解析(内网 http)/ fetch_oss 走 web + kind 修正
# ===========================================================================
def test_oss_url_bad_format():
with pytest.raises(FetchError) as ei:
_resolve_oss_url("oss://onlybucket")
assert ei.value.code == "BAD_OSS"
def test_oss_url_intranet_endpoint_http(monkeypatch):
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-internal.example.com")
url = _resolve_oss_url("oss://seq-bucket/data/x.fa")
assert url == "http://seq-bucket.oss-internal.example.com/data/x.fa"
def test_fetch_oss_marks_kind_and_provenance(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
from pathlib import Path as _Path
async def fake_web(url, dest_dir, timeout=None):
dest = _Path(dest_dir) / "x.fa"
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(">x\nACGT\n")
return {"ok": True, "path": str(dest), "name": "x.fa", "size": 4,
"kind": "web", "provenance": {"source": url, "kind": "web",
"sha256": "0" * 64, "fetched_at": "t"}}
monkeypatch.setattr(sf, "fetch_web", fake_web)
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-cn-hangzhou.aliyuncs.com")
r = _run(sf.fetch_oss("oss://b/k.fa", tmp_path / "out"))
assert r["ok"] is True
assert r["kind"] == "oss"
assert r["provenance"]["kind"] == "oss"
assert r["provenance"]["oss_spec"] == "oss://b/k.fa"
# ===========================================================================
# S0 补强(3)fetch_source:EMPTY_SOURCE / oss 路由 / aliyuncs 预签名路由
# ===========================================================================
def test_fetch_source_empty(tmp_path):
with pytest.raises(FetchError) as ei:
_run(fetch_source("", tmp_path))
assert ei.value.code == "EMPTY_SOURCE"
def test_fetch_source_oss_routing(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
calls = []
async def fake_oss(spec, dest_dir, timeout=None):
calls.append(spec)
return {"ok": True, "kind": "oss"}
monkeypatch.setattr(sf, "fetch_oss", fake_oss)
r = _run(sf.fetch_source("oss://bucket/k.fa", tmp_path))
assert r["kind"] == "oss"
assert calls == ["oss://bucket/k.fa"]
def test_fetch_source_aliyuncs_presigned_routes_to_oss(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
calls = []
async def fake_oss(spec, dest_dir, timeout=None):
calls.append(spec)
return {"ok": True, "kind": "oss"}
monkeypatch.setattr(sf, "fetch_oss", fake_oss)
presigned = "https://bucket.aliyuncs.com/x.fa?sig=abc"
r = _run(sf.fetch_source(presigned, tmp_path))
assert r["kind"] == "oss"
assert calls == [presigned]
# ===========================================================================
# S0 补强(4)局域网:目录 IS_DIR / scp 拉取 / sshpass 缺失 / scp 失败 / 解析边界
# ===========================================================================
def test_fetch_lan_dir_rejected(tmp_path):
d = tmp_path / "adir"
d.mkdir()
with pytest.raises(FetchError) as ei:
_run(fetch_lan(str(d), tmp_path / "out"))
assert ei.value.code == "IS_DIR"
def test_parse_lan_remote_relative_rejected():
assert _parse_lan_remote("host:relative/path") is None
def test_fetch_lan_scp_success(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
from pathlib import Path as _Path
async def fake_scp(*args, **kwargs):
dest = _Path(args[-1])
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(">g\nACGT\n")
class P:
returncode = 0
async def communicate(self):
return (b"", b"")
return P()
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
r = _run(fetch_lan("xzq@192.168.1.5:/data/genome.fa", tmp_path / "out",
credentials={"user": "xzq"}))
assert r["ok"] is True
assert r["kind"] == "lan"
assert r["name"] == "genome.fa"
def test_fetch_lan_scp_sshpass_missing(tmp_path, monkeypatch):
import shutil
from app.services.bioflow import source_fetcher as sf
monkeypatch.setattr(shutil, "which", lambda name: None)
with pytest.raises(FetchError) as ei:
_run(fetch_lan("host:/data/x.fa", tmp_path / "out", credentials={"password": "p"}))
assert ei.value.code == "SSHPASS_MISSING"
def test_fetch_lan_scp_failed(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
async def fake_scp(*args, **kwargs):
class P:
returncode = 1
async def communicate(self):
return (b"", b"scp: no such file")
return P()
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
with pytest.raises(FetchError) as ei:
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
assert ei.value.code == "LAN_ERROR"
def test_scp_fetch_remote_uses_port_and_user(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
from pathlib import Path as _Path
captured = {}
async def fake_scp(*args, **kwargs):
captured["args"] = list(args)
dest = _Path(args[-1])
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(">g\nACGT\n")
class P:
returncode = 0
async def communicate(self):
return (b"", b"")
return P()
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
_run(fetch_lan("host:/data/x.fa", tmp_path / "out",
credentials={"user": "u", "port": "2222"}))
# scp -P 2222 u@host:/data/x.fa <dest>
assert captured["args"][1] == "-P"
assert captured["args"][2] == "2222"
assert captured["args"][-2] == "u@host:/data/x.fa"
def test_fetch_lan_scp_timeout(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
async def fake_scp(*args, **kwargs):
class P:
returncode = 0
async def communicate(self):
return (b"", b"")
return P()
async def fake_wait_for(coro, timeout=None):
coro.close() # 未 await 的 communicate 协程关闭,避免 never-awaited 警告
raise asyncio.TimeoutError()
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
monkeypatch.setattr(asyncio, "wait_for", fake_wait_for)
with pytest.raises(FetchError) as ei:
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
assert ei.value.code == "LAN_TIMEOUT"
def test_fetch_lan_scp_no_file(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
async def fake_scp(*args, **kwargs):
# returncode 0 但未产出目标文件
class P:
returncode = 0
async def communicate(self):
return (b"", b"")
return P()
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
with pytest.raises(FetchError) as ei:
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
assert ei.value.code == "LAN_ERROR"
+10
View File
@@ -0,0 +1,10 @@
#!/bin/bash
# deploy/paper_restart.sh —— 论文撰写服务重启(8097,dev 树明文;不触碰 8090/8091/8098)。
# 用法:bash deploy/paper_restart.sh
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
bash "$REPO_ROOT/paper_writer_service/restart.sh"
echo "paper_writer_service 重启已触发(端口 8097,日志 paper_writer_service/paper.log)"
-141
View File
@@ -1,141 +0,0 @@
#!/usr/bin/env bash
# deploy/toggle_bioflow.sh —— 生信能力按需装卸(热,不重启 QA)+ bioflow_service 启停 + 注册状态自检。
#
# 用法:
# bash deploy/toggle_bioflow.sh <gene-family|rnaseq|extension|all> <on|off>
#
# 原理:
# - 能力装卸经 BFF POST /api/capabilities/install(透传 QA install_manager,热加载生效不重启 QA)
# - on = register(manifest 校验 + 原子写 capabilities.yaml + registry 差分刷新)
# - off = unregister(按 name 移除能力段,其余能力不受影响)
# - 自检 = GET /api/capabilities 的 installed 数组是否含对应能力
# - bioflow_service 启停:全部能力关闭时停 8098;任一开启时启动 8098
# - 卸载活跃任务:三表已持久化,服务恢复后 runner.recover 续跑;QA 侧按「生信服务未启动」降级话术
set -uo pipefail
BFF="http://127.0.0.1:8090"
BIOFLOW_SERVICE_DIR="/home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new/bioflow_service"
PY="/home/xzq/5090/teamWorkspace/zh/plantai-rag-new-venv/bin/python"
BIOFLOW_PORT="${BIOFLOW_PORT:-8098}"
CAP="${1:-}"
ACTION="${2:-}"
CAPS=(gene-family rnaseq extension)
usage() {
echo "用法: $0 <gene-family|rnaseq|extension|all> <on|off>" >&2
exit 2
}
# capability key -> registry 能力名
cap_name() {
case "$1" in
gene-family) echo "bioflow-gene-family" ;;
rnaseq) echo "bioflow-rnaseq" ;;
extension) echo "bioflow-extension" ;;
*) usage ;;
esac
}
# capability key -> register manifest YAML(与 agent/qa/configs/capabilities.yaml 一致)
manifest() {
case "$1" in
gene-family) cat <<'YAML'
capabilities:
- name: bioflow-gene-family
provider: mcp
permission: read_only
description: 基因家族分析(路径规划/建任务/参数确认/文件获取与预处理/状态查询),自然语言触发
session_default: true
nl_triggers: [基因家族, 家族成员, 亚家族, 系统发育, 建树, 进化树, 保守基序, motif, 结构域, 基因结构, 共线性, Ka/Ks, 选择压力, 组合证据]
mcp:
url: http://127.0.0.1:8098/mcp/gene-family/
prompt_pack:
file: bioflow_gene_family_agent_rules.md
YAML
;;
rnaseq) cat <<'YAML'
capabilities:
- name: bioflow-rnaseq
provider: mcp
permission: read_only
description: 转录组分析(实验设计确认/样本配置/路径规划/建任务/差异表达/富集),自然语言触发
session_default: true
nl_triggers: [转录组, RNA-seq, RNAseq, rnaseq, 差异表达, 表达定量, 富集, WGCNA, GSEA, 实验设计, fastq]
mcp:
url: http://127.0.0.1:8098/mcp/rnaseq/
prompt_pack:
file: bioflow_rnaseq_agent_rules.md
YAML
;;
extension) cat <<'YAML'
capabilities:
- name: bioflow-extension
provider: mcp
permission: read_only
description: 用户自定义扩展分析(创建/转换/试运行/校验/发布/运行),自然语言触发
session_default: true
nl_triggers: [扩展, 拓展分析, 扩展分析, 自定义分析, 分析模板, 分析脚本, 分析工具]
mcp:
url: http://127.0.0.1:8098/mcp/extension/
prompt_pack:
file: bioflow_extension_agent_rules.md
YAML
;;
*) usage ;;
esac
}
[ -n "$CAP" ] && [ -n "$ACTION" ] || usage
[ "$ACTION" = "on" ] || [ "$ACTION" = "off" ] || usage
if [ "$CAP" != "all" ]; then
cap_name "$CAP" >/dev/null 2>&1 || usage
fi
# 1. 逐能力 register/unregister(热加载不重启 QA)
targets=("$CAP")
[ "$CAP" = "all" ] && targets=("${CAPS[@]}")
for key in "${targets[@]}"; do
name="$(cap_name "$key")"
if [ "$ACTION" = "on" ]; then
body="$(manifest "$key")"
resp=$(curl -s -m 30 -X POST "$BFF/api/capabilities/install" \
-H 'Content-Type: application/json' \
-d "$(python3 -c 'import json,sys; print(json.dumps({"action":"register","manifest":sys.stdin.read()}))' <<<"$body")")
else
resp=$(curl -s -m 30 -X POST "$BFF/api/capabilities/install" \
-H 'Content-Type: application/json' \
-d "{\"action\":\"unregister\",\"name\":\"$name\"}")
fi
echo "[$ACTION] $name -> $(echo "$resp" | head -c 300)"
done
# 2. 自检:GET /api/capabilities 的 installed 数组
installed=$(curl -s -m 10 "$BFF/api/capabilities")
echo "注册状态: $(echo "$installed" | python3 -c 'import json,sys; d=json.load(sys.stdin); print([i.get("name") for i in d.get("installed") or []])' 2>/dev/null || echo 'parse-fail')"
# 3. bioflow_service 按需启停:任一能力开启则启动,全部关闭则停止
any_on=false
for key in "${CAPS[@]}"; do
name="$(cap_name "$key")"
if echo "$installed" | grep -q "\"$name\""; then
any_on=true
fi
done
if [ "$any_on" = true ]; then
if ! lsof -ti tcp:$BIOFLOW_PORT -sTCP:LISTEN >/dev/null 2>&1; then
echo "启动 bioflow_service :$BIOFLOW_PORT"
( cd "$BIOFLOW_SERVICE_DIR" && setsid "$PY" -m uvicorn app.main:app --host 127.0.0.1 --port "$BIOFLOW_PORT" \
> /tmp/plantai-bioflow.log 2>&1 < /dev/null & ) && disown -a
sleep 2
fi
curl -s -m 8 "http://127.0.0.1:$BIOFLOW_PORT/api/bioflow/health" | head -c 150
echo
else
echo "全部能力已卸载:停止 bioflow_service :$BIOFLOW_PORT"
kill $(lsof -ti tcp:$BIOFLOW_PORT -sTCP:LISTEN) 2>/dev/null || true
fi
echo "DONE"
+51
View File
@@ -0,0 +1,51 @@
#!/bin/bash
# deploy/toggle_paper.sh —— 论文撰写快速开关(dev 树 175;不触碰生产 176 /srv/plantai)。
# 用法:bash deploy/toggle_paper.sh off|on
# 原理:BFF main.py 按 PAPER_WRITER_ENABLED 决定是否注册 /api/paper 路由;
# 不注册时 /api/paper/health 返回 404,前端探活落 offline,触发词落 docgen 单发降级。
set -uo pipefail
ACTION="${1:-}"
SDK_DIR="/home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new/sdk"
ENV_FILE="$SDK_DIR/.env"
PY="/home/xzq/5090/teamWorkspace/zh/plantai-rag-new-venv/bin/python"
BIND="${PAPER_BFF_BIND:-0.0.0.0}"
if [ "$ACTION" != "off" ] && [ "$ACTION" != "on" ]; then
echo "用法: $0 <off|on>" >&2
exit 2
fi
VAL="true"
[ "$ACTION" = "off" ] && VAL="false"
# 1. 改 env(追加或覆盖)
if grep -q "^PAPER_WRITER_ENABLED=" "$ENV_FILE"; then
sed -i "s/^PAPER_WRITER_ENABLED=.*/PAPER_WRITER_ENABLED=$VAL/" "$ENV_FILE"
else
echo "PAPER_WRITER_ENABLED=$VAL" >> "$ENV_FILE"
fi
echo "PAPER_WRITER_ENABLED -> $VAL"
# 2. 重启 BFF 8090(dev 树;仅取 LISTEN 者;venv 必须绝对路径,cd sdk 后才能 import app.main)
kill $(lsof -ti tcp:8090 -sTCP:LISTEN) 2>/dev/null || true
sleep 2
cd "$SDK_DIR" && set -a && . .env && set +a
( cd "$SDK_DIR" && setsid "$PY" -m uvicorn app.main:app --host "$BIND" --port 8090 \
> /tmp/plantai-bff.log 2>&1 < /dev/null & ) && disown -a
# 3. 自检
for i in $(seq 1 20); do
sleep 2
curl -s -m 3 http://127.0.0.1:8090/api/health >/dev/null 2>&1 && break
done
curl -s -m 10 http://127.0.0.1:8090/api/health | head -c 120
echo
if [ "$ACTION" = "off" ]; then
CODE=$(curl -s -o /dev/null -w "%{http_code}" -m 8 http://127.0.0.1:8090/api/paper/health)
echo "paper/health HTTP $CODE (off 期望 404)"
[ "$CODE" = "404" ] && echo "OFF_OK" || echo "OFF_CHECK_FAIL"
else
curl -s -m 8 http://127.0.0.1:8090/api/paper/health | head -c 150
echo
fi
-177
View File
@@ -1,177 +0,0 @@
# 生信流程 · 对话式手工测试手册(S6 更新版)
> 供用户逐项实测。生信模块已从「卡片式交互」改造为「自然语言对话式交互 + MCP 外挂」,
> 本手册以**对话剧本**形式给出每个场景的:对话示例 → 系统应如何回应 → 通过判据。
> 环境:内网 http://192.168.0.175:8090(BFF);生信服务 8098、QA 8091 已部署;
> 三能力(基因家族/转录组/扩展)按 §0 装卸。
---
## 0. 前置条件
1. 浏览器打开 http://192.168.0.175:8090。
2. **能力安装检查**:设置面板(SettingsPanel)应能看到三个生信开关(基因家族/转录组/扩展)。
打开任意一个后,工作区出现「生信任务」「生信档案」TAB;全关则 TAB 消失(**TAB 显隐 = 能力注册状态**,非服务健康)。
3. 服务器 docker 已有 8 个 bioflow 镜像;金标数据在 `bioflow_kernel/assets/examples/test_input/`(Ft 四件套 pep/cds/genome/gff3)。
4. 熔断开关:`QA_V1_ENABLE_BIOFLOW_AGENT=1`(默认 0,关时生信句走普通 QA)。
---
## 1. 多配置开关回归(S6)
| 配置 | 操作 | 期望 | 判据 |
|---|---|---|---|
| 全关(熔断=0 + 三能力卸载) | 发「帮我做基因家族分析」 | 普通 QA 礼貌回答(说明未安装生信能力),**无**路径卡 | 无生信分流 |
| 只开基因家族 | 发「帮我做基因家族分析」 | 进入生信对话(列路径+挂路径卡) | 命中 |
| 只开基因家族 | 发「帮我做转录组差异表达」 | 走普通 QA(转录组能力未装) | 不响应 |
| 只开转录组 | 发「帮我做转录组」 | 进入生信对话 | 命中 |
| 只开转录组 | 发「帮我做基因家族分析」 | 走普通 QA | 不响应 |
| 全开但非生信提问 | 发「光合作用的电子传递链是什么」 | 走普通 QA(非生信意图不调工具) | 不调工具 |
---
## 2. 对话剧本 A:基因家族分析(卡片辅助通道)
**第 1 轮 —— 需求与路径**
> 用户:帮我做水稻 SPL 基因家族分析
> 系统(自然语言):列「核心流程(必选)+ 可选分析」,说明每个可选项生物学意义;
> **同帧挂出路径选择卡**。
> 判据:回复含路径列表 + 路径卡出现(卡片不替代自然语言)。
**第 2 轮 —— 卡片操作 → 系统复述**
> 用户(路径卡勾选「核心流程 + Ka/Ks + 启动子」,点确认;打字补充文件位置)
> 系统:先自然语言**复述**用户选择("您选择了核心流程 + Ka/Ks + 启动子分析…"),再请确认。
> 判据:系统复述内容与卡片勾选一致;建任务前必有一轮复述确认(不直接建任务)。
**第 3 轮 —— 网络文件 + 非标准文件预处理**
> 用户:文件在 `https://…/rice.fa` 与 `oss://seq-bucket/rice.gff3`,另传 `spl_list.xlsx`
> 系统:拉取 web/OSS 文件(**下载前检查磁盘余量、记录 provenance**);对 `spl_list.xlsx`
> 做内容嗅探后**自然语言说明列结构并提问**("推测第二列是蛋白序列,备注列忽略?")。
> 判据:文件获取成功;Excel 触发预处理提问(列表方式,可卡片点选也可打字回答)。
**第 4 轮 —— 建任务执行**
> 用户:都对,备注列不要;确认,开始吧
> 系统:复述整理结果("已整理为 FASTA 3 条序列")→ 创建任务 → 消息内出现**只读精简进度块**,
> 右侧「生信任务」TAB 完整滚动日志实时跟随。
> 判据:进度块只读(无按钮);TAB 日志自动滚底。
**执行中 —— 进度转述 + 随时提问**
> 用户:什么是组合证据?
> 系统:经流程知识库自然语言回答(带引用),**不打断任务**。
> 判据:提问得到引用回答,任务继续推进。
**人工确认边界(不可跳过)**
> 系统:第 8 步亚家族预测完成(6 个亚家族)。⚠ 需您人工确认,我不能代您确认。
> 用户:把 SPL3 从亚家族 A 移到 B → 系统复述修改 → 用户「确认」。
> 判据:subfamily_confirm 必停、必复述、绝不代确认。
**收尾 —— 存档**
> 系统:全部完成(19 成员 6 亚家族),同帧挂出存档卡。
> 用户(存档卡填名称点「存档」或打字「存档,叫 SPL 水稻鉴定」)
> 系统:实际执行 `archive_job` 存档(复用 REST 语义),复述存档三要素(名称/位置/大小)。
> 判据:存档成功(重名提示 ARCHIVE_EXISTS)。
---
## 3. 对话剧本 B:转录组分析(纯自然语言通道,全程不点卡片)
**第 1 轮 —— 必确认信息**
> 用户:帮我做转录组,比较干旱处理和对照
> 系统:列主流程 + 可选;**必须确认**四项(物种/参考基因组版本、FASTQ 单双端、
> 对照处理组重复数、链特异性),列表展示。
> 判据:实验设计确认边界被明确列出,未确认前不得建差异表达任务。
**第 2 轮 —— OSS 文件 + 样本信息(纯打字,不点卡)**
> 用户:数据在 oss://seq-bucket/drought/,水稻日本晴,干旱 3 重复对照 3 重复,双端非链特异
> 系统:从 OSS 拉取 12 个 FASTQ 并核对成对;复述样本设计("干旱 3 vs 对照 3,双端,非链特异"),
> 请确认基线(对照为基线、比较方向「干旱 vs 对照」)。
> 判据:纯自然语言路径走通(**卡片与打字两条通道服务端语义一致**)。
**第 3 轮 —— 执行 + 结果转述**
> 用户:对 → 系统执行 QC→比对→定量→样本质控→差异表达→富集,逐段转述(接头率/Q30/唯一比对率/差异基因数/PCA 分离)。
> 判据:每步完成后自然语言转述关键指标;样本质控 PCA 结论有转述。
---
## 4. 三种网络文件来源用例
| 来源 | 示例 | 期望 |
|---|---|---|
| web | 「文件在 https://…/x.fa」 | httpx 流式下载,不设大小上限;下载前查磁盘余量;记录 provenance |
| OSS | 「数据在 oss://bucket/key 或预签名 URL」 | 经 BIOFLOW_OSS_ENDPOINT 解析或预签名直通;无 endpoint 时提示改用预签名 |
| 局域网 | 「文件在 192.168.x.x:/data/x.fa,账号 xx 密码 xx」 | ssh/scp 拉取(凭据单次会话内存用,不落盘、审计不记明文) |
判据:文件落 `bioflow_data/jobs/{job}/input/`,先内容嗅探后登记,provenance 记录来源/哈希/时间。
---
## 5. 非标准文件预处理(Excel / 乱码文本)闭环
1. 上传 `xxx.xlsx` / 乱码 txt → 系统 `inspect_file` 探测(列结构/格式猜测/乱码检测)。
2. 系统自然语言说明 + 列表提问(哪列是序列、是否去重、是否去表头)。
3. 用户回答 → `transform_file` 转换(LLM 定规则 + 脚本执行,大文件抽样定规则)。
4. 结果过格式验证器(fasta 可解析/gff3 9 列/tsv 列一致/samples.yaml schema);
失败说明问题并追问,循环 ≤3 次后建议人工整理。
5. 转换后 `register_job_files` 登记,provenance 记「原始→规则→产物」链。
判据:全程自然语言引导;转换产物格式校验通过;原文件永不覆盖。
---
## 6. 卡片辅助 vs 纯自然语言双通道等价
| 场景 | 卡片通道 | 纯 NL 通道 | 等价判据 |
|---|---|---|---|
| 路径选择 | 路径卡勾选 | 打字「核心流程 + Ka/Ks」 | 两者复述的确认内容一致,建任务入参一致 |
| 参数确认 | 参数卡填值 | 打字「用 8 核、padj 用 0.01」 | confirm_stage 入参一致 |
| 文件提供 | 文件卡拖拽/填路径 | 打字「文件在 …」 | register_job_files / fetch_source_file 入参一致 |
| 存档 | 存档卡点存档 | 打字「存档,叫 X」 | archive_job 入参一致 |
判据:卡片操作产生的是 `card_result` 结构化消息进 QA 流(**不直连 bioflow 控制面**),
与打字在服务端语义完全一致(QA 对话状态机是唯一决策点)。
---
## 7. 旧会话兼容(Legacy 块只读)
1. 打开一个改造前创建、含旧生信块(bioflow_qa / 旧 bioflow_* 块)的会话。
2. 期望:旧块以只读形式降级渲染(标题 + 只读引用/提示),**不报错、无可交互控件**。
---
## 8. 扩展子系统对话化(第三能力单元)
1. 打开「扩展」能力开关。
2. 对话创建扩展:描述分析脚本(代码文件 + 运行命令)→ `create_extension`(draft)。
3. `build` → `trial_run` → `validate` → `confirm` → `publish` → `run` 逐步走(卡片或打字等价)。
4. 试运行失败 → 「AI 修复」触发 `repair_extension`(修复上限 20 轮);校验失败/状态不足时系统给出确定性提示。
判据:扩展生命周期全程对话化;非法状态返回清晰提示(EXT_STATUS),不报 500。
---
## 9. 后端金标验证(服务器命令行,S6 回归)
```bash
cd /home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new
BIOFLOW_DATA=/home/xzq/5090/teamWorkspace/zh/bioflow-data \
bash bioflow_kernel/tests/smoke_golden.sh bioflow-gf:dev
```
期望:4 步全绿——python 3.11.14 / ldd 0 缺库 / 无 conda / 金标 identify 24 ID diff 空(`GOLDEN_DIFF_OK 24 IDs`)。
---
## 10. 端到端主线(一次性走全流程,双通道各一遍)
1. 卡片通道:§2 全程(路径卡→文件卡→参数卡→进度块→存档卡)。
2. 纯 NL 通道:§3 全程(不点任何卡片)。
3. 含一次 Excel 预处理、一次局域网/OSS 文件拉取。
4. 任务完成后存档 → 生信档案 TAB 回看 → 复用发起。
期望:两条通道均走通,7 项引导(路径/文件/参数/报告/存档/复用/过程展示)逐项达成,
进度块实时流转、右侧日志滚底。
---
*本手册为 S6 对话式改造更新版,替代旧卡片式 `MANUAL_TEST_BIOFLOW.md` 的生信流程部分。*
@@ -0,0 +1,42 @@
# feat(review): journal_directory 启动自动 seed(部署即运行)
## 背景
source_v4.xlsx(1056 条目录)此前需手动跑 import_journals.py 入库,部署后忘跑即出现"目录数据未导入"。本次实现启动时自动检测 + 自动导入。线上当前环境已由人工完成导入(review.db 内 1056 行),本功能消除的是新环境部署/重建库时的手动步骤风险。
## 修改内容
| 文件 | 变更 |
|---|---|
| `app/services/journal_seed.py`(新增) | 共享核心:schema/清洗/解析/upsert 从 CLI 原样迁入;`extract_rows_from_xlsx()`(口径 779/277 与 id 冲突闸门,不符不写库)、`upsert_to_db()`、`ensure_journal_directory_seeded()`(空表才导,四键 dict 契约,绝不抛异常) |
| `scripts/import_journals.py` | 瘦身为薄 CLI(-431/+92 行):对账报告/退出码 0/1/2/3 全保留,核心复用 journal_seed |
| `app/config.py` | +`seed_on_startup=True`(REVIEW_SEED_ON_STARTUP 可关)、+`journal_directory_seed_path` |
| `app/main.py` | lifespan:开关门控 → `asyncio.to_thread` 导入(不堵 event loop)→ try/except 不阻塞启动;路径走 `db.path` 注入链 |
| `tests/conftest.py`(新增) | 全局关 seed,防止单测触碰真库/拖慢套件 |
| `tests/test_seed.py`(新增) | 6 用例(见下) |
不动:source_v4.xlsx、journal_directory 路由/表 schema、db.py、前端。
## 测试用例与结果(pytest 78 passed,新增 6 条)
1. 空库 → seed 成功,rows=1056(en=779/zh=277)✓
2. 非空库(1 行)→ skip,count 不变 ✓
3. xlsx 不存在 → 返回 error,不抛异常,不写库 ✓
4. 幂等:连调两次,第二次 skip ✓
5. 开关 off(lifespan)→ 表不创建 ✓
6. 开关 on(lifespan)→ tmp 库 1056 行 ✓
## 部署验证
```bash
bash review_service/restart.sh # 非空库:日志 "already has 1056 rows, skip"
curl 'http://127.0.0.1:8100/api/review/journals/directory?lang=en' # total=779
# 空库冷启动(本机已验证等价路径):日志 "seeded 1056 rows (en=779, zh=277, N.Ns)"
# 特殊部署:REVIEW_SEED_ON_STARTUP=false → 日志 "seed skipped"
```
本机已实测:真机重启走 skip 路径(`INFO:app.main:already has 1056 rows, skip`),CLI 薄壳回归 exit=0 / `en=779 zh=277 → ✓ 相符`。
## 是否需要立即部署
待管理员确认。线上已手动导入 1056 行,本改动属部署便利性补强(消除新环境忘跑导入脚本的风险),可随下次正常发版评估带上。
@@ -0,0 +1,56 @@
# fix(review): 冲突降级真影响 recommendation(major → minor 真降级)
- 分支:`feature/academic-review-assistant`
- 日期:2026-10-12
- 服务:`review_service`(:8100)/ 前端 `PresubmitPanel.vue`
- 状态:**未 commit**,待验收
## 背景
s41588(已发表 Nature Genetics 论文)真实链路里,关键词扫描器报告"3 处冲突已降级为非硬性问题",但 `recommendation` 仍为大修、`major_weaknesses` 仍 5 条。诊断确认:原降级只做了"剥 hard_rule 标签 + 记 conflicts",**条目仍在 major 列表内**,`major_count` / 引擎输入未变,故判定不变。本次把"降级"从标签维度升级为**列表维度的真降级**(major → minor)。
## 变更
**后端**
- `paper_scanner.reconcile_hard_rules`:命中冲突的 major 从列表**移出**,`hard_rule=None` + `scanner_downgraded=true`,落入 demoted 列表;`fatal=true` 条目例外(只记 conflict 不降级,须人工过目)。
- `routers/presubmit.py`:demoted 追加进 `report.minor_weaknesses`;`major_weaknesses` 为过滤后列表;explanation 文案改"已降级为小修级问题"。
- `self_review.WeaknessItem`:新增 `scanner_downgraded: bool = False`(coerce 不读、LLM 无法伪造)。
- 引擎 `recommend_engine.decide_recommendation` **零改动**——计数因列表变短而自动真实。
**前端**
- `PresubmitPanel.vue`:minor 卡片对 `scanner_downgraded` 显示黄徽标"冲突降级"(与 major 卡片"硬性"红徽标对仗);冲突明细块保留。
## 测试
- pytest **104 passed**(新增:`test_reconcile_fatal_not_demoted`、`test_http_partial_conflict_leaves_residual_soft`;改写 `test_reconcile_*` 与 `test_http_scanner_conflict_downgrades` 为真降级断言)。
- vitest **382 passed / 5 failed**(5F 为既有他人 `benchpanel`+`Composer .mode-pill` labels.js 引发,与本改动无关);`npm run build` ✓。
## 验收(真 LLM,:8100)
| 样本 | rec | rule | M/S/minor | conflicts | downgraded→minor |
|---|---|---|---|---|---|
| diag_strong | minor_revision | R5 | 0/0/4 | 0 | 无 |
| diag_weak | major_revision | R1 | 6/2/7 | 1(hic) | 有(真降级生效) |
弱稿仍大修:4 条硬性命中(busco/accession/voucher/abs)为真缺,扫描器无冲突、不降级——真缺陷不放行。
## 验收:s41588(真 LLM,:8100)
输入 `/tmp/opencode/s41588-025-02113-5.pdf`,target_journal=Genome Biology(tier 1 override)。
| 项 | 修复前 | 修复后 |
|---|---|---|
| recommendation | major_revision | **minor_revision**(R5) |
| major_count / soft / minor | 5 / — / — | **0 / 0 / 6** |
| hard_hits | 含 busco/accession… | **[]** |
| scanner signals | — | accession_found=True, busco=partial, hic_found=True |
| major_weaknesses 列表 | 5 条 | **0 条** |
| scanner_conflicts | — | **2 条保留**(busco_missing 格式不完整、accession_missing 部分未给号)→ 均真降级进 minor(`scanner_downgraded=true`) |
判定链:扫描器证伪 2 条硬性 → 真降级 → 剩余为 minor → R5(tier1 accept 门槛 minor≤1,6 条 → minor_revision)。conflicts 透明保留、explanation 明示"已降级为小修级问题,请人工核对"。
注:本次 LLM 未再把"树图注未标数值 / GWAS 校正描述"判成 major(负向清单 Prompt 生效),故 major 从 5 归 0,优于预期的"降到 2"。
## 部署
待管理员确认。属判定正确性修复(真论文误判大修),建议随下次正常发版带上。
@@ -0,0 +1,81 @@
# fix(review): presubmit 四 Bug 修复——截断保真 / 分档 / 词表 / 扫描器兜底
## 背景
真实案例:已发表 Nature Genetics 论文(s41588-026-02682-z,黄瓜泛基因组)被判
"大修",但论文实有 BUSCO 98.57%(P2)、94.18% sequences anchored(P2)、
PRJCA038097 等 accession(P17 Data Availability)。诊断出四个独立缺陷:
1. **PDF 截断**:`max_text_chars=60000` 从头切,30 页论文 P17 的 Data Availability 被截掉;
2. **期刊分档**:Nature Genetics 不在 5 本精选 → 默认 tier 2(应 tier 1);
3. **硬性规则词表过窄**:缺 PRJCA/CRA(中国 NGDC/GSA)、Hi-C 同义词只认 anchoring rate、
BUSCO 仅百分比也算缺失;
4. **无程序层兜底**:hard_rule 全靠 LLM 输出,无关键词证伪机制。
## 修改内容
| 文件 | 变更 |
|---|---|
| `app/routers/presubmit.py` | **Bug 1**:段落感知装箱 `pack_feed_text`(超预算按
Abstract→Data Availability 全文→Methods 前 60%→Results→Discussion 优先保留,
dropped_sections 入 meta);无节标题回退头+尾窗(捞页尾 availability 块)。
**双层保护**:估算 token(len/1.5)>48K → 自动降 100K 重装;LLM 400/413 → 降 80K
重试一次。**Bug 4 接线**:coerce 后扫描器证伪 → rationale.scanner_conflicts |
**Bug 2**:tier_source ∈ override/featured/jif/默认 |
| `app/services/paper_scanner.py`(新增) | 关键词扫描器(纯 stdlib):accession
词表 PRJNA/PRJEB/PRJCA/SRP/ERP/DRP/CRA/SRR/ERR/DRR/GCA_/GCF_ **只在 Data Availability
段扫**(正文引用他人数据不误判);BUSCO 三级 none/partial/full;Hi-C 同义词
(anchored to chromosome(s)/anchoring rate/pseudo-chromosome(s)/chromosome-level)。
`reconcile_hard_rules`:LLM 报缺 X + 扫描器见 X → 剥硬性标签降 soft + 记冲突
(只证伪不证实,不引入新误判) |
| `app/services/recommend_engine.py` | **Bug 2**:tier 查找优先级 override
(13 本顶刊硬编码,"The " 前缀归一化)→ featured(journals.json)→ jif 启发式
(journal_directory 只读 jif 列;jif≥15→tier1;综述刊名含 Review/Trends/Annual
Review 除外)→ 默认 tier 2 |
| `app/services/self_review.py` | **Bug 3**:Prompt 词表扩充(PRJCA/CRA/SRR…);
BUSCO 三级判定(仅百分比→minor 不打 hard_rule);Hi-C 同义词清单。
**2026-10-12 追加 soft major 收紧**(s41588 复盘):维度三系统发育树分诊重写
(完全未做→major / 已声明仅图未标数值→presentation / 附带树→minor);新增
【负向清单】节(图未标统计量、标准方法描述不全、阈值缺数值、引用排版 →
minor/presentation,严禁列 major;总纲"只有补实验/补数据/重跑分析/改核心论证
才是 major";与各维度冲突时以本节为准) |
| `app/config.py` | `max_text_chars` 60000 → **150000**(REVIEW_MAX_TEXT_CHARS 可覆盖) |
| `ui/.../PresubmitPanel.vue` | 冲突黄标块(scanner_conflicts + 降级提示);截断
dropped_sections 美化提示;截断文案改为"保摘要/数据可用性/方法/结果/讨论" |
| `tests/test_presubmit.py` | 新增 15 用例(装箱/回退/token 守卫/400 重试/扫描器
三类信号/DA 段限定/冲突降级/端到端/tier 优先级矩阵/Prompt 新词表) |
## 测试用例与结果
- **pytest:102 passed**(87 基线 + 15 新增)。
- **vitest:379 passed / 5 failed**(5F 为既有他人 i18n labels 改动引发,与本改动
无关;新增冲突黄标/截断提示 2 条全绿)。`npm run build` ✓。
- **真实链路验收**(真 LLM,target_journal=Genome Biology → tier 1 override):
| 样本 | 修复前 | 修复后 | 扫描器 |
|---|---|---|---|
| 强稿 diag_strong.pdf | minor_revision | **minor_revision**(R5,0/0/5) | accession/busco=full/hic 全命中,0 冲突 ✓ |
| 弱稿 diag_weak.pdf | major_revision | **major_revision**(R1,硬性 4 + soft 4 + minor 5) | accession/busco 未命中(真缺不降);Hi-C 冲突降级并记录 ✓ |
| **真实案例 s41588**(NG 论文) | major_revision | 见下"验收"节 | — |
## 部署验证
```bash
bash review_service/restart.sh
curl -X POST http://127.0.0.1:8100/api/review/presubmit \
-F "file=@paper.pdf" -F "target_journal=Nature Genetics"
# 检查:meta.tier=1/tier_source=override;meta.dropped_sections;
# report.recommendation_rationale.scanner_conflicts
```
## 是否需要立即部署
待管理员确认。四修复均为判定正确性问题(真实发表论文误判大修),建议随下次正常发版带上。
## 遗留
- tier jif 启发式仅覆盖 journal_directory 已录入的 1056 刊;未收录刊名走默认 tier 2。
- 扫描器 accession 仅扫 DA 段——若论文把 accession 放 Methods 数据声明处且无 DA 节,
扫描器不证伪(保守方向,LLM 硬性判定仍生效,不会误放行)。
- BUSCO full 判定要求 S: 与 D: 标记在 200 字符窗内相邻——极端排版可能判 partial
(只会更严,不误放)。
@@ -0,0 +1,61 @@
# fix(review): presubmit recommendation 判定过严修复(配额制 → 规则引擎)
## 背景
用户反馈 AI 预审稿多次测试均返回 major_revision。2026-10-12 诊断确认根因是
**配额制**而非"尺子紧":
- `self_review.py` 硬性规则要求 `major_weaknesses 3-8 条`(强制配额)→ accept/minor_revision 几乎不可达;
- LLM 直接输出 recommendation,无程序层规则;JSON 示例锚定 `major_revision`、coerce 缺省回落 `major_revision`;
- 对照实验:近 GB 完备强稿(BUSCO 96.8% / Hi-C 98.7% / PRJNA+GCA+figshare 齐 / UFBoot 1000)仍被凑出 6 条 major,与弱稿(7 条)几乎无区分度;
- `target_journal` 传入但 Prompt 零分档标准。
## 修改内容
| 文件 | 变更 |
|---|---|
| `app/services/recommend_engine.py`(新增) | 纯函数规则引擎:R0 fatal→reject / R1 硬性 major→大修(不可协商,与 tier 无关)/ R2' tier3 仅 1-2 条 soft 可降小修 / R2 soft≥1→大修 / R4 0 major 且 minor≤tier 阈值→接收 / R5→小修;rationale 模板拼装(rule/hard_hits/计数/tier/explanation,含 tier 降级提示句);`resolve_tier` 按 journals.json `review_tier` 匹配,失败默认 tier 2 |
| `app/services/self_review.py` | Prompt 重写:严重分级标准(major/minor/presentation),**三类均不设下限、可 0 条、严禁凑数**;硬性规则 5 项打标签(busco/hic/accession/voucher/abs_missing);删除 recommendation 输出字段与示例锚定;coerce 不再读 recommendation(防注入),hard_rule 白名单过滤 |
| `app/routers/presubmit.py` | coerce 后接引擎;meta 增加 tier/tier_label/tier_source |
| `data/journals/journals.json` | 精选 5 本加 `review_tier`(GB/MP=1;BMC PB/FPS/Rice=2)+ notes 一行 |
| `ui/.../PresubmitPanel.vue` | 判定依据块(explanation + 硬性 chips);判定档位徽标;reject 红标"建议人工复核";顶部 AI 免责行;major 卡片"硬性/fatal"徽标 |
| `tests/test_presubmit.py` | 用例 1-10(引擎分支全覆盖 + tier 对照 + 注入忽略 + rationale 字段) |
判定链与旧链路唯一对外契约变化:`report.recommendation` 语义不变(四档枚举),
新增 `report.recommendation_rationale` 与 `meta.tier*`。
## 测试用例与结果
- **pytest:87 passed**(78 基线 + 9 新增/重写),含用例 1-10:
强稿形态不判大修、弱稿形态判大修且 hard_hits 正确、0 major 0/多 minor 分支、
soft major 分支、fatal→reject、tier3 降级与硬性豁免、tier1 accept 门槛、
LLM recommendation 注入被忽略、hard_rule 白名单过滤、rationale 含 tier/explanation。
- **vitest:377 passed / 5 failed**(5F 为既有他人 i18n labels 改动引发的 benchpanel
失败,与本改动无关;新增 4 条 rationale/档位/fatal 展示用例全绿)。
- **真实链路验收**(诊断同款样本、同 target_journal=Genome Biology,真 LLM 调用):
| 样本 | 修复前 | 修复后 | 依据 |
|---|---|---|---|
| 弱稿 diag_weak.pdf | major_revision(7 major) | **major_revision**(R1;硬性 5 命中:BUSCO/Hi-C/Accession/voucher/ABS + soft 2 + minor 6) | 真缺陷不漏判 ✓ |
| 强稿 diag_strong.pdf | major_revision(凑出 6 major) | **minor_revision**(R5;0 major / 5 minor) | 区分度恢复 ✓ |
## 部署验证
```bash
bash review_service/restart.sh
# 上传强/弱样本对比(或任意 PDF):
curl -X POST http://127.0.0.1:8100/api/review/presubmit \
-F "file=@paper.pdf" -F "target_journal=Genome Biology"
# 检查 report.recommendation_rationale.rule / meta.tier;前端看"判定依据"块
```
## 是否需要立即部署
待管理员确认。属判定质量修复(误报大修影响用户体验与信任),建议随下次正常发版带上。
## 遗留(Phase 2 候选)
- 程序层兜底 `_sanity_check_missing_hard_rules`:正文无 PRJNA/GCA_/SRP 关键词但 LLM
未报 accession_missing → 追加 warning(不进 recommendation 计算);先观察 LLM 是否
"躺平"再决定。
- tier 数据仅覆盖精选 5 本;扩展到 journal_directory 1056 条需人工核验后另行入库。
@@ -0,0 +1,161 @@
# PR 报告:Rebuttal Phase 1(审稿回复助手)
- 日期:2026-10-12
- 分支:`feature/academic-review-assistant`(未 commit,按用户约定统一处理)
- 范围:`review_service/` 新增 5 文件 + main 注册;`ui/` 新增面板与 store/api 扩展;无 sdk/、agent/、vstore/ 改动;无新增依赖。
## 功能概述
粘贴审稿意见文本(主入口)/ 上传批注版 PDF(次要入口)→ LLM+正则拆条(Reviewer→Comment)
→ 逐条填回复要点(5 类下拉,无 auto)→ 生成草稿(反幻觉校验前置)
→ 逐条勾确认 → 拼装 response letter → 下载 `.md`(human_ack 门槛)。docx 属 Phase 2。
## 10 个 API 端点(review_service:8100;BFF 8095 通用透传已验证可用)
| 方法 | 路径 | 说明 |
|---|---|---|
| POST | /api/review/rebuttal/jobs | 文本粘贴主入口 |
| POST | /api/review/rebuttal/jobs/pdf | PDF 次要入口(vstore 解析;非 PDF/超限→VALIDATION) |
| GET | /api/review/rebuttal/jobs/{id} | 全量还原 |
| PATCH | .../comments/{cid}/response | 保存 parts(改后旧 draft/validation 作废) |
| POST | .../comments/{cid}/draft | 生成草稿(校验C前置;MISSING_RESPONSE 时不调 LLM) |
| PATCH | .../comments/{cid}/structure | 手工 merge(≥2)/ split(≥2 非空子项) |
| PATCH | .../comments/{cid}/confirm | 逐条勾;后端重算 human_ack(全勾才 true,空 job 不置位) |
| POST | .../letter | 拼装 letter_md |
| GET | .../letter.md | 下载(human_ack=false → 400 HUMAN_ACK_REQUIRED) |
| DELETE | .../jobs/{id} | 手动删(替代 TTL) |
信封错误复用约定:HTTP 200 + `{ok:false,error:{code,message,degraded}}`。
## 数据模型
- Job:`{id,status,source:{kind,filename,raw_text},template:'generic',reviewers:[...],letter_md,validation,human_ack,created_at,updated_at}`
- Comment:`{id,group_id,severity,title,body,sub_items[],cross_refs[],response:{parts[]},draft,validation,human_confirmed}`
- part.type 枚举:`new_data | reanalysis | location | clarification | rebuttal`(决策 2:无 auto)
- SQLite 表 `rebuttal_jobs` 落在既有 `review_service/review.db`(自含短连接,不动 app/db.py);
启动幂等建表(main lifespan)。
## 关键实现决策(方案未写清处,按"保守、可测、不猜测")
1. **LLM 注入形态**:parse/draft 用 `async (messages, temperature) → dict` 回调,路由内适配
`llm.complete_json(system, user)`;单测直接 monkeypatch `llm.complete_json`。
2. **拆条双轨**:正则先切骨架(Reviewer 段头 + 编号行,兼容行内 `. 2.` 编号);
LLM 逐段出 `{title,severity,sub_items[]}`。LLM 失败/结构非法 → 该段回退正则
(title=首 60 字,severity=major)。`sub_items ≥2 且 max confidence ≥0.7` 才拆
(决策 3),拆出条目共享 group_id;未达阈值多子问题并回一条且 **body 保留段原文**
(不采信 LLM 改写)。
3. **交叉引用**:`see comment N / 参见第 N 条 / 同上` → 解析目标 id;解析不到
`target_id=null` 保留 raw(不猜)。
4. **反幻觉校验(负向词表收窄版)**:
- location:`shows that/as shown` 且无 `added/新增` → `LOCATION_EVIDENCE_WEAK` FAIL;
- rebuttal:命中 `we agree/accept/concede/you are right/we were wrong` →
`REBUTTAL_NEGATIVE` FAIL;无立场词(maintain/disagree/respectfully…)→
`REBUTTAL_NO_STANCE` FAIL;
- 其余三类 Phase 1 仅查非空。校验在 draft 端点前置,FAIL 不调 LLM。
5. **human_ack**:confirm 端点改完后由 store 重算(所有 comment 全 true 才置位;
空 job 不置位防平凡真)。前端无法直接置位。
6. **letter.md 下载**:纯文本端点(非信封),门槛失败返回 400 + 明文
`HUMAN_ACK_REQUIRED:…`;前端 `rebuttalDownloadLetterMd` 适配成 ApiError。
7. **MISSING_RESPONSE**:draft 端点 parts 为空或全空内容 → 信封错误,LLM 调用次数=0
(单测断言)。
8. **PDF 入口**:复用 vstore `parse_pdf_bytes`;校验顺序 = 非 .pdf → 空文件 → 超
`max_pdf_bytes`(50MB)→ 解析。与任务清单"PDF 超限/非 PDF"信封一致。
9. **前端状态**:`store.rebuttal` 纯内存(切面板不丢、刷新丢——与任务"状态持久化"
不同诉求,Phase 1 未要求 localStorage);draft 失败信封里的 job/validation 一并镜像。
10. **不改禁区**:sdk/agent/vstore/未动;PresubmitPanel 等既有面板未动;无新增依赖。
## 测试
### 后端 pytest(9 例新增,全量 113 passed)
```
tests/test_rebuttal.py::test_single_reviewer_single_comment PASSED
tests/test_rebuttal.py::test_multi_reviewer_attribution PASSED
tests/test_rebuttal.py::test_subitem_split_shares_group_id PASSED
tests/test_rebuttal.py::test_location_validation_fail_and_pass PASSED
tests/test_rebuttal.py::test_rebuttal_validation PASSED
tests/test_rebuttal.py::test_missing_response_no_llm_call PASSED
tests/test_rebuttal.py::test_cross_ref_unresolved_keeps_raw PASSED
tests/test_rebuttal.py::test_envelope_errors PASSED
tests/test_rebuttal.py::test_human_ack_gate_on_letter_download PASSED
全量: 113 passed in 10.05s(104 基线 + 9 新增)
```
### 前端 vitest(5 例新增)
```
rebuttal.panel.test.js 5 passed(5 类下拉无 auto / location hint 条件显示 /
未填禁用生成 / 未全确认禁用下载+横幅 / merge-split 仅 active 表单)
全量: 387 passed | 5 failed(失败全部为已知 labels.js 引发的 benchpanel×4 +
Composer .mode-pill×1 预存失败,与本次改动无关;基线 382P/5F → 387P/5F)
```
### build
`npm run build` ✓ built in 3.04s
## 手工冒烟(curl,8100 真 LLM + 真 SQLite)
```
POST /jobs(真实双 Reviewer 意见)→ ok=True job=rbt_ed245cd63b3a reviewers=2 comments=4
titles: 缺少FDR校正与q值报告 / Figure 3 blurry… / Missing coordinates… / Overstated novelty…
POST draft(未填) → ok=False code=MISSING_RESPONSE(LLM 0 调用)
PATCH response(location 弱) + POST draft
→ ok=False VALIDATION_FAILED findings=[LOCATION_EVIDENCE_WEAK]
PATCH response(location 合法) + POST draft → ok=True(真 LLM 草稿)
PATCH response(rebuttal 让步词) + draft → VALIDATION_FAILED findings=[REBUTTAL_NEGATIVE]
3 条合理回复(new_data/clarification/rebuttal)→ draft 全 ok=True
GET letter.md(未全勾) → HTTP 400 HUMAN_ACK_REQUIRED
PATCH confirm ×3 → human_ack=False(c0001 漏勾,恰好验证逐条门槛)
PATCH confirm c0001 → human_ack=True
POST letter → ok=True letter_md len=2731
GET letter.md → HTTP 200(/tmp/opencode/letter.md 落盘,含
Response to Reviewers + 4 条原文/回复)
DELETE job → ok=True meta={deleted:…};GET → NOT_FOUND
BFF 8095 探测 /rebuttal/jobs/nonexistent → HTTP 200(透传信封,REVIEW_ENABLED=1)
```
## 交付物清单
| 文件 | 动作 |
|---|---|
| review_service/app/services/rebuttal_store.py | 新增(SQLite CRUD + human_ack 重算) |
| review_service/app/services/rebuttal_parse.py | 新增(正则骨架 + LLM 拆分 + 交叉引用) |
| review_service/app/services/rebuttal_draft.py | 新增(5 类 Prompt + 反幻觉校验 + letter) |
| review_service/app/routers/rebuttal.py | 新增(10 端点) |
| review_service/app/main.py | +import +include_router +lifespan 建表(约 8 行) |
| review_service/tests/test_rebuttal.py | 新增(9 例) |
| ui/src/api/review.js | +10 API 封装 + rebuttal 错误码未进 CODE_HINTS(后端原话直达) |
| ui/src/stores/edu.js | +SECTIONS 'rebuttal' + rebuttal 状态块 + 10 actions |
| ui/src/components/edu/panels/RebuttalPanel.vue | 新增(拆条树/逐条表单/校验红条/draft 预览/手工拆合/letter) |
| ui/src/components/edu/EduSideNav.vue | +1 行菜单(审稿回复助手 💬) |
| ui/src/components/edu/EduWorkspace.vue | +import +panelMap 挂载 |
| tests/frontend/unit/edu/rebuttal.panel.test.js | 新增(5 例) |
| docs/pr-reports/2026-10-12-rebuttal-phase1.md | 本报告 |
## git status --short(新增部分)
```
?? docs/pr-reports/2026-10-12-rebuttal-phase1.md
?? review_service/app/routers/rebuttal.py
?? review_service/app/services/rebuttal_draft.py
?? review_service/app/services/rebuttal_parse.py
?? review_service/app/services/rebuttal_store.py
?? review_service/tests/test_rebuttal.py
?? tests/frontend/unit/edu/rebuttal.panel.test.js
?? ui/src/components/edu/panels/RebuttalPanel.vue
M review_service/app/main.py
M ui/src/api/review.js
M ui/src/components/edu/EduSideNav.vue
M ui/src/components/edu/EduWorkspace.vue
M ui/src/stores/edu.js
(此前 AI 预审稿修复的未 commit 改动保持原样,未混入本报告范围)
```
## 已知边界(Phase 1 明确不做)
- docx 导出(Phase 2,python-docx 已在 lock 备用);
- job TTL 自动清理(决策 5:手动 DELETE 替代);
- 跨 Reviewer 合并(merge 限同 Reviewer 内);
- 刷新页面后 job 恢复(纯内存态;GET /jobs/{id} 已备,Phase 2 可接列表入口)。
@@ -0,0 +1,111 @@
# PR 报告:P3b 编译 PDF Tab(论文重排版面板)
- 日期:2026-10-13
- 分支:`feature/p3b-compile-ui`(自最新 main 新开;未 commit,按用户约定统一处理)
- 范围:`ui/` 面板 + store/api 扩展;`review_service/` 仅 restart.sh 调起预热脚本 + 新增
scripts/prewarm_tectonic.sh。**后端 compile.py / latex_compile.py(P3a)零改动**;
无 sdk/、agent/、vstore/ 改动;无新增依赖。
## 概要
P3b:论文重排版面板启用"编译 PDF" Tab,实现"上传 .tex → 服务器编译 → 下载 PDF"的完整闭环。
## 修改内容
| 文件 | 变更 |
|---|---|
| ui/src/components/edu/panels/ReformatPanel.vue | 编译 Tab 状态机:无 job_id 引导 / 编译按钮 / loading 动画 + 首次 1-3 分钟黄条(可关闭)/ 成功 blob URL iframe 预览 + 下载 + 耗时 + 重新编译 / 失败错误摘要 + log 尾 200 行折叠 + 完整 log 下载;按钮不 await,状态全由 store 驱动(切 Tab/切面板编译继续) |
| ui/src/stores/edu.js | compileStatus/compileResult/compileError/compileLogTail + compilePdfUrl(blob URL,fetch 失败静默回退 pdf_url 直链)+ compileFirstHint;submitCompile(loading 重入保护)/ clearCompile(revoke blob);重新排版自动 clearCompile |
| ui/src/api/review.js | CODE_HINTS 增 QUEUE_FULL 固定文案"编译队列已满(最多 5 个任务),请稍后重试"(后端 message 带动态 pending 数,规范统一口径);compileTex() 为 P3a 已有封装 |
| review_service/restart.sh | 启动后后台调起 prewarm_tectonic.sh(nohup 非阻塞,日志 prewarm.log) |
| review_service/scripts/prewarm_tectonic.sh(新增) | 最小 hello.tex 实编译以缓存 TeX 环境。**注**:tectonic v0.17.0 无 `download` 子命令(实测 `no subcommand download`),改用 tools/README 的最小 .tex 预编译法;失败仅告警不影响服务启动 |
| tests/frontend/unit/edu/reformat.compile.test.js(新增) | 6 个用例(真实 reviewErrorMessage 经 importOriginal 保留,QUEUE_FULL 走 CODE_HINTS) |
## 关键实现决策
1. **无异步轮询**:后端编译同步阻塞返回(单槽队列,常规 10-30s,上限 `compile_queue_limit=5`),
无 stage 可显示 → loading 只做不确定时长旋转动画 + 文案说明。
2. **QUEUE_FULL**:HTTP 429 + 同款信封,http.js 抛 ApiError(code=QUEUE_FULL);
前端 CODE_HINTS 固定文案,报错后"重新编译"按钮立即可用(store finally 解锁)。
3. **PDF blob URL**:成功后异步 `fetch(pdf_url) → blob → createObjectURL`,
预览/下载不暴露后端路径;jsdom/网络失败回退 pdf_url 直链,不阻塞渲染。
4. **失败 log**:信封 detail = log 尾 200 行(折叠 `<pre>`);完整 log 下载兜底拼
`/api/review/compile/{job_id}/log`(失败信封不带 log_url,但 job_id 即 anchor_id)。
5. **首次编译提示条**:`compileFirstHint` 在首次 submitCompile 置位,loading 期间显示,
可手动关闭;`clearCompile`(重新排版)复位。
6. **预热命令修正**:任务原文 `./tools/tectonic -X download default` 在钉死版本
v0.17.0 不存在——实测后改为最小 .tex 实编译预热(同 tools/README.md 建议)。
## 测试结果
- 后端 pytest(review_service,后端未动):**113 passed**
- 前端 vitest 全量:**403 passed / 5 failed**(5F = 既有 labels.js 基线失败
benchpanel×4 + Composer pill×1,与本 PR 无关)
- vitest 新增 reformat.compile.test.js:**6/6 passed**
(无 job_id 引导 / 按钮可点 / 编译中+黄条+重入保护 / 成功 iframe+下载+耗时 /
失败 log 折叠 / QUEUE_FULL 文案+重试可用)
- npm run build:✓ 3.12s
## 手工冒烟(8100 真实链路)
- reformat /tmp/opencode/reformat_sample.tex → ok=True job_id=b3631727af68
- POST /compile {job_id=b3631727af68} → ok=True seconds=0.61 pdf_bytes=3596
- GET /compile/{cid}/pdf → 200,3596 bytes,application/pdf(file: PDF 1.5)
- GET /compile/{cid}/log → 200,3533 bytes
- job_id 不存在 / 格式非法 → VALIDATION 信封
- 坏 tex(\thisCommandDoesNotExist)→ COMPILE_FAILED + detail 尾 200 行
(含 "! Undefined control sequence")
- QUEUE_FULL → 429 + 信封(后端路径实测于 P3a;前端文案 vitest 6/6 覆盖)
- scripts/prewarm_tectonic.sh → exit=0 OK,宏包缓存就绪
## EPS 兼容性修复 v2(2026-10-14 追加)
背景:真实 MDPI 官方 zip(v6.5a)上传编译在 XDV→PDF 阶段失败:
`xdvipdfmx: image inclusion failed for "Definitions/logo-mdpi.eps"`。
v1 回退逻辑 gate 在"zip 自带 `*-eps-converted-to.pdf` 预转换版"——真实官方 zip
**不带**该文件(v1 本地 e2e 通过是因自造样本带了;测试样本与真实样本不一致,教训已记录)。
v2 方案:**不依赖扫描引用与 zip 自带预转换版**,解压后扫描项目内所有 `.eps`
批量现场转同名 `.pdf`(`latex_compile.process_eps_images`):
1. 工具优先级 **gs → epstopdf → ps2pdf**(`shutil.which` 逐个 fallback);gs 命令
约束1 强制 `-dSAFER -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -dPDFSETTINGS=/prepress`
(-dSAFER 防 EPS 内嵌任意 PostScript 执行);无新 Python 依赖(subprocess)
2. 约束2:单文件转换 **15s 超时** → warning 行 `[eps-convert] xxx.eps 转换超时`
并跳过该文件继续
3. 约束3:单文件转换后 >20MB → warning;累计转换产物 >100MB →
`InspectError("ZIP_TOO_LARGE")` 中止(信封原样透出该 code)
4. 约束4:`[eps-convert] {src} → {dst} ({in} → {out}, {elapsed}s, {tool})`
写入 `output.log`(正常/超时两条路径都落盘,引擎 log 之后、chatter 之前)
5. 转换成功 → 显式 `X.eps` 引用字节替换为 `X.pdf`(.tex/.cls/.sty/.cfg/.def 全扫,
不碰编码、不改正文);无扩展名引用 `{.../X}` 由 graphicx 命中新 `X.pdf`
(解析顺序 .pdf→…→.eps,规避 .eps)
6. 全工具失败 → log 行 `[eps-convert] 全部工具失败,图片 xxx.eps 未转换` →
编译自然失败 → 前端提示「项目包含 EPS 图片,服务器转换失败,请转成 PDF/PNG 后重试」
真实官方 zip 实测(`scripts/debug_compile.py` 重打包 fba37ae3c853 现场 + 8100 HTTP 链路):
| 指标 | 结果 |
|---|---|
| 转换 EPS 数 | 2(logo-mdpi.eps、logo-updates.eps,均 gs) |
| 每个耗时 | 563678→13057B 0.27s;287998→16588B 0.27s(合计 0.54s) |
| 显式引用改写 | 1 个文件(Definitions/mdpi.cls) |
| 编译 | 2.83s(debug 脚本)/ 2.99s(HTTP) |
| 产出 PDF | 70688 / 70687 bytes,7 页,`%PDF-` ✓ |
测试:pytest **175 passed**(原 169 全保留,移除 3 个 v1 语义已变的 EPS 用例,
新增 9 个 v2 用例:preset 免转换 / 真实 gs 转换 / 工具 fallback / 全失败 /
超时 warning / 20MB warning / 100MB 中止 / 端到端成功 / 友好报错)。
教训(关键):**外部下载的模板类场景,e2e 必须用真实官方文件**——v1 的自造样本
带了预转换版,真实官方 zip 不带,导致修复上线后失效。
## 部署验证(部署后执行)
```bash
bash review_service/restart.sh
# 观察日志:prewarm_tectonic.sh 后台跑(非阻塞),prewarm.log 出现 "[prewarm] OK"
# 浏览器 → 教育科研 → 论文自动重排版 → 上传 .tex → 选期刊 → 重排版
# → 切"编译 PDF" → 点编译
# 期望:首次 1-3 分钟(若预热已完成则秒级)、后续秒回;失败可见 log 尾 200 行
```
@@ -0,0 +1,131 @@
# P5 科研汇报助手 — PR 报告(2026-10-14,v4 更新 2026-10-15)
## 概要
新增「科研汇报助手」(P5):论文 PDF/粘贴文本 → vstore 解析 → map-reduce LLM 大纲
(5 场景 × 4 时长 × **14 版式**,逐条证据可溯)→ 逐页编辑 → python-pptx 本地生成
16:9 PPTX 初稿(论文真图卡片化嵌入、分隔页主副标题、讲者备注携带证据引用)。
全程只新增 `review_service/` 内代码 + BFF 既有 `REVIEW_ENABLED` 条件反代,
禁区(RAG 核心、既有服务、前端 router/db)零改动。
**v4 迭代(2026-10-15)**:① 副标题风格数改为动态读取 `GET /ppt/styles`(缺数据回落
「多风格·图文混排」)② 章节分隔页加副标题(prompt 强制 `subtitle`、渲染 40pt 主标题 +
20pt muted 副标题 + 底部 accent 装饰线、编辑器可改)③ 论文白底原图与页面底色融合
(surface_2 圆角卡片 + 图注 + 宽/高上限 + 单图页居中留白)。
## 修改内容
### 后端(`review_service/`)
| 类别 | 文件 | 说明 |
| --- | --- | --- |
| 改 | `app/config.py` | +`ppt_jobs_dir` / `ppt_styles_dir` 及 abs 属性(+14 行) |
| 改 | `app/main.py` | 导入 ppt 路由、lifespan `ppt_ensure_table()`、挂载(+8 行) |
| 改 | `tooling/requirements.lock` | **+3 行**:`pillow==12.3.0` / `python-pptx==1.0.2` / `xlsxwriter==3.2.9`(三者均已实装于统一 venv) |
| 改 | `.gitignore` | +9 行:`ppt_jobs/` 运行数据、`prewarm.log`、`data/compile_jobs/`、`scripts/`、`data/review.db` |
| 新 | `app/routers/ppt.py`(317 行) | **9 端点**:`GET /styles`、`POST /jobs`、`POST /jobs/pdf`、`GET/PATCH /jobs/{id}`、`POST /jobs/{id}/build`、`GET /jobs/{id}/pptx`、`GET /jobs/{id}/preview/{page}`、`DELETE /jobs/{id}` |
| 新 | `app/services/ppt_outline.py`(917 行) | map-reduce 大纲核心(索引/配额/重验/裁剪/膨胀回修/分隔页 subtitle 规则) |
| 新 | `app/services/ppt_build.py`(1176 行) | python-pptx 生成器(14 版式 token、字体栈、真图卡片容器、分隔页主副标题、讲者备注) |
| 新 | `app/services/ppt_figure_index.py`(264 行) | 论文真图索引(para/图号 → 图片落盘与状态) |
| 新 | `app/services/ppt_preview.py`(230 行) | 逐页 PNG 预览(供面板缩略图,失败不阻断) |
| 新 | `app/services/ppt_store.py`(155 行) | `ppt_jobs` SQLite 持久化(约定 DDL 逐字实现) |
| 新 | `data/ppt_styles/*.json`(14 个) | 版式 token 源码资产(随 git 走,`GET /styles` 的唯一数据源) |
| 新 | `tests/test_ppt_*.py`(6 个 / 102 例) | outline / build / figure_index / preview / router / store |
### 前端(`ui/` + `tests/`)
| 类别 | 文件 | 说明 |
| --- | --- | --- |
| 改 | `ui/src/api/review.js`(+50 行) | PPT API 段 + `pptStyles()` |
| 改 | `ui/src/stores/edu.js`(+147 行) | `ppt` 状态块 + actions + `pptLoadStyles` + `SECTIONS` 增列 |
| 改 | `ui/src/components/edu/EduSideNav.vue` / `EduWorkspace.vue` | 菜单项 + PptPanel 注册(各 +2 行) |
| 新 | `ui/src/components/edu/panels/PptPanel.vue`(578 行) | 生成表单 + 编辑器(动态风格数、分隔页副标题输入、逐页预览) |
| 新 | `ui/src/components/edu/panels/ppt-styles.css`(40 行) | 面板样式 |
| 新 | `ui/src/assets/ppt-style-cards/*.png`(14 个,约 21KB/个) | 版式卡面截图 |
| 新 | `tests/frontend/unit/edu/ppt.panel.test.js`(12 例)/ `ppt.store.test.js`(7 例) | P5 前端单测 |
| 改 | `tests/frontend/unit/b2/components.test.js`、`b3/benchpanel.test.js`、`bioflow/benchpanel.bioflow.test.js`、`tests/e2e/bench.spec.js` | **预存失败清零**:① tab 文案「提取任务」→「上传文献」(对齐 `51532a3`)② 模式 pill 断言改为「已随 `43a7c6c` 移除」 |
**依赖偏差声明**:`python-pptx==1.0.2`(PyPI 最高 1.0.2;需求中 1.1.2 实为
python-docx 版本,系记混,已在实现时更正)。`xlsxwriter==3.2.9` 当前无代码直接引用,
属统一 venv `pip freeze` 的真实记录,一并同步锁文件——**合并前仍需人工确认这 3 行锁变更**。
**DDL**(与约定逐字一致):
```sql
ppt_jobs(id, status /*outline_ready|building|done|error*/, source_kind, filename,
raw_text, scenario, minutes, style_id, pages_json, pptx_path,
warnings_json, created_at, updated_at)
```
**API 前缀** `/api/review/ppt`(9 端点;业务错误一律 200 信封
`{ok:false, error:{code,message,degraded}}`;二进制下载缺失时回同款信封 JSON)。
### 关键渲染约定(v4)
- **分隔页**:`section_bg` 底(缺则 `surface_2`→`cover_bg`→`bg`),40pt 居中粗体主标题 +
20pt muted 副标题(`bullets[0]` → `subtitle` → 该章第一页 title 三级兜底,浅底自动换深字)+
底部 30% 宽 accent 短线。
- **真图卡片**:`surface_2` 圆角矩形(8px)容器 + 5mm 内边距,图居中、图注
`Fig N(论文 S?P?)` 10–12pt muted、与图区留 3mm;混排页图区 ≤55% 页宽且 ≤60% 页高,
单独图页居中、上下留白 ≥5%。
## 测试结果(2026-10-15)
```
pytest(review_service 全库): 215 passed in 13.7s (P5 新增 102 → 113 + 102)
vitest edu(tests/frontend/unit/edu): 43 passed (P5 新增 19:panel 12 + store 7)
vitest 全量(tests/frontend/unit): 32 files / 421 passed / 0 failed
—— 修掉 b2/b3/bioflow 共 5 个预存失败(文案改名 + 模式按钮移除)
npm run build: ✓ built(chunk > 500kB 警告为预存)
版式目检: 3 风格 × 10 页预览 PNG 渲染核对(分隔页主副标题/对比度回退、
混排图卡 ≤55%、单独图页居中留白)均符合预期
```
## 真实 e2e(v1 阶段记录:官方 MDPI 模板实例 PDF,16813 字符,真实 DeepSeek)
```
PDF 模式创建: 21.77s → 12 页(目标 [11,14])warnings=0 llm_calls=7 unverified_evidence=0
PATCH 编辑: 改写要点 + 新增无证据行 → evidence 同长重验、结构清洗 ✓
build: 0.13s → 12 slides / 64157 B / 16:9(13.33×7.50in)逐页讲者备注 ✓
python-pptx 回读: 12 页,S3 首条 = 人工改写文本 ✓
编辑后下载: ok=false NOT_FOUND(盘上旧文件存在但拒发,pptx_path 清空生效)✓
text 模式: 15.0s → 9 页(目标 [8,10])group_meeting / 10min / botanic-forest ✓
短文本: 200 信封 VALIDATION「正文过短(<300 字),无法生成汇报大纲」✓
清理: 双 DELETE → ppt_jobs 0 行、产物目录空 ✓
```
## 部署验证步骤
1. 依赖:同步 `tooling/requirements.lock`(+3 行)并安装到统一 venv。
2. 重启服务:`cd review_service && bash restart.sh`(8100)。
3. 健康检查:`curl http://127.0.0.1:8100/health` → `status=ok`、`vstore connected=true`。
4. 路由:`curl .../openapi.json` 确认 9 个 `/api/review/ppt/*` 路径在列(含 `/styles`)。
5. BFF:`sdk/.env` 确认 `REVIEW_ENABLED=1`,走 8090 反代冒烟
`POST /api/review/ppt/jobs`(短文本应得 200 VALIDATION 信封,不打 LLM)。
6. 真实链路:上传一篇论文 PDF → 出大纲 → 编辑分隔页副标题并保存 → build → 下载 PPTX →
用 python-pptx 核对页数、16:9 尺寸与分隔页副标题文本。
7. 回归:`cd review_service && <venv>/bin/python -m pytest -q`;
`cd ui && npx vitest run` + `npm run build`。
## 部署决策
**建议:只合并,下次批量部署。** 本次改动对存量服务零影响(纯新增模块 +
开关化反代),无数据迁移、无 schema 破坏性变更;单点重启 8100 即可生效,
不值得为它单独发版,随下一次批量部署一并上即可。
## 风险与回滚
| 风险 | 影响 | 缓解 / 回滚 |
| --- | --- | --- |
| `python-pptx==1.0.2` 版本与需求(1.1.2)不一致 | 依赖锁未按原文落库 | 已声明偏差,合并前人工确认 3 行锁变更;如需回退直接还原 lock |
| `xlsxwriter==3.2.9` 无代码引用 | 锁文件带入未用依赖 | venv 实况记录,可保留;若要求最小化可单独剔除该行 |
| LLM 大纲偶发超时/无效 JSON | 创建接口报 `LLM_ERROR`/`LLM_INVALID` | 已分片 reduce + 单次膨胀回修 + 警告降级;前端可重试,无脏数据落库 |
| quote 反查失败残留 evidence(2 条) | 个别要点 `unverified` | **Phase 2**:最长公共子串(≥20 字符)匹配补 `para_id`,失败保持 unverified(边际价值低,v4 不做) |
| 编辑后旧 PPTX 残留盘上 | 无(`pptx_path` 清空后拒发,仅占磁盘) | 删除 job 时 `rmtree` 一并清理 |
| 回滚 | — | 反向还原 12 个已跟踪文件改动 + 删除 44 个新增文件(后端 26 + 前端 18)即可;`ppt_jobs` 表与产物目录可留可删,不被其他模块引用 |
## 与 `feature/p3b-compile-ui#4f9c077` 的关系
v1 版报告曾以同一路径提交于该分支(`docs: add P5 PR report`),内容停留在 3 版式 /
7 端点 / 208 pytest 阶段。本文件为 v4 权威版本;若两分支先后合入,同路径会产生一次
文本冲突,取本文件(v4)内容即可。
+13
View File
@@ -0,0 +1,13 @@
Reviewer #1:
1. The GWAS analysis lacks multiple-testing correction. Please re-run with FDR control and report q-values for all significant loci. In addition, the sample size justification is missing — please add a power analysis or at least discuss the detectable effect size.
2. Figure 3 is blurry and the axis labels are unreadable. As noted in Comment 3, the same issue applies to Figure S2. Please replace all figures with high-resolution versions.
3. Minor: please add the standard error estimates to Supplementary Table 2.
Reviewer #2:
1. The claim of novelty overstates the advance relative to Zhang et al. 2023. The introduction should more clearly delineate what is new.
2. How were the batch effects between the two sequencing centers corrected? This is essential to support the conclusions.
+1
View File
@@ -0,0 +1 @@
"""paper_writer_service —— 论文撰写独立 MCP 服务(8097)。"""
+53
View File
@@ -0,0 +1,53 @@
"""config.py —— paper 服务(8097)环境变量驱动配置(pydantic-settings)。
全部键可用环境变量覆盖(前缀 PAPER_;LLM 沿用 docgen 的 DOCGEN_LLM_* 直连模式,
由 services/paper/llm.py 直接读 os.environ)。零硬编码宿主路径:相对路径按 __file__ 推导。
"""
from __future__ import annotations
from functools import lru_cache
from pathlib import Path
from pydantic_settings import BaseSettings, SettingsConfigDict
_SERVICE_ROOT = Path(__file__).resolve().parent.parent # paper_writer_service/
def _resolve(v: str) -> str:
"""仓库内相对路径按 __file__ 推导服务根解析为绝对路径。"""
p = Path(v).expanduser()
if not p.is_absolute():
p = _SERVICE_ROOT / p
return str(p)
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
env_prefix="PAPER_",
)
app_name: str = "plantai-paper-writer-service"
version: str = "0.1.0"
db_path: str = "paper.db" # 独立 SQLite(与 BFF 完全隔离)
vstore_base_url: str = "http://127.0.0.1:8094"
read_timeout: float = 360.0 # write_paper_section 同步长调用读超时
@property
def db_path_abs(self) -> str:
return _resolve(self.db_path)
@property
def vstore_base(self) -> str:
return self.vstore_base_url.rstrip("/")
@lru_cache(maxsize=1)
def get_settings() -> Settings:
return Settings()
settings = get_settings()
+168
View File
@@ -0,0 +1,168 @@
"""db.py —— paper 服务独立 SQLite(aiosqlite + WAL,无 ORM),三张表。
- papers:论文主表(brief/outline/final_markdown/references/disclaimer)
- paper_sections:章节表(逐节 markdown + 状态)
- citations:结构化引用(temp_n 生成期全局递增,final_n complete 按出现序重排写回)
模式沿用 bioflow/docgen 同款:CREATE TABLE IF NOT EXISTS + aiosqlite 单连接 + WAL + 写锁。
"""
from __future__ import annotations
import asyncio
import json
import logging
import time
from collections.abc import AsyncIterator, Iterable
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import aiosqlite
from .config import get_settings
logger = logging.getLogger(__name__)
_SCHEMA = """
CREATE TABLE IF NOT EXISTS papers (
id TEXT PRIMARY KEY,
session_id TEXT NOT NULL DEFAULT '',
title TEXT NOT NULL DEFAULT '',
paper_type TEXT NOT NULL DEFAULT '',
language TEXT NOT NULL DEFAULT '',
format_req TEXT NOT NULL DEFAULT '',
style TEXT NOT NULL DEFAULT '',
status TEXT NOT NULL DEFAULT 'outline',
brief_json TEXT NOT NULL DEFAULT '{}',
outline_json TEXT NOT NULL DEFAULT '{}',
final_markdown TEXT NOT NULL DEFAULT '',
references_json TEXT NOT NULL DEFAULT '[]',
disclaimer TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_papers_session ON papers(session_id);
CREATE INDEX IF NOT EXISTS idx_papers_status ON papers(status);
CREATE TABLE IF NOT EXISTS paper_sections (
id TEXT PRIMARY KEY,
paper_id TEXT NOT NULL REFERENCES papers(id) ON DELETE CASCADE,
section_id TEXT NOT NULL,
title TEXT NOT NULL DEFAULT '',
key_points TEXT NOT NULL DEFAULT '[]',
status TEXT NOT NULL DEFAULT 'pending',
markdown TEXT NOT NULL DEFAULT '',
word_count INTEGER NOT NULL DEFAULT 0,
updated_at TEXT NOT NULL,
UNIQUE(paper_id, section_id)
);
CREATE INDEX IF NOT EXISTS idx_paper_sections_paper ON paper_sections(paper_id);
CREATE TABLE IF NOT EXISTS citations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
paper_id TEXT NOT NULL REFERENCES papers(id) ON DELETE CASCADE,
section_id TEXT NOT NULL DEFAULT '',
temp_n INTEGER,
final_n INTEGER,
kind TEXT NOT NULL DEFAULT 'title',
ref_text TEXT NOT NULL DEFAULT '',
source_doc_id TEXT NOT NULL DEFAULT '',
snippet TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_citations_paper ON citations(paper_id);
"""
def now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def dumps_json(value: Any, default: Any = None) -> str:
try:
return json.dumps(value, ensure_ascii=False)
except (TypeError, ValueError):
return json.dumps(default, ensure_ascii=False)
def loads_json(text: str | None, default: Any = None) -> Any:
if text is None:
return default
try:
return json.loads(text)
except (TypeError, ValueError):
return default
class Database:
"""aiosqlite 单连接封装(WAL;写串行化,同 bioflow 模式)。"""
def __init__(self, path: str | Path | None = None) -> None:
self._path: Path | None = Path(path) if path is not None else None
self._conn: aiosqlite.Connection | None = None
self._lock = asyncio.Lock()
@property
def path(self) -> Path:
if self._path is None:
self._path = Path(get_settings().db_path_abs)
return self._path
async def connect(self) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
self._conn = await aiosqlite.connect(self.path)
self._conn.row_factory = aiosqlite.Row
await self._conn.execute("PRAGMA journal_mode=WAL")
await self._conn.execute("PRAGMA foreign_keys=ON")
await self._conn.execute("PRAGMA busy_timeout=5000")
await self._conn.executescript(_SCHEMA)
await self._conn.commit()
async def close(self) -> None:
if self._conn is not None:
await self._conn.close()
self._conn = None
@property
def conn(self) -> aiosqlite.Connection:
if self._conn is None:
raise RuntimeError("Database 未初始化:先 await init_db()")
return self._conn
async def fetchall(self, sql: str, params: Iterable[Any] = ()) -> list[aiosqlite.Row]:
async with self._conn.execute(sql, tuple(params)) as cursor:
return list(await cursor.fetchall())
async def fetchone(self, sql: str, params: Iterable[Any] = ()) -> aiosqlite.Row | None:
async with self._conn.execute(sql, tuple(params)) as cursor:
return await cursor.fetchone()
async def execute(self, sql: str, params: Iterable[Any] = ()) -> int:
async with self._lock:
cur = await self._conn.execute(sql, tuple(params))
await self._conn.commit()
return cur.lastrowid
db = Database()
async def init_db(path: str | Path | None = None) -> Database:
# 原地改单例路径后连接(不做 db = Database(path) 重赋值):
# 服务模块以 `from ...db import db` 持有了该单例引用,重赋值会让它们拿到旧对象。
if path is not None:
db._path = Path(path)
await db.connect()
return db
async def get_db() -> AsyncIterator[Database]:
yield db
async def close_db() -> None:
await db.close()
def monotonic() -> float:
return time.monotonic()
+84
View File
@@ -0,0 +1,84 @@
"""main.py —— paper 服务应用入口(uvicorn 入口,明文白名单文件)。
- 独立部署树 paper_writer_service/(FR:独立进程 8097 + 独立 SQLite paper.db)。
- lifespan:init_db(幂等建表)→ 显式进入 MCP session_manager.run()(Starlette 不自动
运行子 Mount 的 lifespan,streamable-http 处理请求时要求 task group 已初始化)→ 清理。
- 端点:/health(GET,BFF 探活反代)+ /mcp(MCP streamable-http)。
- 不依赖 BFF/生产服务(8090 等绝不探测/连接)。
"""
from __future__ import annotations
import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from .config import get_settings
from .db import close_db, db, init_db
from .mcp import paper_mcp
from .services.paper import search
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# MCP streamable-http app(调用即创建 session_manager,供 lifespan 进入 run())。
# 端点地址:/mcp(streamable_http_path="/" + 挂载前缀 /mcp)。
_paper_mcp_app = paper_mcp.streamable_http_app(streamable_http_path="/")
def create_app() -> FastAPI:
settings = get_settings()
app = FastAPI(title=settings.app_name, version=settings.version)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=False,
allow_methods=["*"],
allow_headers=["*"],
)
@asynccontextmanager
async def _lifespan(_app: FastAPI):
# 测试注入:db 已由测试预置(init_db(tmp_path)),跳过启动初始化。
try:
await init_db()
except Exception:
logger.exception("main: 数据库初始化失败(health 将报 db 降级)")
# MCP session manager 的 task group 必须在服务运行期内保持活跃。
async with paper_mcp.session_manager.run():
try:
yield
finally:
try:
await close_db()
except Exception:
pass
await search.close_client()
app.router.lifespan_context = _lifespan
@app.get("/health")
async def health() -> dict[str, object]:
db_ok = True
db_error: str | None = None
try:
await db.fetchone("SELECT 1 AS ok")
except Exception as exc: # noqa: BLE001
db_ok = False
db_error = str(exc)
return {
"ok": True,
"status": "ok" if db_ok else "degraded",
"db": {"connected": db_ok, "error": db_error},
"version": settings.version,
}
# paper MCP 能力外挂(mcp 2.x:MCPServer.streamable_http_app)。
app.mount("/mcp", _paper_mcp_app)
return app
app = create_app()
+48
View File
@@ -0,0 +1,48 @@
"""app/mcp.py —— paper_writer_service MCP server(单 streamable-http 端点 /mcp)。
9 个工具:propose_paper_outline / write_paper_section / get_paper_progress / search_refs /
revise_paper_section / revise_paper / get_paper / complete_paper / cancel_paper。
mcp 2.x 注意:MCPServer(非 FastMCP,来自 mcp.server.mcpserver);工具返回 dict,由 MCP
序列化为 CallToolResult(structuredContent + content:[{type:"text",text:"<json>"}])。
本文件位于 app/ 层(非 services/),可用 pydantic(mcp 自身依赖 pydantic 构造 schema)。
"""
from __future__ import annotations
from typing import Any
from .services.paper import complete, outline, search, write
def build_paper_mcp() -> Any:
from mcp.server.mcpserver import MCPServer
mcp = MCPServer(
"paper-writer",
title="PlantAI 论文撰写",
description="论文撰写能力单元:大纲生成/逐节撰写/文献引用/全文合并重排/修订。",
)
mcp.tool(name="propose_paper_outline",
description="根据论文主题与写作要求生成大纲并建论文(内部检索知识库注入选题线索)。")(outline.propose_paper_outline)
mcp.tool(name="write_paper_section",
description="撰写论文某一章节(同步长调用,90-180s)。按全局引用编号表生成正文与引用。")(write.write_paper_section)
mcp.tool(name="get_paper_progress",
description="查询论文与各章节撰写进度/字数/引用数。")(complete.get_paper_progress)
mcp.tool(name="search_refs",
description="在知识库(vstore)中检索与查询相关的文献片段。")(search.search_refs)
mcp.tool(name="revise_paper_section",
description="按用户意见重写论文某一章节,新引用追加到引用表末尾。")(write.revise_paper_section)
mcp.tool(name="revise_paper",
description="按用户意见判定需要修改哪些章节,返回受影响节清单与修订计划。")(write.revise_paper)
mcp.tool(name="get_paper",
description="拉取论文全文 markdown、参考文献列表与免责声明。")(complete.get_paper)
mcp.tool(name="complete_paper",
description="合并全文、按首次出现顺序重排引用编号、渲染 GB/T 7714 文献列表。")(complete.complete_paper)
mcp.tool(name="cancel_paper",
description="取消论文(置 cancelled)。")(complete.cancel_paper)
return mcp
paper_mcp = build_paper_mcp()
+10
View File
@@ -0,0 +1,10 @@
"""schemas.py —— paper 服务共享常量(明文白名单文件)。
MCP 工具入参/出参均为 dict(由 mcp 层序列化为 content:[{type:text,text:<json>}]),
本文件只放状态/引用类型常量,避免服务层散落魔法字面量。
"""
from __future__ import annotations
PAPER_STATUSES = ("outline", "writing", "complete", "cancelled", "error")
SECTION_STATUSES = ("pending", "writing", "done", "error")
CITATION_KINDS = ("doi", "pmid", "title")
@@ -0,0 +1,65 @@
"""cite.py —— 引用协议:GB/T 7714 渲染、[n] 校验、complete 按出现序重排。"""
from __future__ import annotations
import re
from typing import Any
_CITE_RE = re.compile(r"\[(\d+)\]")
def find_cite_numbers(markdown: str | None) -> list[int]:
"""正文中出现的所有 [n] 编号(含重复,按出现顺序)。"""
return [int(m) for m in _CITE_RE.findall(markdown or "")]
def render_gbt7714(meta: dict[str, Any], *, fallback_year: str = "2026") -> str:
"""按可用元数据渲染 GB/T 7714 近似条目。
- 有 DOI → [J/OL] + DOI
- 有 PMID → [J] + PMID
- 无 DOI/PMID(知识库资料)→ 降级格式 + [EB/OL]
"""
title = (meta.get("title") or "").strip()
journal = (meta.get("journal") or "").strip()
authors = (meta.get("authors") or "").strip()
year = (meta.get("year") or fallback_year).strip()
doi = (meta.get("doi") or "").strip()
pmid = (meta.get("pmid") or "").strip()
head = f"{authors} " if authors else ""
if doi:
jpart = f"{journal}, {year}." if journal else f"{year}."
return f"{head}{title}[J/OL]. {jpart} DOI:{doi}."
if pmid:
jpart = f"{journal}, {year}." if journal else f"{year}."
return f"{head}{title}[J]. {jpart} PMID:{pmid}."
return f"{head}{title}. 知识库资料, {year}. [EB/OL]."
def render_disclaimer() -> str:
return "参考文献依据知识库资料生成,格式为 GB/T 7714 近似,建议人工核对。"
def first_appearance_order(full_markdown: str, temp_numbers: list[int]) -> list[int]:
"""按全文首次出现顺序返回 temp_n;未出现的排最后(保持原序)。"""
seen: dict[int, int] = {}
for m in _CITE_RE.finditer(full_markdown or ""):
n = int(m.group(1))
if n not in seen and n in temp_numbers:
seen[n] = m.start()
ordered = sorted((n for n in temp_numbers if n in seen), key=lambda n: seen[n])
ordered += [n for n in temp_numbers if n not in seen]
return ordered
def renumber_map(temp_numbers: list[int], full_markdown: str) -> dict[int, int]:
"""temp_n → final_n(按全文首次出现顺序 1..N)。"""
ordered = first_appearance_order(full_markdown, temp_numbers)
return {temp: final for final, temp in enumerate(ordered, start=1)}
def replace_numbers(markdown: str | None, mapping: dict[int, int]) -> str:
def _sub(m: re.Match[str]) -> str:
n = int(m.group(1))
return f"[{mapping.get(n, n)}]"
return _CITE_RE.sub(_sub, markdown or "")
@@ -0,0 +1,109 @@
"""complete.py —— get_paper / get_paper_progress / complete_paper / cancel_paper。"""
from __future__ import annotations
from typing import Any
from ...db import db, dumps_json, loads_json, now_iso
from .cite import render_disclaimer, renumber_map, replace_numbers
async def _load(paper_id: str) -> dict[str, Any]:
paper = await db.fetchone("SELECT * FROM papers WHERE id = ?", (paper_id,))
if not paper:
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
sections = [dict(r) for r in await db.fetchall(
"SELECT * FROM paper_sections WHERE paper_id = ? ORDER BY rowid", (paper_id,))]
citations = [dict(r) for r in await db.fetchall(
"SELECT * FROM citations WHERE paper_id = ? ORDER BY temp_n", (paper_id,))]
return {"ok": True, "paper": dict(paper), "sections": sections, "citations": citations}
def _merge_sections(paper: dict[str, Any], sections: list[dict[str, Any]]) -> str:
parts = [f"# {paper.get('title', '')}"]
for s in sections:
parts.append(f"\n## {s['title']}\n\n{s.get('markdown', '')}")
return "\n".join(parts)
async def get_paper(paper_id: str, format: str = "md") -> dict[str, Any]:
p = await _load(paper_id)
if not p.get("ok"):
return p
paper, sections, citations = p["paper"], p["sections"], p["citations"]
full_markdown = paper.get("final_markdown") or _merge_sections(paper, sections)
references = loads_json(paper.get("references_json"), []) or [
{"n": c.get("final_n") or c.get("temp_n"), "text": c["ref_text"], "kind": c["kind"],
"source_doc_id": c["source_doc_id"]} for c in citations
]
brief = loads_json(paper.get("brief_json"), {})
return {
"ok": True, "paper_id": paper_id, "title": paper["title"], "status": paper["status"],
"template_key": brief.get("template_key") or "document",
"brief": brief,
"sections": [
{"section_id": s["section_id"], "title": s["title"], "status": s["status"],
"word_count": s["word_count"]} for s in sections
],
"full_markdown": full_markdown, "references": references,
"disclaimer": paper.get("disclaimer", ""),
}
async def get_paper_progress(paper_id: str) -> dict[str, Any]:
paper = await db.fetchone("SELECT * FROM papers WHERE id = ?", (paper_id,))
if not paper:
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
paper = dict(paper)
sections = [dict(r) for r in await db.fetchall(
"SELECT * FROM paper_sections WHERE paper_id = ? ORDER BY rowid", (paper_id,))]
cit_rows = [dict(r) for r in await db.fetchall(
"SELECT section_id, COUNT(*) AS c FROM citations WHERE paper_id = ? GROUP BY section_id", (paper_id,))]
cit_by_sec = {r["section_id"]: r["c"] for r in cit_rows}
current = next((s["section_id"] for s in sections if s["status"] != "done"), None)
return {
"ok": True, "paper_id": paper_id, "status": paper["status"], "current_section_id": current,
"sections": [
{"section_id": s["section_id"], "title": s["title"], "status": s["status"],
"word_count": s["word_count"], "citation_count": cit_by_sec.get(s["section_id"], 0)}
for s in sections
],
}
async def complete_paper(paper_id: str) -> dict[str, Any]:
p = await _load(paper_id)
if not p.get("ok"):
return p
paper, sections, citations = p["paper"], p["sections"], p["citations"]
full = _merge_sections(paper, sections)
temp_numbers = [c.get("temp_n") for c in citations if c.get("temp_n")]
mapping = renumber_map(temp_numbers, full)
final_full = replace_numbers(full, mapping)
for c in citations:
await db.execute("UPDATE citations SET final_n = ? WHERE id = ?", (mapping.get(c.get("temp_n")), c["id"]))
for s in sections:
await db.execute("UPDATE paper_sections SET markdown = ? WHERE id = ?",
(replace_numbers(s.get("markdown"), mapping), s["id"]))
ordered = sorted(citations, key=lambda c: mapping.get(c.get("temp_n"), 0))
references = [
{"n": mapping.get(c.get("temp_n"), c.get("temp_n")), "text": c["ref_text"],
"kind": c["kind"], "source_doc_id": c["source_doc_id"]} for c in ordered
]
has_degraded = any(c["kind"] == "title" for c in citations)
disclaimer = render_disclaimer() if has_degraded else ""
await db.execute(
"UPDATE papers SET status = 'complete', final_markdown = ?, references_json = ?, disclaimer = ?, updated_at = ? WHERE id = ?",
(final_full, dumps_json(references), disclaimer, now_iso(), paper_id),
)
return {"ok": True, "full_markdown": final_full, "references": references,
"renumber_map": mapping, "disclaimer": disclaimer}
async def cancel_paper(paper_id: str) -> dict[str, Any]:
paper = await db.fetchone("SELECT id FROM papers WHERE id = ?", (paper_id,))
if not paper:
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
await db.execute("UPDATE papers SET status = 'cancelled', updated_at = ? WHERE id = ?", (now_iso(), paper_id))
return {"ok": True}
@@ -0,0 +1,187 @@
"""services/paper/llm.py —— LLM 调用(httpx 裸调,拷贝自 sdk/app/services/docgen/llm.py)。
- complete_text(system, user, *, temperature=0.3, max_tokens=8000) -> str
- complete_json(...)(response_format json_object + 一次 JSON 修复重试)
- 4 次尝试(1.5s / 3s / 6s 指数退避,429 / 5xx / 超时)
- env 键:DOCGEN_LLM_BASE_URL / DOCGEN_LLM_API_KEY / DOCGEN_LLM_MODEL / DOCGEN_LLM_TIMEOUT
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
from typing import Any
import httpx
logger = logging.getLogger(__name__)
DEFAULT_LLM_BASE_URL = "https://api.deepseek.com/v1"
DEFAULT_LLM_MODEL = "deepseek-chat"
DEFAULT_LLM_TIMEOUT = 300.0
BACKOFF_SECONDS = (1.5, 3.0, 6.0)
MAX_ATTEMPTS = 4
class LLMError(Exception):
"""LLM 调用失败(重试尽/响应异常)。"""
def _llm_config() -> dict[str, Any]:
return {
"base_url": (os.environ.get("DOCGEN_LLM_BASE_URL") or DEFAULT_LLM_BASE_URL).rstrip("/"),
"api_key": os.environ.get("DOCGEN_LLM_API_KEY", ""),
"model": os.environ.get("DOCGEN_LLM_MODEL") or DEFAULT_LLM_MODEL,
"timeout": float(os.environ.get("DOCGEN_LLM_TIMEOUT") or DEFAULT_LLM_TIMEOUT),
}
def _extract_text(payload: dict[str, Any]) -> str:
try:
choices = payload.get("choices") or []
if choices:
content = choices[0].get("message", {}).get("content")
if isinstance(content, str):
return content
if isinstance(content, list):
return "".join(
str(c.get("text", "")) if isinstance(c, dict) else str(c) for c in content
)
except Exception:
pass
return ""
async def _chat_once(
client: httpx.AsyncClient,
cfg: dict[str, Any],
system: str,
user: str,
*,
temperature: float,
max_tokens: int,
json_mode: bool,
timeout: float | None = None,
) -> str:
headers = {"Content-Type": "application/json"}
if cfg["api_key"]:
headers["Authorization"] = f"Bearer {cfg['api_key']}"
body: dict[str, Any] = {
"model": cfg["model"],
"messages": [
{"role": "system", "content": system},
{"role": "user", "content": user},
],
"temperature": temperature,
"max_tokens": max_tokens,
}
if json_mode:
body["response_format"] = {"type": "json_object"}
resp = await client.post(
f"{cfg['base_url']}/chat/completions",
headers=headers,
json=body,
timeout=timeout if timeout is not None else cfg["timeout"],
)
resp.raise_for_status()
payload = resp.json()
if not isinstance(payload, dict):
raise LLMError("LLM 响应形态异常(非 JSON 对象)")
text = _extract_text(payload)
if not text:
raise LLMError("LLM 响应缺少 choices[0].message.content")
return text
async def _complete(
system: str,
user: str,
*,
temperature: float,
max_tokens: int,
json_mode: bool,
timeout: float | None = None,
) -> str:
cfg = _llm_config()
last_exc: BaseException | None = None
async with httpx.AsyncClient() as client:
for attempt in range(MAX_ATTEMPTS):
try:
return await _chat_once(
client, cfg, system, user,
temperature=temperature, max_tokens=max_tokens, json_mode=json_mode,
timeout=timeout,
)
except LLMError:
raise
except (httpx.HTTPStatusError, httpx.HTTPError, httpx.TimeoutException) as exc:
last_exc = exc
status = getattr(getattr(exc, "response", None), "status_code", None)
retryable = status is None or status == 429 or status >= 500
if not retryable or attempt == MAX_ATTEMPTS - 1:
raise LLMError(f"LLM 调用失败(HTTP {status or '网络'}):{exc}") from exc
await asyncio.sleep(BACKOFF_SECONDS[min(attempt, len(BACKOFF_SECONDS) - 1)])
raise LLMError(f"LLM 调用失败:{last_exc}")
async def complete_text(
system: str,
user: str,
*,
temperature: float = 0.3,
max_tokens: int = 8000,
timeout: float | None = None,
) -> str:
"""文本补全。失败抛 LLMError。"""
return await _complete(
system, user, temperature=temperature, max_tokens=max_tokens, json_mode=False,
timeout=timeout,
)
def _repair_json(text: str) -> str:
text = text.strip()
if text.startswith("```"):
text = text.strip("`")
if text.lower().startswith("json"):
text = text[4:]
text = text.strip()
if not text:
raise LLMError("LLM 返回空内容")
if text.startswith("{"):
return text
start = text.find("{")
end = text.rfind("}")
if start >= 0 and end > start:
return text[start:end + 1]
raise LLMError("LLM 输出中未找到 JSON 对象")
async def complete_json(
system: str,
user: str,
*,
temperature: float = 0.3,
max_tokens: int = 8000,
) -> dict[str, Any]:
"""JSON 补全(json_object + 一次修复重试)。"""
text = await _complete(
system, user, temperature=temperature, max_tokens=max_tokens, json_mode=True
)
try:
return json.loads(text)
except ValueError:
pass
text2 = await _complete(
system + "\n(重要:你的回答必须是单个合法 JSON 对象,不要输出其他任何内容)",
user,
temperature=temperature,
max_tokens=max_tokens,
json_mode=True,
)
try:
return json.loads(text2)
except ValueError:
repaired = _repair_json(text2)
return json.loads(repaired)
@@ -0,0 +1,113 @@
"""outline.py —— propose_paper_outline:建 papers 行 → search_refs 注入线索 → LLM 出大纲 → 预填 sections。"""
from __future__ import annotations
import uuid
from typing import Any
from ...db import db, dumps_json, now_iso
from . import llm, search
_OUTLINE_SYSTEM = (
"你是植物学论文撰写助手。根据用户给出的论文主题与写作要求,生成论文大纲。"
"只输出一个合法 JSON 对象,结构为:"
'{"title": 论文标题, "abstract_expect": 摘要预期内容(一句话), '
'"sections": [{"section_id": "s1", "title": "节标题", "key_points": ["要点1", "要点2"], "target_words": 1200}, ...]}。'
"章节 5-6 节,覆盖 引言/材料与方法/结果/讨论/结论 等综述常见结构;target_words 每节 1200-2500。"
)
_DEFAULT_SECTIONS = [
("s1", "引言"),
("s2", "材料与方法"),
("s3", "结果"),
("s4", "讨论"),
("s5", "结论与展望"),
]
async def propose_paper_outline(
topic: str,
requirements: str = "",
paper_type: str = "",
language: str = "",
length: str = "",
format_req: str = "",
style: str = "",
kb_hints: list[dict[str, Any]] | None = None,
template_key: str = "",
) -> dict[str, Any]:
# 1. 知识库线索:优先用调用方传入的 kb_hints,否则现场 search_refs(失败静默降级)
hints = list(kb_hints or [])
if not hints:
try:
r = await search.search_refs(query=topic, top_k=8)
hints = list(r.get("items") or [])
except Exception:
hints = []
# 2. LLM 出大纲
user = f"论文主题:{topic}\n"
if paper_type:
user += f"论文类型:{paper_type}\n"
if requirements:
user += f"写作要求:{requirements}\n"
if language:
user += f"语言:{language}\n"
if length:
user += f"篇幅:{length}\n"
if format_req:
user += f"格式:{format_req}\n"
if style:
user += f"风格:{style}\n"
if hints:
user += "可参考知识库文献线索(仅选题依据,非硬性引用):\n" + "\n".join(
f"- {h.get('title', '')}: {str(h.get('snippet', ''))[:120]}" for h in hints[:5]
)
try:
outline = await llm.complete_json(_OUTLINE_SYSTEM, user)
except llm.LLMError as exc:
return {"ok": False, "code": "LLM_ERROR", "message": f"大纲生成失败:{exc}"}
title = str(outline.get("title") or topic).strip() or topic
sections = outline.get("sections") or []
if not sections:
sections = [
{"section_id": sid, "title": t, "key_points": [], "target_words": 1500}
for sid, t in _DEFAULT_SECTIONS
]
# 3. 落库 papers + paper_sections
paper_id = uuid.uuid4().hex[:16]
await db.execute(
"INSERT INTO papers (id, title, paper_type, language, format_req, style, status, brief_json, outline_json, created_at, updated_at) "
"VALUES (?, ?, ?, ?, ?, ?, 'outline', ?, ?, ?, ?)",
(
paper_id, title, paper_type, language, format_req, style,
dumps_json({
"topic": topic, "requirements": requirements, "length": length,
"kb_hints": hints, "template_key": template_key,
}),
dumps_json({
"title": title,
"abstract_expect": outline.get("abstract_expect", ""),
"sections": sections,
}),
now_iso(), now_iso(),
),
)
for s in sections:
await db.execute(
"INSERT INTO paper_sections (id, paper_id, section_id, title, key_points, status, updated_at) "
"VALUES (?, ?, ?, ?, ?, 'pending', ?)",
(
uuid.uuid4().hex[:16], paper_id,
s.get("section_id") or "s1", s.get("title", ""),
dumps_json(s.get("key_points") or []), now_iso(),
),
)
return {
"ok": True,
"paper_id": paper_id,
"title": title,
"abstract_expect": outline.get("abstract_expect", ""),
"sections": sections,
}
@@ -0,0 +1,101 @@
"""search.py —— vstore 8094 知识库检索客户端(search_refs 后端)。
- httpx 裸调 GET /api/vstore/search-chunks
- 失败不抛异常:返回 {"ok": False, "error": {...}},工具层降级为空结果
- vstore chunk 条目无 doi/pmid/journal/year(仅 doc_id/doc_name/snippet/score),
故 search_refs 的 meta 为尽力而为(通常为空)→ 引用多为 title kind → 降级格式 + disclaimer
"""
from __future__ import annotations
import asyncio
import os
from typing import Any
import httpx
DEFAULT_VSTORE_BASE_URL = "http://127.0.0.1:8094"
RETRY_BACKOFF = (0.5, 1.0, 2.0)
MAX_RETRIES = 2
DEFAULT_TIMEOUT = 30.0
_client: httpx.AsyncClient | None = None
def _base_url() -> str:
return (os.environ.get("VSTORE_BASE_URL") or DEFAULT_VSTORE_BASE_URL).rstrip("/")
def get_client() -> httpx.AsyncClient:
global _client
if _client is None:
_client = httpx.AsyncClient(
base_url=_base_url(), timeout=httpx.Timeout(DEFAULT_TIMEOUT, connect=5.0)
)
return _client
def set_client(client: httpx.AsyncClient) -> None:
"""注入 client(测试用 MockTransport)。"""
global _client
_client = client
async def close_client() -> None:
global _client
if _client is not None:
await _client.aclose()
_client = None
async def search_chunks(
query: str,
*,
top_k: int = 8,
min_score: float = 0.30,
timeout: float = 12.0,
) -> dict[str, Any]:
"""GET /api/vstore/search-chunks。失败返回 {ok:False, error:{...}}(不抛)。"""
client = get_client()
last_exc: BaseException | None = None
for attempt in range(MAX_RETRIES + 1):
try:
resp = await client.get(
"/api/vstore/search-chunks",
params={"q": query, "top_k": top_k, "min_score": min_score},
timeout=timeout,
)
if resp.status_code < 500:
try:
payload = resp.json()
except ValueError:
return {"ok": False, "error": {"code": "BAD_RESPONSE", "message": "vstore 响应非 JSON"}}
if not isinstance(payload, dict):
return {"ok": False, "error": {"code": "BAD_RESPONSE", "message": "vstore 响应形态异常"}}
return payload
last_exc = RuntimeError(f"vstore {resp.status_code}")
except (httpx.HTTPError, httpx.TimeoutException) as exc:
last_exc = exc
if attempt < MAX_RETRIES:
await asyncio.sleep(RETRY_BACKOFF[min(attempt, len(RETRY_BACKOFF) - 1)])
return {"ok": False, "error": {"code": "UPSTREAM_DOWN", "message": f"vstore 不可达:{last_exc}"}}
async def search_refs(paper_id: str = "", query: str = "", top_k: int = 8) -> dict[str, Any]:
"""search_refs 工具:透传 vstore 检索,映射为 {doc_id,title,snippet,score,meta}。"""
if not query:
return {"ok": False, "code": "BAD_REQUEST", "message": "缺少检索词 query"}
r = await search_chunks(query, top_k=max(1, min(int(top_k), 20)))
if not r.get("ok"):
return {"ok": True, "items": [], "warning": (r.get("error") or {}).get("message", "")}
items: list[dict[str, Any]] = []
for it in r.get("items") or []:
if not isinstance(it, dict):
continue
items.append({
"doc_id": it.get("doc_id") or "",
"title": it.get("doc_name") or it.get("doc_id") or "",
"snippet": it.get("snippet") or "",
"score": it.get("score"),
"meta": {}, # vstore chunk 无 doi/pmid/journal/year,尽力为空
})
return {"ok": True, "items": items}
@@ -0,0 +1,251 @@
"""write.py —— 章节撰写与修订(write_paper_section / revise_paper_section / revise_paper)。"""
from __future__ import annotations
from typing import Any
from ...db import db, dumps_json, loads_json, now_iso
from . import llm
from .cite import find_cite_numbers, render_gbt7714
_WRITE_SYSTEM = (
"你是植物学论文撰写助手,正在撰写论文的某一章节。"
"引用文献一律用 [n] 指代,n 必须取自下方提供的全局编号列表;如需引用新文献,编号按当前最大编号依次 +1。"
"你只能引用「知识库线索」中列出的文献;新引用的元数据只能照抄线索里的 title 和 doc_id。"
"知识库线索里没有 DOI/PMID/作者/期刊/年份,这些字段一律输出空字符串,禁止编造任何 DOI/PMID/作者/期刊。"
"只输出一个合法 JSON 对象:{\"markdown\": 章节正文(不含章标题), \"citations_used\": "
"[{\"temp_n\": 编号, \"title\": \"...\", \"authors\": \"\", \"journal\": \"\", "
"\"year\": \"\", \"doi\": \"\", \"pmid\": \"\", \"source_doc_id\": \"...\", \"snippet\": \"...\"}]}。"
"citations_used 必须列出本节正文中出现的每一个 [n] 引用(既有编号也列出其已有元数据)。"
)
_REVISE_SYSTEM = (
"你是植物学论文撰写助手,正在按用户意见修订论文的某一章节。"
"保持引用协议不变:[n] 指代,n 只能取自全局编号列表或最大编号 +1;禁止杜撰元数据。"
"只输出一个合法 JSON 对象:{\"markdown\": 修订后的章节正文, \"citations_used\": [...]}。"
)
# ---------------------------------------------------------------------------
# 读取辅助
# ---------------------------------------------------------------------------
async def _get_paper(paper_id: str) -> dict[str, Any] | None:
row = await db.fetchone("SELECT * FROM papers WHERE id = ?", (paper_id,))
return dict(row) if row else None
async def _get_sections(paper_id: str) -> list[dict[str, Any]]:
rows = await db.fetchall(
"SELECT * FROM paper_sections WHERE paper_id = ? ORDER BY rowid", (paper_id,)
)
return [dict(r) for r in rows]
async def _get_citations(paper_id: str) -> list[dict[str, Any]]:
rows = await db.fetchall(
"SELECT * FROM citations WHERE paper_id = ? ORDER BY temp_n", (paper_id,)
)
return [dict(r) for r in rows]
def _global_citations_text(citations: list[dict[str, Any]]) -> str:
if not citations:
return "(当前无已编号文献;新引用从 [1] 开始)"
return "\n".join(f"[{c['temp_n']}] {c['ref_text']}" for c in citations)
def _max_temp_n(citations: list[dict[str, Any]]) -> int:
return max((c.get("temp_n") or 0 for c in citations), default=0)
def _build_write_user(paper: dict, outline: dict, sec: dict, cit_text: str, style: str, cite_mode: str) -> str:
parts: list[str] = []
parts.append(f"论文标题:{paper.get('title', '')}")
parts.append(f"论文类型:{paper.get('paper_type') or '综述'}")
parts.append(f"语言:{paper.get('language') or '中文'}")
parts.append(f"格式要求:{paper.get('format_req') or '无'}")
parts.append(f"写作风格:{style or paper.get('style') or '学术严谨'}")
parts.append("")
parts.append("论文大纲:")
for s in outline.get("sections") or []:
mark = "(当前撰写节)" if s.get("section_id") == sec.get("section_id") else ""
parts.append(f"- {s.get('section_id')}: {s.get('title')}{mark} 要点:{';'.join(s.get('key_points') or [])}")
parts.append("")
parts.append(f"现在撰写章节「{sec.get('title', '')}」(section_id={sec.get('section_id')})。")
if sec.get("key_points"):
parts.append("本节要点:" + ";".join(loads_json(sec.get("key_points"), [])))
parts.append("")
parts.append("全局已编号文献列表([n] 引用只能从这里取值,或从最大编号 +1 起新增):")
parts.append(cit_text)
brief = loads_json(paper.get("brief_json"), {})
kb_hints = brief.get("kb_hints") or []
if kb_hints:
parts.append("")
parts.append("知识库线索(新引用只能从这里选,元数据只照抄 title/doc_id):")
for h in kb_hints:
parts.append(f"- title={h.get('title', '')} | doc_id={h.get('doc_id', '')} | snippet={str(h.get('snippet', ''))[:120]}")
return "\n".join(parts)
def _validate_numbers(markdown: str, valid: set[int]) -> list[int]:
return [n for n in find_cite_numbers(markdown) if n not in valid]
def _kind_of(meta: dict[str, Any]) -> str:
if meta.get("doi"):
return "doi"
if meta.get("pmid"):
return "pmid"
return "title"
def _sanitize_citation(meta: dict[str, Any]) -> dict[str, Any]:
"""丢弃 LLM 可能杜撰的 doi/pmid/authors/journal/year(知识库线索无这些字段),
只保留 title/source_doc_id/snippet,强制走 title kind 降级格式。"""
return {
"title": (meta.get("title") or "").strip(),
"source_doc_id": meta.get("source_doc_id") or "",
"snippet": meta.get("snippet") or "",
}
async def _persist_new_citations(paper_id: str, section_id: str, used: list[Any], existing_nums: set[int]) -> list[dict[str, Any]]:
"""把 citations_used 中的新引用(temp_n 不在既有集合)落库,返回新增摘要。"""
new_citations: list[dict[str, Any]] = []
seen_new: set[int] = set()
for c in used:
if not isinstance(c, dict):
continue
tn = int(c.get("temp_n") or 0)
if not tn or tn in existing_nums or tn in seen_new:
continue
seen_new.add(tn)
meta = _sanitize_citation(c)
ref_text = render_gbt7714(meta)
await db.execute(
"INSERT INTO citations (paper_id, section_id, temp_n, kind, ref_text, source_doc_id, snippet, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
(
paper_id, section_id, tn, _kind_of(meta), ref_text,
meta.get("source_doc_id") or "", meta.get("snippet") or "", now_iso(),
),
)
new_citations.append({"temp_n": tn, "ref_text": ref_text})
return new_citations
# ---------------------------------------------------------------------------
# 写节
# ---------------------------------------------------------------------------
async def write_paper_section(paper_id: str, section_id: str, style: str = "", cite_mode: str = "") -> dict[str, Any]:
paper = await _get_paper(paper_id)
if not paper:
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
sections = await _get_sections(paper_id)
sec = next((s for s in sections if s["section_id"] == section_id), None)
if not sec:
return {"ok": False, "code": "NOT_FOUND", "message": "章节不存在"}
citations = await _get_citations(paper_id)
outline = loads_json(paper.get("outline_json"), {})
cit_text = _global_citations_text(citations)
existing_nums = {c.get("temp_n") for c in citations}
user = _build_write_user(paper, outline, sec, cit_text, style, cite_mode)
markdown = ""
used: list[Any] = []
bad: list[int] = []
for attempt in range(2):
try:
res = await llm.complete_json(_WRITE_SYSTEM, user, max_tokens=8000)
except llm.LLMError as exc:
return {"ok": False, "code": "LLM_ERROR", "message": f"写节 LLM 失败:{exc}"}
markdown = res.get("markdown") or ""
used = res.get("citations_used") or []
used_nums = {int(c.get("temp_n") or 0) for c in used if isinstance(c, dict)}
valid = existing_nums | used_nums
bad = _validate_numbers(markdown, valid)
if not bad:
break
user += f"\n(上次正文出现的引用编号 {sorted(bad)} 非法;只能使用已有编号或从 {_max_temp_n(citations) + 1} 起的新编号,重试。)"
else:
return {"ok": False, "code": "CITE_VALIDATION_FAILED", "message": f"引用编号非法:{sorted(bad)}"}
new_citations = await _persist_new_citations(paper_id, section_id, used, existing_nums)
await db.execute(
"UPDATE paper_sections SET markdown = ?, word_count = ?, status = 'done', updated_at = ? WHERE paper_id = ? AND section_id = ?",
(markdown, len(markdown), now_iso(), paper_id, section_id),
)
await db.execute("UPDATE papers SET status = 'writing', updated_at = ? WHERE id = ?", (now_iso(), paper_id))
return {"ok": True, "section_id": section_id, "status": "done", "markdown": markdown,
"citations_used": used, "new_citations": new_citations}
# ---------------------------------------------------------------------------
# 修订
# ---------------------------------------------------------------------------
async def revise_paper_section(paper_id: str, section_id: str, instruction: str) -> dict[str, Any]:
paper = await _get_paper(paper_id)
if not paper:
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
sections = await _get_sections(paper_id)
sec = next((s for s in sections if s["section_id"] == section_id), None)
if not sec:
return {"ok": False, "code": "NOT_FOUND", "message": "章节不存在"}
citations = await _get_citations(paper_id)
outline = loads_json(paper.get("outline_json"), {})
existing_nums = {c.get("temp_n") for c in citations}
user = _build_write_user(paper, outline, sec, _global_citations_text(citations), "", "")
user += "\n\n原章节正文:\n" + (sec.get("markdown") or "")
user += f"\n\n修订要求:{instruction}"
markdown = ""
used: list[Any] = []
bad: list[int] = []
for attempt in range(2):
try:
res = await llm.complete_json(_REVISE_SYSTEM, user, max_tokens=8000)
except llm.LLMError as exc:
return {"ok": False, "code": "LLM_ERROR", "message": f"修订 LLM 失败:{exc}"}
markdown = res.get("markdown") or ""
used = res.get("citations_used") or []
used_nums = {int(c.get("temp_n") or 0) for c in used if isinstance(c, dict)}
valid = existing_nums | used_nums
bad = _validate_numbers(markdown, valid)
if not bad:
break
user += f"\n(引用编号 {sorted(bad)} 非法,重试。)"
else:
return {"ok": False, "code": "CITE_VALIDATION_FAILED", "message": f"引用编号非法:{sorted(bad)}"}
new_citations = await _persist_new_citations(paper_id, section_id, used, existing_nums)
await db.execute(
"UPDATE paper_sections SET markdown = ?, word_count = ?, status = 'done', updated_at = ? WHERE paper_id = ? AND section_id = ?",
(markdown, len(markdown), now_iso(), paper_id, section_id),
)
return {"ok": True, "section_id": section_id, "markdown": markdown,
"citations_used": used, "citation_delta": len(new_citations), "new_citations": new_citations}
async def revise_paper(paper_id: str, instruction: str) -> dict[str, Any]:
"""LLM 判定受影响章节,返回受影响节清单 + 修订计划(由调用方逐节 revise_paper_section)。"""
paper = await _get_paper(paper_id)
if not paper:
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
sections = await _get_sections(paper_id)
outline = loads_json(paper.get("outline_json"), {})
system = (
"你是论文修订计划助手。根据用户修订意见与论文大纲,判定需要修改哪些章节。"
"只输出一个合法 JSON 对象:{\"affected_section_ids\": [\"s1\", ...], \"plan\": 修订计划(一句话)}。"
)
sec_list = "\n".join(f"{s['section_id']}: {s['title']}" for s in sections)
user = f"论文标题:{paper.get('title')}\n大纲章节:\n{sec_list}\n\n修订意见:{instruction}"
try:
res = await llm.complete_json(system, user, max_tokens=2000)
except llm.LLMError as exc:
return {"ok": False, "code": "LLM_ERROR", "message": f"修订计划 LLM 失败:{exc}"}
affected = [s for s in (res.get("affected_section_ids") or []) if any(x["section_id"] == s for x in sections)]
return {"ok": True, "affected_section_ids": affected, "plan": res.get("plan", "")}
+119
View File
@@ -0,0 +1,119 @@
#!/usr/bin/env bash
# build.sh —— PAPER 服务编译硬化(复制自 bioflow_service/build.sh,白名单改造)。
# 白名单(明文):app/main.py(uvicorn 入口)、app/config.py(运维可读面)、
# app/schemas.py(常量面)、app/db.py(aiosqlite 协程取消热区——Cython 协程断流断言
# abort 是既有教训,明文保留)、app/mcp.py(MCP 工具层,mcp 依赖 pydantic + async 工具)、
# services/paper/{llm,search,write,outline,complete}.py(长 LLM/httpx/aiosqlite 调用,
# 断流崩溃高危点,明文走 CPython 原生协程)+ 全部 __init__.py。
# 仅纯同步逻辑(cite.py 等)可编译。
# 用法:build.sh <source-root> <output-dir> [python]
set -euo pipefail
umask 027
SOURCE_ROOT="${1:?Usage: $0 <source-root> <output-dir> [python]}"
OUTPUT_DIR="${2:?Usage: $0 <source-root> <output-dir> [python]}"
PYTHON="${3:-python3}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VISIBILITY_TOOL="${ELF_VISIBILITY_TOOL:-$SCRIPT_DIR/../tooling/compile/tools/elf_visibility.py}"
SOURCE_ROOT="$(realpath "$SOURCE_ROOT")"
OUTPUT_DIR="$(realpath -m "$OUTPUT_DIR")"
[[ -d "$SOURCE_ROOT" ]] || { echo "Paper service source missing: $SOURCE_ROOT" >&2; exit 2; }
[[ -x "$PYTHON" ]] || { echo "Python is not executable: $PYTHON" >&2; exit 2; }
[[ -f "$VISIBILITY_TOOL" ]] || { echo "ELF visibility tool missing: $VISIBILITY_TOOL" >&2; exit 2; }
for command_name in gcc strip rsync; do command -v "$command_name" >/dev/null; done
case "$OUTPUT_DIR" in
"/"|"$(realpath -m "$HOME")"|"$SOURCE_ROOT")
echo "Unsafe output directory: $OUTPUT_DIR" >&2; exit 2 ;;
esac
[[ ! -e "$OUTPUT_DIR" ]] || { echo "Output exists; refusing to overwrite: $OUTPUT_DIR" >&2; exit 2; }
echo "[PAPER BUILD 1/5] Copying source (excluding tests)"
mkdir -p "$OUTPUT_DIR"
rsync -a --exclude '__pycache__' --exclude 'test_*.py' --exclude 'data' --exclude '.env' \
--exclude 'build.sh' --exclude 'restart.sh' --exclude 'pytest.ini' \
"$SOURCE_ROOT/" "$OUTPUT_DIR/"
WHITELIST_PY=(
"app/main.py"
"app/config.py"
"app/schemas.py"
"app/db.py"
"app/mcp.py"
"app/services/paper/llm.py"
"app/services/paper/search.py"
"app/services/paper/write.py"
"app/services/paper/outline.py"
"app/services/paper/complete.py"
)
_is_whitelisted() {
local rel="$1"
[[ "$(basename "$rel")" == "__init__.py" ]] && return 0
for w in "${WHITELIST_PY[@]}"; do
case "$rel" in
$w) return 0 ;;
esac
done
return 1
}
echo "[PAPER BUILD 2/5] Discovering Python modules"
PY_FILES=()
while IFS= read -r -d '' path; do PY_FILES+=("$path"); done \
< <(find "$OUTPUT_DIR" -type f -name '*.py' -print0)
FILTERED=()
for py_file in "${PY_FILES[@]}"; do
rel="${py_file#"$OUTPUT_DIR"/}"
if _is_whitelisted "$rel"; then
echo " [whitelist] $rel (明文保留)"
continue
fi
FILTERED+=("$py_file")
done
PY_FILES=("${FILTERED[@]}")
TOTAL="${#PY_FILES[@]}"
(( TOTAL > 0 )) || { echo "No paper service modules selected" >&2; exit 3; }
INCLUDE_PY="$("$PYTHON" -c 'import sysconfig; print(sysconfig.get_config_var("INCLUDEPY") or "")')"
EXT_SUFFIX="$("$PYTHON" -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX") or ".so")')"
echo "[PAPER BUILD 3/5] Cython preflight ($TOTAL modules)"
COUNT=0; FAILURES=0
for py_file in "${PY_FILES[@]}"; do
COUNT=$((COUNT+1))
directory="$(dirname "$py_file")"
module="$(basename "$py_file" .py)"
c_file="$directory/$module.c"
echo " [$COUNT/$TOTAL] ${py_file#"$OUTPUT_DIR"/}"
if ! "$PYTHON" -m cython -3 \
--directive always_allow_keywords=true,annotation_typing=False \
--no-docstrings "$py_file" -o "$c_file" 2>/dev/null; then
FAILURES=$((FAILURES+1))
fi
done
(( FAILURES == 0 )) || { echo "Paper service Cython failed: $FAILURES" >&2; exit 3; }
echo "[PAPER BUILD 4/5] GCC hidden visibility + strip"
COUNT=0
for py_file in "${PY_FILES[@]}"; do
COUNT=$((COUNT+1))
directory="$(dirname "$py_file")"
module="$(basename "$py_file" .py)"
c_file="$directory/$module.c"
so_file="$directory/$module$EXT_SUFFIX"
gcc -shared -pthread -fPIC -O2 -fvisibility=hidden \
-I"$INCLUDE_PY" -o "$so_file" "$c_file"
strip --strip-unneeded "$so_file"
"$PYTHON" -B "$VISIBILITY_TOOL" --hide-non-pyinit --check "$so_file" >/dev/null
rm -f -- "$c_file" "$py_file"
done
echo "[PAPER BUILD 5/5] Final checks"
SO_COUNT="$(find "$OUTPUT_DIR" -type f -name '*.so' | wc -l)"
[[ "$SO_COUNT" -eq "$TOTAL" ]] || { echo "count mismatch: $TOTAL vs $SO_COUNT" >&2; exit 4; }
if find "$OUTPUT_DIR" -type f -name '*.c' -print -quit | grep -q .; then
echo "Generated C source remains" >&2; exit 4
fi
echo "Paper service compile+harden complete: modules=$SO_COUNT output=$OUTPUT_DIR"
+6
View File
@@ -0,0 +1,6 @@
[pytest]
python_files = test_*.py
markers =
paper_unit: 论文服务单元测试(引用协议/大纲/写节/重排/complete)
paper_mcp: MCP 协议互操作测试
asyncio_mode = auto
+34
View File
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# restart.sh —— PAPER 服务重启脚本(8097;不触碰生产服务 8090/8091/8092/8093/8094/8098)。
# 零硬编码:venv/python/仓库路径全部从脚本位置推导或环境变量覆盖。
# 用法: restart.sh [python] [port]
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
PYTHON="${1:-$REPO_ROOT/../plantai-rag-new-venv/bin/python}"
PORT="${2:-8097}"
HOST="${PAPER_BIND_HOST:-127.0.0.1}"
[[ -x "$PYTHON" ]] || { echo "Python not found: $PYTHON" >&2; exit 2; }
APP_MODULE="app.main:app"
if command -v lsof >/dev/null 2>&1; then
PID="$(lsof -ti tcp:"$PORT" -sTCP:LISTEN 2>/dev/null || true)"
if [[ -n "$PID" ]]; then
echo "[restart] 停止 $PORT 旧进程 pid=$PID"
kill $PID 2>/dev/null || true
sleep 2
fi
fi
cd "$SCRIPT_DIR"
# 加载运行时配置(.env 不入 git):services/paper/llm.py 直接读 DOCGEN_LLM_* 裸键,
# search.py 读 VSTORE_BASE_URL——需 set -a 显式导出(同 bioflow_service/restart.sh 先例)。
set -a
[ -f .env ] && source .env
set +a
echo "[restart] 启动论文撰写服务 :$PORT($PYTHON)"
nohup "$PYTHON" -m uvicorn "$APP_MODULE" --host "$HOST" --port "$PORT" \
> "$SCRIPT_DIR/paper.log" 2>&1 &
echo "[restart] 完成;日志 paper_writer_service/paper.log"
+27
View File
@@ -0,0 +1,27 @@
# review_service/.env.example —— 运行时配置模板
# 用法: cp .env.example .env 后填写(.env 已被仓库 .gitignore 忽略,勿提交)
# restart.sh 会 `set -a; source .env` 导出给 uvicorn 进程。
# ---- LLM(DeepSeek;REVIEW_LLM_* 优先 → DOCGEN_LLM_* 回退 → 内置默认,见 services/llm.py)----
# 最简做法:从部署树抄一行即可复用同一把 key:
# grep '^DOCGEN_LLM_API_KEY=' /home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new/sdk/.env >> .env
REVIEW_LLM_BASE_URL=https://api.deepseek.com/v1
REVIEW_LLM_API_KEY=
REVIEW_LLM_MODEL=deepseek-chat
#REVIEW_LLM_TIMEOUT=300
# ---- 上游(只读消费)----
#REVIEW_VSTORE_BASE_URL=http://127.0.0.1:8094
#REVIEW_METHODOLOGY_BASE_URL=http://127.0.0.1:8092 # 预留(MinerU 重链路,当前未用)
# ---- PDF 解析(经 vstore)----
#REVIEW_PARSE_TIMEOUT=180 # ingest→取文→清理 总读超时(秒)
#REVIEW_PARSE_DELETE_AFTER=1 # 1=解析后删除 vstore 临时文档(0=保留便于排查)
# ---- 自审稿 ----
#REVIEW_PRESUBMIT_READ_TIMEOUT=360 # LLM 长调用读超时(秒)
#REVIEW_MAX_TEXT_CHARS=60000 # 送 LLM 的正文截断上限
#REVIEW_MAX_PDF_BYTES=52428800 # 上传 PDF 上限 50MB
# ---- 服务(restart.sh 读取)----
#REVIEW_BIND_HOST=127.0.0.1
+7
View File
@@ -0,0 +1,7 @@
"""review_service —— 科研投稿与智能审稿助手(8100)。
独立部署树(模式同 paper_writer_service 8097 / bioflow_service 8098):
- BFF(sdk 8090) 经 routers/review_proxy.py 条件反代 /api/review/*(REVIEW_ENABLED)。
- 上游只读消费:vstore 8094(PDF 解析)、DeepSeek(LLM,REVIEW_LLM_*/DOCGEN_LLM_*)。
- 约束见根目录 AGENTS.md:审稿相关新代码只允许落在本目录。
"""
+1
View File
@@ -0,0 +1 @@
"""review 服务(8100)应用包。"""
+154
View File
@@ -0,0 +1,154 @@
"""config.py —— review 服务(8100)环境变量驱动配置(pydantic-settings)。
全部键可用环境变量覆盖(前缀 REVIEW_)。LLM 沿用 docgen 的裸键模式:
优先 REVIEW_LLM_*,回退 DOCGEN_LLM_*(与 paper_writer_service 共享同一套
DeepSeek 凭据,由 services/llm.py 直接读 os.environ)。零硬编码宿主路径。
"""
from __future__ import annotations
from functools import lru_cache
from pathlib import Path
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
env_prefix="REVIEW_",
)
app_name: str = "plantai-review-service"
version: str = "0.1.0"
# ---- 上游(只读消费;不修改 vstore/methodology 任何代码)----
# vstore 8094:POST /api/vstore/ingest + GET/DELETE /api/vstore/documents/{id}
# (pymupdf 段落解析,同步秒级;methodology 8092 的 MinerU 链路是
# 分钟级异步 job,不适配同步自审稿,仅预留地址供后续扩展)
vstore_base_url: str = "http://127.0.0.1:8094"
methodology_base_url: str = "http://127.0.0.1:8092"
# ---- PDF 解析 ----
parse_timeout: float = 180.0 # vstore ingest→fetch→delete 读超时(秒)
parse_delete_after: bool = True # 解析完成后删除 vstore 临时文档(不污染私有库)
# ---- 自审稿(presubmit)----
presubmit_read_timeout: float = 360.0 # LLM 长调用读超时(秒)
max_text_chars: int = 150000 # 送 LLM 的正文截断上限(REVIEW_MAX_TEXT_CHARS 覆盖;
# token 超 48K 自动降 100K,LLM 400/413 再降 80K 重试)
max_pdf_bytes: int = 50 * 1024 * 1024 # 上传 PDF 上限 50MB(对齐 extraction FR-EXTRACT-004)
# ---- 期刊情报与投稿推荐(recommend)----
db_path: str = "review.db" # 独立 SQLite(相对 review_service/;REVIEW_DB_PATH 覆盖)
journals_seed_path: str = "data/journals/journals.json" # 种子(相对 review_service/)
journal_directory_seed_path: str = "data/journals/source_v4.xlsx" # 目录源 xlsx(相对 review_service/)
seed_on_startup: bool = True # 启动自动 seed journal_directory(空表才导;false 关闭)
vstore_match_timeout: float = 30.0 # scope 入库/向量检索读超时(秒)
recommend_read_timeout: float = 60.0 # LLM 重排读超时(秒)
recommend_query_chars: int = 3000 # 送 vstore 的检索 query 截断(防嵌入模型超长)
recommend_scope_top_k: int = 10 # search-chunks top_k(vstore 上限 20)
# ---- 论文重排版(reformat,P1:.tex 源码级变换,不编译)----
max_tex_bytes: int = 2 * 1024 * 1024 # 上传 .tex 上限 2MB(PDF 的 50MB 不适用)
reformat_dir: str = "data/reformat_jobs" # 产物落盘(相对 review_service/)
# ---- LaTeX 编译 PDF(compile,P3:tectonic 单文件二进制)----
tectonic_path: str = "tools/tectonic" # 相对 review_service/;也可绝对路径
tectonic_cache_dir: str = "data/tectonic-cache" # 宏包缓存(部署时预热,运行期只读命中)
compile_timeout_s: float = 60.0 # 单次编译墙钟上限,超时整进程组 kill
max_compile_pdf_bytes: int = 50 * 1024 * 1024 # 产物 PDF 上限 50MB
compile_queue_limit: int = 5 # 等待槽位上限(>5 → QUEUE_FULL 429)
compile_retention_days: int = 7 # 编译产物保留天数(后台清扫)
# ---- LaTeX 项目 zip 编译(P4:两段式 inspect + run)----
max_zip_bytes: int = 50 * 1024 * 1024 # 上传 zip 上限 50MB
max_extract_bytes: int = 200 * 1024 * 1024 # 单项目解压总大小硬顶 200MB(实写字节计数)
max_zip_entries: int = 2000 # 条目数上限(预扫第一件事)
max_zip_entry_bytes: int = 50 * 1024 * 1024 # 单条目解压上限 50MB
max_zip_ratio: float = 100.0 # 单条目压缩比 file_size/compress_size 上限
max_zip_path_depth: int = 16 # 路径深度上限(深层嵌套拒绝)
max_zip_path_len: int = 200 # 单条目路径长度上限
compile_project_timeout_s: float = 120.0 # 项目 zip 编译墙钟上限(单文件仍 compile_timeout_s)
inspect_dir: str = "data/compile_inspect" # zip 解压暂存(1 小时 TTL,相对 review_service/)
inspect_ttl_s: int = 3600 # inspect 暂存过期秒数
inspect_sweep_interval_s: float = 300.0 # inspect 后台清扫周期(lifespan 定时任务)
compile_jobs_dir: str = "data/compile_jobs" # 项目编译产物(pdf/log/meta,7 天 sweep)
failed_project_ttl_s: int = 24 * 3600 # 编译失败保留 project/ 现场秒数(诊断用)
keep_failed_projects: bool = False # 失败 project/ 永久保留(调试开关;
# 亦读裸环境变量 KEEP_FAILED_PROJECTS=1)
# ---- 科研汇报 PPT(P5:map-reduce 大纲 + python-pptx 初稿)----
ppt_jobs_dir: str = "data/ppt_jobs" # 产物(draft.pptx + meta,相对 review_service/)
ppt_styles_dir: str = "data/ppt_styles" # 风格 tokens JSON(源码资产,随 git 走)
@property
def _service_root(self) -> "Path":
# review_service/app/config.py → review_service/(相对本文件定位,不依赖 cwd)
return Path(__file__).resolve().parent.parent
@property
def db_path_abs(self) -> "Path":
# 绝对 db_path 会被 Path 除法原样返回;相对路径落在 review_service/ 下
return self._service_root / self.db_path
@property
def journals_seed_abs(self) -> "Path":
return self._service_root / self.journals_seed_path
@property
def journal_directory_seed_abs(self) -> "Path":
return self._service_root / self.journal_directory_seed_path
@property
def reformat_dir_abs(self) -> "Path":
p = Path(self.reformat_dir)
return p if p.is_absolute() else self._service_root / p
@property
def ppt_jobs_abs(self) -> "Path":
p = Path(self.ppt_jobs_dir)
return p if p.is_absolute() else self._service_root / p
@property
def ppt_styles_abs(self) -> "Path":
p = Path(self.ppt_styles_dir)
return p if p.is_absolute() else self._service_root / p
@property
def tectonic_abs(self) -> "Path":
p = Path(self.tectonic_path)
return p if p.is_absolute() else self._service_root / p
@property
def tectonic_cache_abs(self) -> Path:
p = Path(self.tectonic_cache_dir)
return p if p.is_absolute() else self._service_root / p
@property
def inspect_dir_abs(self) -> Path:
p = Path(self.inspect_dir)
return p if p.is_absolute() else self._service_root / p
@property
def compile_jobs_dir_abs(self) -> Path:
p = Path(self.compile_jobs_dir)
return p if p.is_absolute() else self._service_root / p
@property
def vstore_base(self) -> str:
return self.vstore_base_url.rstrip("/")
@property
def methodology_base(self) -> str:
return self.methodology_base_url.rstrip("/")
@lru_cache(maxsize=1)
def get_settings() -> Settings:
return Settings()
settings = get_settings()
+191
View File
@@ -0,0 +1,191 @@
"""app/db.py —— review 服务独立 SQLite(aiosqlite + WAL,无 ORM):journals 表。
- 单连接 + asyncio 写锁(模式对齐 paper_writer_service/bioflow,但表结构与种子
逻辑为本模块自研);路径默认 review_service/review.db(REVIEW_DB_PATH 覆盖)。
- 种子:data/journals/journals.json → journals 表(幂等 upsert,vstore_doc_id 列
永不被种子覆盖——该列记录 scope_desc 在 vstore 8094 的 document_id)。
- 只在 review_service/ 内使用,不触碰 BFF/生产服务数据库。
"""
from __future__ import annotations
import asyncio
import json
from pathlib import Path
from typing import Any, Iterable
import aiosqlite
from .config import get_settings
_SCHEMA = """
CREATE TABLE IF NOT EXISTS journals (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
scope_desc TEXT NOT NULL,
impact_factor REAL,
review_time_weeks REAL,
acceptance_rate REAL,
submission_deadline TEXT,
official_url TEXT,
last_updated TEXT NOT NULL,
vstore_doc_id TEXT
);
"""
# 种子 upsert 列(刻意不含 vstore_doc_id:种子刷新不得清掉向量库映射)
_SEED_COLS = (
"id",
"name",
"scope_desc",
"impact_factor",
"review_time_weeks",
"acceptance_rate",
"submission_deadline",
"official_url",
"last_updated",
)
_UPSERT_SQL = (
"INSERT INTO journals (" + ", ".join(_SEED_COLS) + ") VALUES ("
+ ", ".join("?" for _ in _SEED_COLS) + ") "
+ "ON CONFLICT(id) DO UPDATE SET "
+ ", ".join(f"{c}=excluded.{c}" for c in _SEED_COLS if c != "id")
)
class SeedError(Exception):
"""种子文件缺失/损坏(journals.json 不可读)。"""
class Database:
"""aiosqlite 单连接封装(WAL;写串行化)。"""
def __init__(self, path: str | Path | None = None) -> None:
self._path: Path | None = Path(path) if path is not None else None
self._conn: aiosqlite.Connection | None = None
self._lock = asyncio.Lock()
@property
def path(self) -> Path:
if self._path is None:
self._path = get_settings().db_path_abs
return self._path
def set_path(self, path: str | Path) -> None:
"""测试注入:在进入 TestClient lifespan 之前预置库路径(须尚未 connect)。"""
if self._conn is not None:
raise RuntimeError("Database 已连接,无法再改路径(先 close_db())")
self._path = Path(path)
@property
def connected(self) -> bool:
return self._conn is not None
async def connect(self) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
self._conn = await aiosqlite.connect(self.path)
self._conn.row_factory = aiosqlite.Row
await self._conn.execute("PRAGMA journal_mode=WAL")
await self._conn.execute("PRAGMA busy_timeout=5000")
await self._conn.executescript(_SCHEMA)
await self._conn.commit()
async def close(self) -> None:
if self._conn is not None:
await self._conn.close()
self._conn = None
@property
def conn(self) -> aiosqlite.Connection:
if self._conn is None:
raise RuntimeError("Database 未初始化:先 await init_db()")
return self._conn
async def fetchall(self, sql: str, params: Iterable[Any] = ()) -> list[dict[str, Any]]:
async with self.conn.execute(sql, tuple(params)) as cur:
rows = await cur.fetchall()
return [dict(r) for r in rows]
async def fetchone(self, sql: str, params: Iterable[Any] = ()) -> dict[str, Any] | None:
async with self.conn.execute(sql, tuple(params)) as cur:
row = await cur.fetchone()
return dict(row) if row is not None else None
async def execute(self, sql: str, params: Iterable[Any] = ()) -> int:
async with self._lock:
cur = await self.conn.execute(sql, tuple(params))
await self._conn.commit() # type: ignore[union-attr]
return cur.lastrowid or 0
async def executemany(self, sql: str, seq: list[tuple[Any, ...]]) -> None:
async with self._lock:
await self.conn.executemany(sql, seq)
await self._conn.commit() # type: ignore[union-attr]
db = Database()
async def init_db(path: str | Path | None = None) -> None:
"""幂等初始化:已连接则跳过(测试可在 lifespan 前 set_path 注入路径)。"""
if path is not None:
db.set_path(path)
if db.connected:
return
await db.connect()
async def close_db() -> None:
await db.close()
def set_path(path: str | Path) -> None:
"""同步注入库路径(测试 fixture 用;须在 init_db 之前调用)。"""
db.set_path(path)
async def ensure_journals_seeded() -> int:
"""装载种子期刊(幂等)。返回 journals 表行数。"""
row = await db.fetchone("SELECT COUNT(*) AS c FROM journals")
if row and int(row["c"]) > 0:
return int(row["c"])
seed_file = get_settings().journals_seed_abs
try:
payload = json.loads(Path(seed_file).read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise SeedError(f"期刊种子文件不存在:{seed_file}") from exc
except (OSError, ValueError) as exc:
raise SeedError(f"期刊种子文件不可读:{exc}") from exc
journals = payload.get("journals") if isinstance(payload, dict) else None
if not isinstance(journals, list) or not journals:
raise SeedError(f"期刊种子文件缺少 journals 数组:{seed_file}")
rows: list[tuple[Any, ...]] = []
for item in journals:
if not isinstance(item, dict) or not item.get("id") or not item.get("name"):
continue
rows.append(tuple(item.get(c) for c in _SEED_COLS))
if not rows:
raise SeedError(f"期刊种子文件无有效条目:{seed_file}")
await db.executemany(_UPSERT_SQL, rows)
return len(rows)
async def list_journals() -> list[dict[str, Any]]:
"""全部期刊(impact_factor 降序,NULL 靠后)。"""
return await db.fetchall(
"SELECT * FROM journals ORDER BY impact_factor DESC, name ASC"
)
async def get_journal(journal_id: str) -> dict[str, Any] | None:
return await db.fetchone("SELECT * FROM journals WHERE id = ?", (journal_id,))
async def set_vstore_doc_id(journal_id: str, doc_id: str) -> None:
"""记录 scope_desc 在 vstore 的 document_id(向量白名单检索用)。"""
await db.execute(
"UPDATE journals SET vstore_doc_id = ? WHERE id = ?", (doc_id, journal_id)
)
+167
View File
@@ -0,0 +1,167 @@
"""main.py —— review 服务应用入口(uvicorn 入口)。
- 独立部署树 review_service/(科研投稿与智能审稿助手,端口 8100)。
- 端点:/health(GET,BFF 探活反代)+ /api/review/*(自审稿等业务路由)。
- 不依赖 BFF/生产服务(8090 等绝不探测/连接);上游仅消费 vstore 8094。
"""
from __future__ import annotations
import asyncio
import logging
import time
from contextlib import asynccontextmanager
import httpx
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from .config import get_settings
from .db import close_db, db, init_db
from .routers import (
compile,
conferences,
format_check,
journal_directory,
journals,
ppt,
presubmit,
reformat,
rebuttal,
)
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def create_app() -> FastAPI:
settings = get_settings()
app = FastAPI(title=settings.app_name, version=settings.version)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=False,
allow_methods=["*"],
allow_headers=["*"],
)
@asynccontextmanager
async def _lifespan(_app: FastAPI):
# journals 推荐依赖独立 SQLite:启动即建表(幂等;测试可先 set_path 注入
# tmp 路径,本函数沿用注入路径)
await init_db()
# rebuttal Phase 1:独立表,启动幂等建表(自含连接,不走 db.py)
try:
from .services.rebuttal_store import ensure_table
ensure_table()
except Exception: # noqa: BLE001
logger.warning("rebuttal 建表失败(忽略)", exc_info=True)
# P5 科研汇报:ppt_jobs 独立表,启动幂等建表
try:
from .services.ppt_store import ensure_table as ppt_ensure_table
ppt_ensure_table()
except Exception: # noqa: BLE001
logger.warning("ppt 建表失败(忽略)", exc_info=True)
# 编译产物过期清扫:启动跑一次(失败不阻塞启动)
try:
from .services.latex_compile import sweep_expired
sweep_expired()
except Exception: # noqa: BLE001
logger.warning("compile sweep 启动清扫失败(忽略)", exc_info=True)
# P4:inspect 暂存(1h TTL)+ 项目编译产物(7 天)启动清扫
try:
from .services.latex_compile import (
sweep_compile_jobs_expired,
sweep_inspect_expired,
)
sweep_inspect_expired()
sweep_compile_jobs_expired()
except Exception: # noqa: BLE001
logger.warning("P4 sweep 启动清扫失败(忽略)", exc_info=True)
# P4:inspect 暂存后台定时清扫(默认 300s;补充 1——上传不点编译不堆积)
sweep_task: asyncio.Task[None] | None = None
try:
from .services.latex_compile import sweep_inspect_expired as _sweep_insp
async def _inspect_sweep_loop() -> None:
interval = get_settings().inspect_sweep_interval_s
while True:
await asyncio.sleep(interval)
try:
await asyncio.to_thread(_sweep_insp)
except Exception: # noqa: BLE001
logger.warning("inspect 后台清扫失败(忽略)", exc_info=True)
sweep_task = asyncio.create_task(_inspect_sweep_loop())
except Exception: # noqa: BLE001
logger.warning("inspect 后台清扫任务启动失败(忽略)", exc_info=True)
# journal_directory 自动 seed:空表才导(xlsx 解析约 1-3s,不堵 event loop;
# 路径沿用 db.path 注入链——生产=settings.db_path_abs,测试=tmp)
if settings.seed_on_startup:
try:
from .services.journal_seed import ensure_journal_directory_seeded
_t0 = time.time()
_result = await asyncio.to_thread(
ensure_journal_directory_seeded, db.path)
if _result.get("seeded"):
_c = _result.get("counts") or {}
logger.info(
"seeded %s rows (en=%s, zh=%s, %.1fs)",
_result["rows"], _c.get("en"), _c.get("zh"),
time.time() - _t0)
elif _result.get("error"):
logger.warning(
"journal_directory 启动 seed 失败(忽略,不阻塞启动): %s",
_result["error"])
else:
logger.info("%s", _result.get("reason") or "journal_directory seed skipped")
except Exception: # noqa: BLE001
logger.warning("journal_directory 启动 seed 失败(忽略,不阻塞启动)", exc_info=True)
else:
logger.info("journal_directory seed skipped (REVIEW_SEED_ON_STARTUP=false)")
try:
yield
finally:
if sweep_task is not None:
sweep_task.cancel()
try:
await sweep_task
except asyncio.CancelledError:
pass
except Exception: # noqa: BLE001
pass
await close_db()
app.router.lifespan_context = _lifespan
@app.get("/health")
async def health() -> dict[str, object]:
vstore_ok = False
vstore_error: str | None = None
try:
async with httpx.AsyncClient(timeout=2.0) as client:
resp = await client.get(f"{settings.vstore_base}/api/vstore/health")
resp.raise_for_status()
vstore_ok = bool(resp.json().get("ok"))
except Exception as exc: # noqa: BLE001
vstore_error = str(exc)
return {
"ok": True,
"status": "ok" if vstore_ok else "degraded",
"upstreams": {"vstore": {"connected": vstore_ok, "error": vstore_error}},
"version": settings.version,
}
app.include_router(presubmit.router)
app.include_router(rebuttal.router)
app.include_router(journals.router)
app.include_router(format_check.router)
app.include_router(conferences.router)
app.include_router(journal_directory.router)
app.include_router(reformat.router)
app.include_router(compile.router)
app.include_router(ppt.router)
return app
app = create_app()
+1
View File
@@ -0,0 +1 @@
"""review 服务路由包。"""
+342
View File
@@ -0,0 +1,342 @@
"""routers/compile.py —— LaTeX 编译 PDF(P3a:tectonic)。
端点:
POST /api/review/compile multipart:file(.tex 可选) | job_id(可选,二选一)
→ {ok, compile_id, pdf_url, log_url, meta}
GET /api/review/compile/{cid} 编译元信息(meta.json)
GET /api/review/compile/{cid}/pdf output.pdf(FileResponse)
GET /api/review/compile/{cid}/log output.log(text/plain)
信封:业务错误 = HTTP 200 + {ok:false,error:{code,message,detail?,degraded:false}}
(QUEUE_FULL 例外:HTTP 429 + 同款信封体,前端仍按 ok/error 单一口径处理)。
detail = log 尾 200 行(COMPILE_FAILED / TIMEOUT)。
"""
from __future__ import annotations
import json
import logging
import re
import uuid
from pathlib import Path
from typing import Any
from fastapi import APIRouter, File, Form, UploadFile
from fastapi.responses import FileResponse, JSONResponse, PlainTextResponse
from pydantic import BaseModel
from ..config import get_settings
from ..services import latex_compile as lc
from ..services.latex_compile import (
CompileError,
QueueFullError,
ToolUnavailableError,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-compile"])
_CID_RE = re.compile(r"^[0-9a-f]{12}$")
class CompileResponse(BaseModel):
ok: bool
compile_id: str | None = None
pdf_url: str | None = None
log_url: str | None = None
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
def _error(code: str, message: str, detail: str = "") -> CompileResponse:
err: dict[str, Any] = {"code": code, "message": message, "degraded": False}
if detail:
err["detail"] = detail
return CompileResponse(ok=False, error=err)
def _jobs_root() -> Path:
d = get_settings().reformat_dir_abs
d.mkdir(parents=True, exist_ok=True)
return d
@router.post("/compile", response_model=CompileResponse)
async def compile_tex(
file: UploadFile | None = File(None, description="LaTeX 源码(.tex,与 job_id 二选一)"),
job_id: str = Form("", description="P1 重排版任务 id(优先于 file)"),
) -> CompileResponse:
settings = get_settings()
# ---- 输入解析:job_id 优先,否则直接上传的 .tex ----
jid = (job_id or "").strip()
if jid:
if not _CID_RE.fullmatch(jid):
return _error("VALIDATION", "job_id 格式非法(应为 12 位 hex)")
src = _jobs_root() / f"{jid}.tex"
if not src.is_file():
return _error("VALIDATION", "job_id 对应的重排版结果不存在,请重新排版")
try:
source = src.read_text("utf-8")
except UnicodeDecodeError:
return _error("VALIDATION", "任务源码不是合法 UTF-8")
anchor_id = jid
else:
if file is None:
return _error("VALIDATION", "必须提供 file(.tex)或 job_id")
filename = file.filename or "paper.tex"
if Path(filename).name != filename or ".." in filename:
return _error("VALIDATION", "非法文件名")
if not filename.lower().endswith(".tex"):
return _error("VALIDATION", "仅支持 .tex 文件")
content = await file.read()
if not content:
return _error("VALIDATION", "上传文件为空")
if len(content) > settings.max_tex_bytes:
return _error("VALIDATION", "源码超过 2MB 上限")
try:
source = content.decode("utf-8")
except UnicodeDecodeError:
return _error("VALIDATION", "源码必须为 UTF-8 编码")
anchor_id = uuid.uuid4().hex[:12]
# ---- 编译(隔离目录;失败信封带 log 尾)----
lc.prepare_input(anchor_id, source)
try:
result = await lc.compile_tex(anchor_id)
except ToolUnavailableError as exc:
return _error("TOOL_UNAVAILABLE", str(exc))
except QueueFullError as exc:
return JSONResponse(
status_code=429,
content={
"ok": False,
"compile_id": None, "pdf_url": None, "log_url": None, "meta": None,
"error": {"code": "QUEUE_FULL", "message": str(exc), "degraded": False},
},
)
except CompileError as exc:
return _error(exc.code, str(exc), detail=exc.log_tail)
meta_path = lc.compile_dir(anchor_id) / "meta.json"
try:
meta = json.loads(meta_path.read_text("utf-8"))
except (OSError, ValueError):
meta = {"compile_id": anchor_id, "seconds": result["seconds"]}
return CompileResponse(
ok=True,
compile_id=anchor_id,
pdf_url=f"/api/review/compile/{anchor_id}/pdf",
log_url=f"/api/review/compile/{anchor_id}/log",
meta=meta,
)
# ================================================================ P4:项目 zip(两段式)
def _p4_error(code: str, message: str, detail: str = "",
compile_id: str | None = None,
log_url: str | None = None) -> CompileResponse:
"""P4 错误信封(补充 3:失败也带 compile_id + log_url,前端可下载完整 log)。"""
err: dict[str, Any] = {"code": code, "message": message, "degraded": False}
if detail:
err["detail"] = detail
return CompileResponse(
ok=False, compile_id=compile_id, log_url=log_url, error=err)
def _project_pdf_url(cid: str) -> str:
return f"/api/review/compile/zip/{cid}/pdf"
def _project_log_url(cid: str) -> str:
return f"/api/review/compile/zip/{cid}/log"
@router.post("/compile/inspect", response_model=CompileResponse)
async def inspect_zip_project(
file: UploadFile = File(..., description="LaTeX 项目 zip(含 main.tex/.cls/图片/.bib)"),
) -> CompileResponse:
"""两段式之①:安全解压 + 主文件识别 → job_id(1 小时 TTL,run 前不占编译队列)。
暂用 request.body() 整体缓冲(≤50MB);Phase 2 改流式上传
"""
settings = get_settings()
filename = file.filename or "project.zip"
if Path(filename).name != filename or ".." in filename:
return _p4_error("VALIDATION", "非法文件名")
if not filename.lower().endswith(".zip"):
return _p4_error("VALIDATION", "仅支持 .zip 文件")
# 暂用 request.body() 整体缓冲(≤50MB);Phase 2 改流式上传
content = await file.read()
if not content:
return _p4_error("VALIDATION", "上传文件为空")
if len(content) > settings.max_zip_bytes:
return _p4_error(
"VALIDATION",
f"zip 超过 {settings.max_zip_bytes // (1024 * 1024)}MB 上限")
try:
meta = await lc.inspect_zip(content)
except lc.InspectError as exc:
return _p4_error(exc.code, exc.message)
return CompileResponse(ok=True, meta=meta)
class RunZipRequest(BaseModel):
job_id: str
@router.post("/compile/run", response_model=CompileResponse)
async def run_zip_compile(payload: RunZipRequest) -> CompileResponse:
"""两段式之②:原子消费 inspect(rename 防并发双跑)→ tectonic 编译 → pdf/log。
队列/工具失败发生在消费**之前** → inspect 保留可重试;
编译失败(COMPILE_FAILED/TIMEOUT)信封仍带 compile_id + log_url(补充 3)。
"""
jid = (payload.job_id or "").strip()
if not _CID_RE.fullmatch(jid):
return _p4_error(
"VALIDATION", "job_id 格式非法(应为 12 位 hex)")
try:
result = await lc.compile_project_job(jid)
except lc.InspectError as exc:
return _p4_error(exc.code, exc.message)
except lc.ToolUnavailableError as exc:
return _p4_error("TOOL_UNAVAILABLE", str(exc))
except lc.QueueFullError as exc:
# 队列满发生在 consume_inspect 之前 → job 未消费,前端可直接重试
return JSONResponse(
status_code=429,
content={
"ok": False,
"compile_id": None, "pdf_url": None, "log_url": None, "meta": None,
"error": {"code": "QUEUE_FULL", "message": str(exc),
"degraded": False},
},
)
except lc.CompileError as exc:
# 补充 3:失败也带 compile_id + log_url(cid 由 compile_project_job 附加)
cid = getattr(exc, "compile_id", None)
return _p4_error(
exc.code, str(exc), detail=exc.log_tail,
compile_id=cid,
log_url=_project_log_url(cid) if cid else None)
return CompileResponse(
ok=True,
compile_id=result["compile_id"],
pdf_url=_project_pdf_url(result["compile_id"]),
log_url=_project_log_url(result["compile_id"]),
meta=json.loads((lc.compile_job_dir(result["compile_id"]) / "meta.json")
.read_text("utf-8"))
if (lc.compile_job_dir(result["compile_id"]) / "meta.json").is_file()
else {"compile_id": result["compile_id"], "seconds": result["seconds"]},
)
def _cid_dir(cid: str) -> Path | None:
if not _CID_RE.fullmatch(cid or ""):
return None
d = lc.compile_dir(cid)
return d if d.is_dir() else None
def _zip_cid_dir(cid: str) -> Path | None:
"""P4 项目产物目录 data/compile_jobs/{cid}/(独立于 P3a reformat_dir 树)。"""
if not _CID_RE.fullmatch(cid or ""):
return None
d = lc.compile_job_dir(cid)
return d if d.is_dir() else None
@router.get("/compile/zip/{compile_id}", response_model=CompileResponse)
async def get_zip_compile(compile_id: str) -> CompileResponse:
cdir = _zip_cid_dir(compile_id)
meta_path = cdir / "meta.json" if cdir else None
if meta_path is None or not meta_path.is_file():
return _p4_error("NOT_FOUND", "编译任务不存在或已过期,请重新编译")
try:
meta = json.loads(meta_path.read_text("utf-8"))
except (OSError, ValueError):
return _p4_error("NOT_FOUND", "编译产物损坏,请重新编译")
return CompileResponse(
ok=True,
compile_id=compile_id,
pdf_url=_project_pdf_url(compile_id),
log_url=_project_log_url(compile_id),
meta=meta,
)
@router.get("/compile/zip/{compile_id}/pdf", response_model=None)
async def download_zip_pdf(compile_id: str) -> FileResponse | JSONResponse:
cdir = _zip_cid_dir(compile_id)
pdf = cdir / "output.pdf" if cdir else None
if pdf is None or not pdf.is_file():
return JSONResponse(
status_code=200,
content={"ok": False, "error": {
"code": "NOT_FOUND", "message": "PDF 不存在或已过期",
"degraded": False}},
)
return FileResponse(pdf, media_type="application/pdf",
filename=f"{compile_id}.pdf")
@router.get("/compile/zip/{compile_id}/log", response_model=None)
async def download_zip_log(compile_id: str) -> PlainTextResponse | JSONResponse:
"""补充 3:成功/失败都可下载 output.log。"""
cdir = _zip_cid_dir(compile_id)
log = cdir / "output.log" if cdir else None
if log is None or not log.is_file():
return JSONResponse(
status_code=200,
content={"ok": False, "error": {
"code": "NOT_FOUND", "message": "日志不存在或已过期",
"degraded": False}},
)
return PlainTextResponse(log.read_text("utf-8", errors="replace"))
@router.get("/compile/{compile_id}", response_model=CompileResponse)
async def get_compile(compile_id: str) -> CompileResponse:
cdir = _cid_dir(compile_id)
meta_path = cdir / "meta.json" if cdir else None
if meta_path is None or not meta_path.is_file():
return _error("NOT_FOUND", "编译任务不存在或已过期,请重新编译")
try:
meta = json.loads(meta_path.read_text("utf-8"))
except (OSError, ValueError):
return _error("NOT_FOUND", "编译产物损坏,请重新编译")
return CompileResponse(
ok=True,
compile_id=compile_id,
pdf_url=f"/api/review/compile/{compile_id}/pdf",
log_url=f"/api/review/compile/{compile_id}/log",
meta=meta,
)
@router.get("/compile/{compile_id}/pdf", response_model=None)
async def download_compile_pdf(compile_id: str) -> FileResponse | JSONResponse:
cdir = _cid_dir(compile_id)
pdf = cdir / "output.pdf" if cdir else None
if pdf is None or not pdf.is_file():
return JSONResponse(
status_code=200,
content={"ok": False, "error": {
"code": "NOT_FOUND", "message": "PDF 不存在或已过期", "degraded": False}},
)
return FileResponse(pdf, media_type="application/pdf", filename=f"{compile_id}.pdf")
@router.get("/compile/{compile_id}/log", response_model=None)
async def download_compile_log(compile_id: str) -> PlainTextResponse | JSONResponse:
cdir = _cid_dir(compile_id)
log = cdir / "output.log" if cdir else None
if log is None or not log.is_file():
return JSONResponse(
status_code=200,
content={"ok": False, "error": {
"code": "NOT_FOUND", "message": "日志不存在或已过期", "degraded": False}},
)
return PlainTextResponse(log.read_text("utf-8", errors="replace"))
+209
View File
@@ -0,0 +1,209 @@
"""routers/conferences.py —— 会议投稿时间线(只读静态种子)。
- GET /api/review/conferences?domain=&ccf=&upcoming_days=180
data/journals/conferences.json → 过滤/排序 → 列表 + meta。
- 数据诚实性铁律:每条会议必须带 deadline_source ∈ {placeholder, official}、
last_verified 与 deadline_confidence ∈ {high, medium, low},缺失/非法条目加载时
跳过并记日志(宁缺毋造);本端点无任何生成日期的逻辑,日期只来自种子常量。
v2 共 27/27 全 placeholder,meta.data_source_note 显式声明占位与置信度,
由前端渲染「⚠ 日期待核验」及置信度提示。
- 过滤语义:domain 对 domain_tags 精确匹配(大小写不敏感);ccf ∈ {A,B,C};
upcoming_days=0 不过滤,>0 仅保留 deadline_full_paper ∈ [UTC now, now+N 天],
缺失/不可解析截稿日期在过滤开启时排除。排序:deadline_full_paper 升序,
缺失/不可解析恒排最后。
- 只读无写接口;文件不可读 → ok=false + DATA_UNAVAILABLE 信封;参数非法 →
VALIDATION 信封(均 HTTP 200,对齐本服务"错误信封永远 200"约定)。
- 静态文件首次请求加载进内存(lru_cache),进程内 O(1);不触碰 SQLite/BFF。
"""
from __future__ import annotations
import json
import logging
from datetime import datetime, timedelta, timezone
from functools import lru_cache
from pathlib import Path
from typing import Any
from fastapi import APIRouter
from pydantic import BaseModel
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-conferences"])
# review_service/app/routers/ → parents[2] = review_service(对齐 format_check 写法)
_CONFERENCES_PATH = (
Path(__file__).resolve().parents[2] / "data" / "journals" / "conferences.json"
)
_CCF_OK = frozenset({"A", "B", "C"})
_REQUIRED = ("id", "name", "deadline_source", "last_verified", "deadline_confidence")
_CONFIDENCE_OK = frozenset({"high", "medium", "low"})
_SOURCE_NOTE = (
"会议截稿/通知日期首版均为占位数据(deadline_source=placeholder),"
"接入官方渠道前须人工核验;deadline_confidence 标注日期置信度"
"(high=官方已公布/历年极稳定,medium=按历年规律推算,low=待核验);"
"所有时间按 AoE (UTC-12) 时区。"
)
# ---- 请求/响应模型 ----
class ConferenceItem(BaseModel):
"""会议公开字段(种子全字段白名单;未知键丢弃)。"""
id: str
name: str
full_name: str = ""
domain_tags: list[str] = []
application_tags: list[str] = []
ccf_rank: str | None = None
deadline_abstract: str | None = None
deadline_full_paper: str | None = None
notification_date: str | None = None
conference_date: str | None = None
location: str | None = None
official_url: str | None = None
timezone_note: str | None = None
deadline_source: str
last_verified: str
deadline_confidence: str = ""
class ConferencesResponse(BaseModel):
ok: bool
items: list[ConferenceItem] = []
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
def _error(code: str, message: str) -> ConferencesResponse:
return ConferencesResponse(
ok=False, items=[], meta=None,
error={"code": code, "message": message, "degraded": False},
)
# ---- 种子加载(首次调用读盘,进程内缓存)----
@lru_cache(maxsize=1)
def _load_conferences() -> list[dict[str, Any]] | None:
"""读 conferences.json → 合法会议列表;文件不可读 → None(DATA_UNAVAILABLE)。"""
try:
payload = json.loads(_CONFERENCES_PATH.read_text("utf-8"))
except (OSError, ValueError) as exc:
logger.warning("conferences: 种子文件不可读 %s: %s", _CONFERENCES_PATH, exc)
return None
raw = payload.get("conferences") if isinstance(payload, dict) else None
if not isinstance(raw, list):
logger.warning("conferences: 种子缺少 conferences 数组:%s", _CONFERENCES_PATH)
return None
items: list[dict[str, Any]] = []
for conf in raw:
if not isinstance(conf, dict):
continue
missing = [k for k in _REQUIRED if not conf.get(k)]
if missing:
logger.warning("conferences: 条目缺字段 %s(id=%s),跳过", missing, conf.get("id"))
continue
if conf.get("deadline_source") not in ("placeholder", "official"):
logger.warning(
"conferences: deadline_source 非法(id=%s),跳过", conf.get("id")
)
continue
if str(conf.get("deadline_confidence") or "").strip().lower() not in _CONFIDENCE_OK:
logger.warning(
"conferences: deadline_confidence 非法(id=%s),跳过", conf.get("id")
)
continue
items.append(conf)
return items
def _parse_deadline(value: Any) -> datetime | None:
"""截稿时间 → aware datetime(naive 视为 UTC);非法 → None。"""
if not isinstance(value, str) or not value.strip():
return None
try:
dt = datetime.fromisoformat(value.strip())
except ValueError:
return None
return dt.replace(tzinfo=timezone.utc) if dt.tzinfo is None else dt
# ---- GET /api/review/conferences ----
@router.get("/conferences", response_model=ConferencesResponse)
async def conferences_list(
domain: str = "",
ccf: str = "",
upcoming_days: int = 180,
) -> ConferencesResponse:
"""只读:会议投稿时间线(无写/删/改接口)。
- ?domain= 对 domain_tags 精确匹配(大小写不敏感),空=不过滤;
- ?ccf= ccf_rank ∈ A/B/C,空=不过滤,非法值 → VALIDATION 信封;
- ?upcoming_days= int≥0,默认 180;0=不过滤,>0 仅保留截稿在
[UTC now, now+N 天] 内的会议,负数 → VALIDATION 信封;
- 排序按 deadline_full_paper 升序,缺失/不可解析恒排最后;
- 空结果 → items=[] + meta.count=0(HTTP 200,不报错)。
"""
ccf_norm = (ccf or "").strip().upper()
if ccf_norm and ccf_norm not in _CCF_OK:
return _error("VALIDATION", f"不支持的 ccf={ccf!r},可选:A / B / C")
if upcoming_days < 0:
return _error(
"VALIDATION", f"upcoming_days 必须 ≥ 0,收到 {upcoming_days}"
)
all_items = _load_conferences()
if all_items is None:
return _error("DATA_UNAVAILABLE", "会议种子数据不可读,请联系管理员")
items = list(all_items)
dom = (domain or "").strip().casefold()
if dom:
items = [
c for c in items
if any(
str(t).strip().casefold() == dom
for t in (c.get("domain_tags") or [])
)
]
if ccf_norm:
items = [
c for c in items
if str(c.get("ccf_rank") or "").strip().upper() == ccf_norm
]
if upcoming_days > 0:
now = datetime.now(timezone.utc)
limit = now + timedelta(days=upcoming_days)
items = [
c for c in items
if (dl := _parse_deadline(c.get("deadline_full_paper"))) is not None
and now <= dl <= limit
]
def _sort_key(c: dict[str, Any]) -> tuple[bool, datetime]:
dl = _parse_deadline(c.get("deadline_full_paper"))
if dl is None:
return (True, datetime.max.replace(tzinfo=timezone.utc))
return (False, dl)
items.sort(key=_sort_key)
out = [ConferenceItem(**c) for c in items]
verified = max(
(str(c.get("last_verified") or "") for c in all_items), default=""
)
meta: dict[str, Any] = {
"count": len(out),
"last_verified": verified or None,
"data_source_note": _SOURCE_NOTE,
"query": {
"domain": (domain or "").strip(),
"ccf": ccf_norm,
"upcoming_days": upcoming_days,
},
}
return ConferencesResponse(ok=True, items=out, meta=meta, error=None)
@@ -0,0 +1,81 @@
"""routers/format_check.py —— 版式与规范检查(POST /api/review/format-check)。
纯规则引擎(services/format_check.py):PDF/Word 本地解档取版式元数据 +
journals.json 规则集(format_rules_default + 每条 format_rules 带 rules_source)。
无 LLM / 无 vstore,同步执行通常 <1s;恒 200 + {ok, checks, summary, meta|error} 信封。
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, File, Form, UploadFile
from pydantic import BaseModel
from ..config import get_settings
from ..services import format_check
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-format"])
class CheckItem(BaseModel):
key: str
name: str
status: str # pass | warn | fail
detail: str
suggestion: str = ""
class FormatCheckResponse(BaseModel):
ok: bool
checks: list[CheckItem] | None = None
summary: dict[str, int] | None = None
meta: dict[str, object] | None = None
error: dict[str, object] | None = None
def _error(code: str, message: str, *, degraded: bool = False) -> FormatCheckResponse:
return FormatCheckResponse(
ok=False, error={"code": code, "message": message, "degraded": degraded},
)
@router.post("/format-check", response_model=FormatCheckResponse)
async def format_check_endpoint(
file: UploadFile = File(..., description="待检查的 PDF / Word(.docx) / LaTeX(.tex) 文档"),
target_journal: str = Form("", description="目标期刊(可选,传 journals 的 id)"),
) -> FormatCheckResponse:
settings = get_settings()
filename = file.filename or ""
low = filename.lower()
if low.endswith(".pdf"):
kind = "pdf"
elif low.endswith(".docx"):
kind = "docx"
elif low.endswith(".tex"):
kind = "tex"
else:
return _error("VALIDATION", "仅支持 PDF / Word(.docx)/ LaTeX(.tex)文件")
content = await file.read()
if not content:
return _error("VALIDATION", "上传文件为空")
if len(content) > settings.max_pdf_bytes:
return _error("VALIDATION", "上传文件超过 50MB 上限")
try:
result = format_check.check_document(content, kind, (target_journal or "").strip())
except format_check.FormatCheckError as exc:
logger.warning("format-check: 解档失败:%s", exc)
return _error("PARSE_FAILED", str(exc))
meta = dict(result.get("meta") or {})
meta["filename"] = filename
return FormatCheckResponse(
ok=True,
checks=[CheckItem(**c) for c in result["checks"]],
summary=result["summary"],
meta=meta,
)
@@ -0,0 +1,280 @@
"""routers/journal_directory.py —— 期刊目录浏览(1056 本,只读分页)。
- GET /api/review/journals/directory:journal_directory 表(scripts/import_journals.py
导入的 v4 目录:2025 中科院分区 + 2025 JIF(2026 年 JCR 发布)+ 年报 2025 版
(中文统计年 2024)+ 官网网址/核对状态)→ 过滤/排序/分页 → 列表 + meta。
- 与 GET /journals(journals.json 精选 5 本,供 FormatPanel 下拉/recommend 语义)物理
分离(方案 A);本端点绝不读写 journals 表,recommend/journal_kb/journal_rank 零触碰。
- 读路径只 SELECT;表由导入脚本拥有(db.py 零改动);表不存在 → data_imported=false
优雅降级(HTTP 200 空列表)。
读时归一化(拍板细节 1-3,均在 API 层做,不动脚本/DB 原值):
细节1 lang=zh → cas_partition 恒 null + cas_partition_note="不按中科院分区口径列示";
细节2 jif_note 源文本映射(v4 值域已是用户可读文本,旧值映射保留兜底);
细节3 缺值即矛盾清零:en 且 jif=null → jif_source_url=null。
(zh 的 IF 载体是 composite/comprehensive,其 jif_source_url 本恒 null。)
决策2:响应不下发"来源"文本字段(partition_source/jif_source/data_source_label
已移除);jif_source_url/data_source_url 保留供展开行链接。
决策4(来源展示优化):partition_source_url 不再下发——DB 原值分毫未动
(仍 SELECT * 读出后被模型丢弃),响应改为常量字段 partition_source_note
文案,前端以灰色说明文字替代原"分区来源"超链接。
新增 official_check_status / jcr_release_year。
"""
from __future__ import annotations
import json
import logging
import sqlite3
from typing import Any
from fastapi import APIRouter
from pydantic import BaseModel
from ..db import db
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-directory"])
_LANGS = frozenset({"en", "zh"})
# 排序列白名单(列名不可参数化 → 白名单防注入)
_FIELDS = frozenset({
"name",
"jif",
"composite_impact_factor",
"comprehensive_impact_factor",
"research_direction",
"cas_partition",
})
_ORDERS = frozenset({"asc", "desc"})
_PARTS = {"1": "1区", "2": "2区", "3": "3区", "4": "4区"}
_MAX_SIZE = 200
# 细节2:源核对状态 → 用户可读 note
_JIF_NOTE_MAP = {
"源表标N/A": "未被2024 JCR收录",
"所用年度来源未检得": "本次汇编未查到,建议查阅最新 JCR",
}
_ZH_PART_NOTE = "不按中科院分区口径列示"
# 决策4:分区来源改为常量说明文案(不下发 partition_source_url;DB 原值不动)
_PARTITION_SOURCE_NOTE = (
"2025 中科院官方表(2026 年起停更;官方入口 fenqubiao.com 需机构登录)"
)
_NOTE_EN = (
"分区:2025年中科院公开表(多表比对,2026年起停止更新);"
"JIF:2025年度指标(2026年JCR发布,两份公开汇编一致或已核对出版社官方指标);"
"整理日期2026-10-10。空白=来源未给值≠0;研究方向/期刊标记/文章类型为导航标签,非官方评级。"
)
_NOTE_ZH = (
"指标:国内自然科学期刊年报2025版(统计年2024,official,含复合/综合IF两口径);"
"中文期刊不按中科院分区口径列示;国内IF与JIF统计源不同,不可混排。"
)
_UNIMPORTED_NOTE = "目录数据未导入(请先运行 scripts/import_journals.py)"
# ---- 响应模型 ----
class DirectoryItem(BaseModel):
"""目录公开字段(DB 全列白名单;imported_at 等内部列丢弃)。"""
id: str
lang: str
name: str
research_direction: str | None = None
cas_partition: str | None = None
cas_partition_note: str | None = None
cas_partition_year: int | None = None
jif: float | None = None
jif_year: int | None = None
jif_note: str | None = None
jif_check_status: str | None = None
composite_impact_factor: float | None = None
comprehensive_impact_factor: float | None = None
metric_year: int | None = None
metric_report_year: int | None = None
metric_caliber: str | None = None
annual_report_page: int | None = None
top_flag: str | None = None
publisher_series: str | None = None
journal_mark: str | None = None
article_type: str | None = None
issn: str | None = None
eissn: str | None = None
jcr_disciplines: list[str] = []
supplement_note: str | None = None
evidence_note: str | None = None
official_url: str | None = None
official_check_status: str | None = None
jcr_release_year: int | None = None
jif_source_url: str | None = None
partition_source_note: str = _PARTITION_SOURCE_NOTE # 决策4:常量,替代分区来源链接
data_source_url: str | None = None
class DirectoryResponse(BaseModel):
ok: bool
items: list[DirectoryItem] = []
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
def _error(code: str, message: str) -> DirectoryResponse:
return DirectoryResponse(
ok=False, items=[], meta=None,
error={"code": code, "message": message, "degraded": False},
)
# ---- 读时归一化(拍板细节 1-3)----
def _present(raw: dict[str, Any], lang: str) -> dict[str, Any]:
d = dict(raw)
# jcr_disciplines:JSON 数组字符串 → list
jcr = d.get("jcr_disciplines")
if jcr:
try:
parsed = json.loads(jcr)
d["jcr_disciplines"] = parsed if isinstance(parsed, list) else []
except (TypeError, ValueError):
d["jcr_disciplines"] = []
else:
d["jcr_disciplines"] = []
# 细节2:jif_note 源文本 → 用户可读
note = d.get("jif_note")
if note in _JIF_NOTE_MAP:
d["jif_note"] = _JIF_NOTE_MAP[note]
# 细节1:zh 分区归一(DB 原值"不按此口径列示"只留在库里)
if lang == "zh":
d["cas_partition"] = None
d["cas_partition_note"] = _ZH_PART_NOTE
# 细节3:缺值 → 来源链接清零(消除 null 值却挂来源链接的矛盾;
# 分区来源链接已按决策4 移出响应,改为常量 partition_source_note)
if lang == "en":
if d.get("jif") is None:
d["jif_source_url"] = None
return d
def _escape_like(s: str) -> str:
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
# ---- GET /api/review/journals/directory ----
@router.get("/journals/directory", response_model=DirectoryResponse)
async def journals_directory(
lang: str = "en",
domain: str = "",
cas_partition: str = "",
q: str = "",
page: int = 1,
size: int = 50,
field: str = "",
order: str = "",
) -> DirectoryResponse:
"""只读:期刊目录分页浏览(无写接口)。
- ?lang=en|zh(默认 en;**强制单语,保证国际/中文永不混排**);
- ?domain= 研究方向精确匹配;?cas_partition=1|2|3|4(仅 lang=en,zh 忽略);
- ?q= 期刊名模糊(ASCII 大小写不敏感);?page≥1 ?size 1-200(默认 50);
- ?field= 排序白名单(name/jif/composite_impact_factor/
comprehensive_impact_factor/research_direction/cas_partition),?order=asc|desc;
未传 field → 默认排序 en=jif 降序 / zh=复合IF 降序(NULL 恒排最后);
- 参数非法 → VALIDATION 信封;表不存在 → data_imported=false 空列表(HTTP 200)。
"""
lang_n = (lang or "").strip().lower()
if lang_n not in _LANGS:
return _error("VALIDATION", f"不支持的 lang={lang!r},可选:en / zh")
if page < 1:
return _error("VALIDATION", f"page 必须 ≥ 1,收到 {page}")
if size < 1 or size > _MAX_SIZE:
return _error("VALIDATION", f"size 必须在 1-{_MAX_SIZE},收到 {size}")
field_n = (field or "").strip()
if field_n and field_n not in _FIELDS:
return _error(
"VALIDATION",
f"不支持的 field={field!r},可选:{', '.join(sorted(_FIELDS))}",
)
order_n = (order or "").strip().lower()
if order_n and order_n not in _ORDERS:
return _error("VALIDATION", f"不支持的 order={order!r},可选:asc / desc")
part_q = (cas_partition or "").strip()
if part_q:
part_q = _PARTS.get(part_q, _PARTS.get(part_q.rstrip("区").strip(), ""))
if not part_q:
return _error(
"VALIDATION",
f"不支持的 cas_partition={cas_partition!r},可选:1 / 2 / 3 / 4",
)
# 过滤条件(白名单拼接;参数全部占位符)
where = ["lang = ?"]
params: list[Any] = [lang_n]
domain_n = (domain or "").strip()
if domain_n:
where.append("research_direction = ?")
params.append(domain_n)
if part_q and lang_n == "en": # zh 不分区:忽略(meta 回显)
where.append("cas_partition = ?")
params.append(part_q)
q_n = (q or "").strip()
if q_n:
where.append("name LIKE ? ESCAPE '\\'")
params.append(f"%{_escape_like(q_n)}%")
where_sql = " AND ".join(where)
# 排序(列名出自白名单;NULL 恒排最后)
if field_n:
dir_sql = "DESC" if order_n == "desc" else "ASC"
order_sql = f"{field_n} IS NULL, {field_n} {dir_sql}"
else:
col = "jif" if lang_n == "en" else "composite_impact_factor"
order_sql = f"{col} IS NULL, {col} DESC"
try:
total_row = await db.fetchone(
f"SELECT COUNT(*) AS c FROM journal_directory WHERE {where_sql}", params # noqa: S608
)
total = int(total_row["c"]) if total_row else 0
rows = await db.fetchall(
f"SELECT * FROM journal_directory WHERE {where_sql} " # noqa: S608
f"ORDER BY {order_sql} LIMIT ? OFFSET ?",
[*params, size, (page - 1) * size],
)
dom_rows = await db.fetchall(
"SELECT research_direction AS d, COUNT(*) AS c FROM journal_directory "
"WHERE lang = ? AND research_direction IS NOT NULL "
"GROUP BY 1 ORDER BY c DESC, d ASC",
[lang_n],
)
except sqlite3.OperationalError as exc:
# 表不存在(脚本未跑)→ 优雅降级,HTTP 200
logger.warning("journal_directory: 表不可用(目录未导入?): %s", exc)
return DirectoryResponse(
ok=True, items=[],
meta={
"total": 0, "page": page, "size": size, "lang": lang_n,
"domain": domain_n, "cas_partition": part_q, "q": q_n,
"domains": [], "data_source_note": _UNIMPORTED_NOTE,
"data_imported": False,
},
error=None,
)
items = [DirectoryItem(**_present(dict(r), lang_n)) for r in rows]
meta: dict[str, Any] = {
"total": total,
"page": page,
"size": size,
"lang": lang_n,
"domain": domain_n,
"cas_partition": part_q if lang_n == "en" else "",
"q": q_n,
"domains": [r["d"] for r in dom_rows if r["d"]],
"data_source_note": _NOTE_EN if lang_n == "en" else _NOTE_ZH,
"data_imported": True,
}
return DirectoryResponse(ok=True, items=items, meta=meta, error=None)
+354
View File
@@ -0,0 +1,354 @@
"""routers/journals.py —— 期刊情报与智能投稿推荐。
- GET /api/review/journals 只读:journals 表全量列表(?field=/&order= 排序),
meta 含 count / last_updated / data_source_note。无任何写接口;空表返回
空数组 + meta.count=0(HTTP 200,绝不 500)。
- POST /api/review/recommend 智能投稿推荐:vstore 向量匹配(摘要 vs
scope_desc,白名单检索)→ LLM 按 id 白名单重排(幻觉期刊丢弃、缺漏规则
补位)→ 三档输出:挑战型 / 适配型 / 稳妥型,附匹配理由与风险提示。
降级哲学(同 presubmit):向量失败 → 全量候选 + vector_ok=false;
LLM 失败 → 规则分档 + llm_ok=false;推荐永不 5xx。仅入参非法返回
VALIDATION 错误信封(HTTP 200,与本服务其余端点一致)。
"""
from __future__ import annotations
import json
import logging
from functools import lru_cache
from pathlib import Path
from typing import Any, Literal
from fastapi import APIRouter
from pydantic import BaseModel, Field
from ..config import get_settings
from ..db import SeedError, ensure_journals_seeded, list_journals
from ..services import journal_kb, journal_rank, llm
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-journals"])
# ---- 请求/响应模型 ----
class Preferences(BaseModel):
max_review_weeks: float | None = Field(default=None, ge=1, le=104)
min_impact_factor: float | None = Field(default=None, ge=0, le=200)
tier_focus: Literal["all", "challenge", "fit", "safe"] = "all"
class RecommendRequest(BaseModel):
abstract: str
preferences: Preferences = Preferences()
class JournalItem(BaseModel):
id: str
name: str
impact_factor: float | None = None
review_time_weeks: float | None = None
acceptance_rate: float | None = None
official_url: str | None = None
similarity: float | None = None
tier: str
match_reason: str
risk_note: str
class TierBox(BaseModel):
label: str
items: list[JournalItem]
class RecommendResponse(BaseModel):
ok: bool
tiers: dict[str, TierBox] | None = None
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
# ---- GET /api/review/journals(只读列表)----
class JournalBrief(BaseModel):
"""期刊公开字段(journals 表全列;不含 vstore_doc_id 内部映射)。
标签/分区 6 字段不进 SQLite:GET 时按 id 从 journals.json 内存缓存合并;
JSON 缺该 id → 空数组/null 默认值。
"""
id: str
name: str
scope_desc: str = ""
impact_factor: float | None = None
review_time_weeks: float | None = None
acceptance_rate: float | None = None
submission_deadline: str | None = None
official_url: str | None = None
last_updated: str = ""
scope_tags: list[str] = []
application_tags: list[str] = []
cas_partition: str | None = None
jcr_quartile: str | None = None
partition_source: str | None = None
cross_domain: list[str] = []
class JournalsListResponse(BaseModel):
ok: bool
items: list[JournalBrief] = []
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
# ?field= 排序白名单(列名无法参数化,白名单防注入;非法值 → VALIDATION 信封)
_SORT_FIELDS = frozenset({
"id",
"name",
"impact_factor",
"review_time_weeks",
"acceptance_rate",
"last_updated",
})
_DATA_SOURCE_NOTE = (
"IF/审稿周期为公开近似值,acceptance_rate 为 null 表示暂无公开数据,需人工核验。"
"CAS/JCR 分区为首版占位数据,需核验。"
)
# journals.json 新增字段(标签/分区;刻意不进 SQLite,见 C-2 方案)
_TAG_FIELDS = (
"scope_tags",
"application_tags",
"cas_partition",
"jcr_quartile",
"partition_source",
"cross_domain",
)
@lru_cache(maxsize=1)
def _load_tag_extras() -> dict[str, dict[str, Any]]:
"""journals.json 标签/分区字段 → {id: {6 新字段}}。
首次请求读盘后进程内缓存,后续 GET O(1) 按 id 合并(不做每请求磁盘 IO)。
JSON 缺该 id → 合并时用空数组/null 默认;文件不可读 → 空 dict 降级(不 500)。
"""
seed = get_settings().journals_seed_abs
try:
payload = json.loads(Path(seed).read_text("utf-8"))
except (OSError, ValueError) as exc:
logger.warning("journals: 标签扩展文件不可读,合并降级为默认值:%s", exc)
return {}
raw = payload.get("journals") if isinstance(payload, dict) else None
if not isinstance(raw, list):
return {}
out: dict[str, dict[str, Any]] = {}
for j in raw:
if isinstance(j, dict) and j.get("id"):
out[str(j["id"])] = {k: j.get(k) for k in _TAG_FIELDS}
return out
def _tag_fields(extra: dict[str, Any] | None) -> dict[str, Any]:
"""缓存条目 → JournalBrief 关键字参数(JSON 缺该 id → 空数组/null 默认)。"""
ex = extra or {}
return {
"scope_tags": list(ex.get("scope_tags") or []),
"application_tags": list(ex.get("application_tags") or []),
"cas_partition": ex.get("cas_partition"),
"jcr_quartile": ex.get("jcr_quartile"),
"partition_source": ex.get("partition_source"),
"cross_domain": list(ex.get("cross_domain") or []),
}
def _sort_rows(rows: list[dict[str, Any]], field: str, order: str) -> list[dict[str, Any]]:
"""按白名单字段排序;NULL 恒排最后;次序不稳定时按 db 默认序(稳定排序保留)。"""
present = [r for r in rows if r.get(field) is not None]
missing = [r for r in rows if r.get(field) is None]
present.sort(key=lambda r: r.get(field), reverse=(order == "desc"))
return present + missing
def _list_error(message: str) -> JournalsListResponse:
return JournalsListResponse(
ok=False, items=[], meta=None,
error={"code": "VALIDATION", "message": message, "degraded": False},
)
@router.get("/journals", response_model=JournalsListResponse)
async def journals_list(
field: str = "name",
order: str = "asc",
) -> JournalsListResponse:
"""只读:journals 表全量列表(无写/删/改接口)。
- ?field= 排序字段白名单:id/name/impact_factor/review_time_weeks/
acceptance_rate/last_updated,默认 name;
- ?order=asc|desc,默认 asc;
- 空表 → items=[] + meta.count=0(HTTP 200,不报错)。种子装载由
POST /recommend 首次调用幂等完成,本端点严格只读。
"""
col = (field or "").strip()
if col not in _SORT_FIELDS:
return _list_error(
f"不支持的排序字段 field={field!r},可选:{', '.join(sorted(_SORT_FIELDS))}"
)
ord_norm = (order or "asc").strip().lower()
if ord_norm not in ("asc", "desc"):
return _list_error(f"不支持的排序方向 order={order!r},可选:asc, desc")
rows = await list_journals()
rows = _sort_rows(rows, col, ord_norm)
tag_extras = _load_tag_extras()
items = [
JournalBrief(
id=str(r.get("id") or ""),
name=str(r.get("name") or ""),
scope_desc=str(r.get("scope_desc") or ""),
impact_factor=r.get("impact_factor"),
review_time_weeks=r.get("review_time_weeks"),
acceptance_rate=r.get("acceptance_rate"),
submission_deadline=r.get("submission_deadline"),
official_url=r.get("official_url"),
last_updated=str(r.get("last_updated") or ""),
**_tag_fields(tag_extras.get(str(r.get("id") or ""))),
)
for r in rows
]
latest = max((str(r.get("last_updated") or "") for r in rows), default="")
meta: dict[str, Any] = {
"count": len(items),
"last_updated": latest or None,
"data_source_note": _DATA_SOURCE_NOTE,
}
return JournalsListResponse(ok=True, items=items, meta=meta, error=None)
def _error(code: str, message: str, *, degraded: bool = False) -> RecommendResponse:
return RecommendResponse(
ok=False, tiers=None, meta=None,
error={"code": code, "message": message, "degraded": degraded},
)
def _item_sort_key(it: JournalItem) -> tuple[int, float, float]:
return (
1 if it.similarity is None else 0,
-(it.similarity or 0.0),
-(it.impact_factor or 0.0),
)
def _apply_soft_preferences(item: JournalItem, prefs: Preferences) -> None:
"""软偏好:不满足只加风险提示,不删条目。"""
weeks = item.review_time_weeks
if prefs.max_review_weeks is not None and weeks is not None and weeks > prefs.max_review_weeks:
prefix = f"⚠ 审稿约 {weeks:g} 周,超出你 {prefs.max_review_weeks:g} 周的周期偏好;"
item.risk_note = prefix + ("无" if item.risk_note == "无" else item.risk_note)
inf = item.impact_factor
if prefs.min_impact_factor is not None:
if inf is None:
item.risk_note = "⚠ 影响因子未收录,无法确认满足你的要求;" + item.risk_note
elif inf < prefs.min_impact_factor:
prefix = f"⚠ 影响因子 {inf:g} 低于你 {prefs.min_impact_factor:g} 的要求;"
item.risk_note = prefix + ("无" if item.risk_note == "无" else item.risk_note)
@router.post("/recommend", response_model=RecommendResponse)
async def recommend(req: RecommendRequest) -> RecommendResponse:
settings = get_settings()
# ---- 入参校验(手动,错误信封与本服务其余端点一致)----
abstract = (req.abstract or "").strip()
if len(abstract) < 50:
return _error("VALIDATION", "abstract 至少需要 50 个字符(请提供摘要或正文片段)")
if len(abstract) > 20000:
return _error("VALIDATION", "abstract 超过 20000 字符上限")
prefs = req.preferences
# ---- 种子 + 期刊元数据 ----
try:
await ensure_journals_seeded()
journals = await list_journals()
except SeedError as exc:
logger.error("recommend: 种子装载失败:%s", exc)
return _error("SEED_UNAVAILABLE", str(exc), degraded=True)
if not journals:
return _error("SEED_UNAVAILABLE", "期刊库为空", degraded=True)
# ---- ① vstore 向量匹配(摘要 vs scope_desc)----
try:
candidates, vector_ok = await journal_kb.match(abstract, journals)
except journal_kb.MatchError as exc:
logger.warning("recommend: 向量匹配降级:%s", exc)
candidates, vector_ok = [], False
if not candidates:
# 无有效语义分 → 全量候选(similarity=None),LLM 仍可按 scope 文本重排
candidates = [{"journal": j, "similarity": None} for j in journals]
vector_ok = False
# ---- ② LLM 重排(id 白名单硬校验)----
prefs_dict = prefs.model_dump(exclude_none=True)
llm_ok = True
dropped = 0
repaired = 0
try:
rankings, dropped, repaired = await journal_rank.rerank(abstract, candidates, prefs_dict)
except (llm.LLMError, journal_rank.RankError, ValueError) as exc:
# ValueError:complete_json 最终 json.loads 修复仍失败(同 presubmit 宽容口径)
logger.warning("recommend: LLM 重排降级为规则分档:%s", exc)
rankings = journal_rank.fallback_rankings(candidates)
llm_ok = False
# ---- ③ 三档组装(名称/指标全部来自 DB,不采信 LLM 文本)----
by_id = {str(c["journal"]["id"]): c for c in candidates}
tiers: dict[str, list[JournalItem]] = {t: [] for t in journal_rank.TIERS}
for r in rankings:
cand = by_id.get(r.journal_id)
if cand is None:
continue
j = cand["journal"]
sim = cand.get("similarity")
item = JournalItem(
id=str(j["id"]),
name=str(j.get("name") or j["id"]),
impact_factor=j.get("impact_factor"),
review_time_weeks=j.get("review_time_weeks"),
acceptance_rate=j.get("acceptance_rate"),
official_url=j.get("official_url"),
similarity=round(float(sim), 4) if sim is not None else None,
tier=r.tier,
match_reason=r.match_reason,
risk_note=r.risk_note,
)
_apply_soft_preferences(item, prefs)
tiers.setdefault(r.tier, []).append(item)
# 档内排序:相似度(None 靠后)→ 影响因子
for box in tiers.values():
box.sort(key=_item_sort_key)
# tier_focus:软展示聚焦(默认 all;聚焦时其余档 items 置空,label 保留)
out_tiers: dict[str, TierBox] = {}
for t in journal_rank.TIERS:
items = tiers.get(t, [])
if prefs.tier_focus != "all" and t != prefs.tier_focus:
items = []
out_tiers[t] = TierBox(label=journal_rank.TIER_LABELS[t], items=items)
meta: dict[str, Any] = {
"model": llm._llm_config()["model"],
"vector_ok": vector_ok,
"llm_ok": llm_ok,
"seed_count": len(journals),
"candidate_count": len(candidates),
"dropped_hallucinated": dropped,
"repaired": repaired,
"preferences": prefs_dict,
"abstract_chars": len(abstract),
}
return RecommendResponse(ok=True, tiers=out_tiers, meta=meta, error=None)
+317
View File
@@ -0,0 +1,317 @@
"""routers/ppt.py —— P5 科研汇报助手(9 端点,SQLite job 模式)。
信封与其余 review 端点一致:业务错误 = HTTP 200 + {ok:false,error:{code,message,degraded}}。
- GET /styles:风格清单(data/ppt_styles/*.json 实际数量 → 副标题/风格卡动态渲染)
- 创建(text 主入口 / pdf 次要入口)= 解析 + map-reduce 大纲生成(同步;
map 并发 2 + 单调用超时收紧,总时长压在 BFF 读超时 360s 内)
- PATCH /jobs/{id}:结构清洗(复用 coerce_outline)+ evidence 状态按 raw_text
重验(编辑后 quote 真伪即时刷新);warnings 保留生成期记录(页数/拆页等
结构提示不因编辑丢失)
- POST /jobs/{id}/build:python-pptx 本地生成 → data/ppt_jobs/{id}/draft.pptx
+ 逐页缩略图渲染 → preview/slide_{n}.png(失败仅 warning,不阻断下载)
- GET /jobs/{id}/pptx:FileResponse 附件下载
- GET /jobs/{id}/preview/{page}:第 page 页 PNG 缩略图(旧 job 缺图懒渲染)
- DELETE 手动清行 + 产物目录(无 TTL,对齐 rebuttal 决策 5)
"""
from __future__ import annotations
import logging
import shutil
from pathlib import Path
from typing import Any
from fastapi import APIRouter, File, Form, UploadFile
from fastapi.responses import FileResponse, JSONResponse
from pydantic import BaseModel, Field
from ..config import get_settings
from ..services import paper_parse, ppt_build, ppt_outline, ppt_preview, ppt_store
from ..services.ppt_build import BuildError
from ..services.ppt_figure_index import extract_figures, scan_figure_index
from ..services.ppt_outline import (
OutlineError,
build_index,
coerce_outline,
renumber,
repair_evidence_para_ids,
validate_evidence,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review/ppt", tags=["review-ppt"])
_MAX_RAW_CHARS = 500_000 # 粘贴文本上限(论文正文通常 ≤150K,留编辑余量)
# ---------- 信封 ----------
class PptResponse(BaseModel):
ok: bool
job: dict[str, Any] | None = None
error: dict[str, Any] | None = None
meta: dict[str, Any] | None = None
def _error(code: str, message: str) -> PptResponse:
return PptResponse(ok=False, error={"code": code, "message": message, "degraded": False})
def _ok(job: dict[str, Any], **meta: Any) -> PptResponse:
return PptResponse(ok=True, job=job, meta=meta or None)
class StylesResponse(BaseModel):
"""GET /styles 信封(清单在 items,不用 job 字段)。"""
ok: bool
items: list[dict[str, Any]] = Field(default_factory=list)
meta: dict[str, Any] | None = None
# ---------- 请求体 ----------
class JobCreateBody(BaseModel):
text: str = Field(min_length=1)
scenario: str = "group_meeting"
minutes: int = 15
style_id: str = "academic-minimal"
class PagesBody(BaseModel):
pages: list[dict[str, Any]] = Field(min_length=1)
# ---------- 内部工具 ----------
def _pptx_path(job_id: str) -> Path:
return get_settings().ppt_jobs_abs / job_id / "draft.pptx"
def _preview_dir(job_id: str) -> Path:
return get_settings().ppt_jobs_abs / job_id / "preview"
def _preview_path(job_id: str, page: int) -> Path:
return _preview_dir(job_id) / f"slide_{page:02d}.png"
async def _create_job(
text: str, kind: str, filename: str | None,
scenario: str, minutes: int, style_id: str,
*, job_id: str | None = None, fig_meta: dict[str, Any] | None = None,
) -> PptResponse:
if len(text) > _MAX_RAW_CHARS:
return _error("VALIDATION", f"正文超过 {_MAX_RAW_CHARS} 字符上限")
# v3 修复2:PDF 提取出的原图 label → 大纲生成只从这些图里选(build 必能嵌入)
available_figures = None
if fig_meta:
fig_meta = dict(fig_meta)
available_figures = fig_meta.pop("figure_labels", None) or None
try:
pages, warnings, meta = await ppt_outline.generate_outline(
text, scenario, minutes, style_id,
available_figures=available_figures,
)
except OutlineError as exc:
return _error(exc.code, exc.message)
job = {
"id": job_id or ppt_store.new_id("ppt_"),
"status": "outline_ready",
"source": {"kind": kind, "filename": filename},
"scenario": scenario,
"minutes": minutes,
"style_id": style_id,
"pages": pages,
"warnings": warnings,
"pptx_path": None,
}
ppt_store.insert_job(job, raw_text=text)
if fig_meta:
meta = {**meta, **fig_meta}
return _ok(ppt_store.get_job(job["id"]) or job, **meta)
def _extract_job_figures(pdf_bytes: bytes, job_id: str) -> dict[str, Any]:
"""2.3:PDF 原图 → data/ppt_jobs/{id}/figs/(失败仅告警,不阻塞建 job)。"""
out_dir = get_settings().ppt_jobs_abs / job_id / "figs"
try:
figs = extract_figures(pdf_bytes, out_dir)
except Exception as exc: # noqa: BLE001 - 提取属增强项,绝不 500
logger.warning("图表原图提取失败(该 job 回落文字占位):%s", exc)
return {"figures_extracted": 0, "figures_extract_error": str(exc)[:200]}
# v3 修复2:已提取 label 供大纲只选「磁盘上有图」的条目
return {
"figures_extracted": len(figs),
"figure_labels": [str(r.get("label") or "") for r in figs if r.get("label")],
}
def _not_found(job_id: str) -> PptResponse:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
# ---------- 端点 ----------
@router.get("/styles", response_model=StylesResponse)
async def get_styles() -> StylesResponse:
"""风格清单(data/ppt_styles/*.json 实际数量 + 文案;副标题/风格卡按它渲染)。"""
items = ppt_build.list_styles()
return StylesResponse(ok=True, items=items, meta={"count": len(items)})
@router.post("/jobs", response_model=PptResponse)
async def create_job(body: JobCreateBody) -> PptResponse:
"""文本粘贴主入口(论文纯文本 → 大纲)。"""
return await _create_job(
body.text, kind="text", filename=None,
scenario=body.scenario, minutes=body.minutes, style_id=body.style_id,
)
@router.post("/jobs/pdf", response_model=PptResponse)
async def create_job_pdf(
file: UploadFile = File(...),
scenario: str = Form("group_meeting"),
minutes: int = Form(15),
style_id: str = Form("academic-minimal"),
) -> PptResponse:
"""PDF 次要入口(经 vstore 8094 解析 → 大纲)。"""
settings = get_settings()
filename = file.filename or "paper.pdf"
if not filename.lower().endswith(".pdf"):
return _error("VALIDATION", "仅支持 PDF 文件")
data = await file.read()
if not data:
return _error("VALIDATION", "上传文件为空")
if len(data) > settings.max_pdf_bytes:
return _error("VALIDATION", f"PDF 超过 {settings.max_pdf_bytes // (1024 * 1024)}MB 上限")
try:
text = await paper_parse.parse_pdf_bytes(data, filename=filename)
except Exception as exc: # noqa: BLE001 - ParseError/网络异常统一信封
return _error("UPSTREAM_DOWN", f"PDF 解析失败:{exc}")
# 2.3:先取 job_id 并提取原图(build 侧按 out 同级 figs/ 自动发现真图)
job_id = ppt_store.new_id("ppt_")
fig_meta = _extract_job_figures(data, job_id)
return await _create_job(
text, kind="pdf", filename=filename,
scenario=scenario, minutes=minutes, style_id=style_id,
job_id=job_id, fig_meta=fig_meta,
)
@router.get("/jobs/{job_id}", response_model=PptResponse)
async def get_job(job_id: str) -> PptResponse:
job = ppt_store.get_job(job_id)
if job is None:
return _not_found(job_id)
return _ok(job)
@router.patch("/jobs/{job_id}", response_model=PptResponse)
async def patch_pages(job_id: str, body: PagesBody) -> PptResponse:
"""保存前端编辑后的完整 pages(结构清洗 + evidence 状态重验)。"""
job = ppt_store.get_job(job_id)
if job is None:
return _not_found(job_id)
try:
pages = coerce_outline({"pages": body.pages})
except OutlineError as exc:
return _error("VALIDATION", exc.message)
renumber(pages)
raw_text = ppt_store.get_raw_text(job_id)
if raw_text:
_, index = build_index(raw_text)
repair_evidence_para_ids(pages, index) # 编辑保存时同样反查修正(兼容旧 job)
validate_evidence(pages, index, scan_figure_index(index))
updated = ppt_store.update_job(job_id, pages=pages, pptx_path=None) # 编辑后旧 pptx 失效
return _ok(updated or job)
@router.post("/jobs/{job_id}/build", response_model=PptResponse)
async def build_job(job_id: str) -> PptResponse:
"""大纲 → draft.pptx(python-pptx 本地生成,覆盖重建)。"""
job = ppt_store.get_job(job_id)
if job is None:
return _not_found(job_id)
if not job.get("pages"):
return _error("VALIDATION", "大纲为空,无法生成 PPT")
ppt_store.update_job(job_id, status="building")
try:
stats = ppt_build.build_pptx(
job["pages"], job.get("style_id") or "academic-minimal",
_pptx_path(job_id),
context={"scenario": job.get("scenario"), "minutes": job.get("minutes")},
)
except BuildError as exc:
warnings = list(job.get("warnings") or []) + [f"PPT 生成失败:{exc.message}"]
ppt_store.update_job(job_id, status="error", warnings=warnings)
fresh = ppt_store.get_job(job_id)
return PptResponse(
ok=False, job=fresh,
error={"code": exc.code, "message": exc.message, "degraded": False},
)
# 浏览器内预览:build 成功即渲染缩略图;失败仅告警,不阻断下载
previews = ppt_preview.render_previews(_pptx_path(job_id), _preview_dir(job_id))
stats["previews"] = len(previews)
updated = ppt_store.update_job(job_id, status="done", pptx_path=str(_pptx_path(job_id)))
if not previews:
fresh_warnings = list((updated or job).get("warnings") or []) + ["预览图生成失败(可正常下载 PPTX)"]
updated = ppt_store.update_job(job_id, warnings=fresh_warnings) or updated
return _ok(updated or job, **stats)
@router.get("/jobs/{job_id}/pptx", response_model=None)
async def download_pptx(job_id: str) -> FileResponse | JSONResponse:
"""下载 draft.pptx(二进制流;缺失时回同款信封 JSON,对齐 compile 下载端点)。"""
job = ppt_store.get_job(job_id)
# pptx_ready 同时看 DB 列与磁盘文件:编辑后列被清 → 盘上旧文件不再下发
if job is None or not job.get("pptx_ready"):
resp = _not_found(job_id)
return JSONResponse(status_code=200, content=resp.model_dump())
path = _pptx_path(job_id)
filename = path.name
src_name = (job.get("source") or {}).get("filename")
if src_name:
filename = Path(src_name).stem + ".pptx"
return FileResponse(
str(path),
media_type="application/vnd.openxmlformats-officedocument.presentationml.presentation",
filename=filename,
)
@router.get("/jobs/{job_id}/preview/{page}", response_model=None)
async def preview_slide(job_id: str, page: int) -> FileResponse | JSONResponse:
"""第 page 页缩略图 PNG(1-based;对齐 download_pptx 的信封约定)。
缺 preview 但 pptx 在盘(旧 job,端点上线前构建的)→ 懒渲染一次再回图。
job 不存在 / pptx 未就绪 / 页码越界 → HTTP 200 + 信封 JSON(对齐 pptx 下载)。
"""
job = ppt_store.get_job(job_id)
if job is None or not job.get("pptx_ready"):
return JSONResponse(status_code=200, content=_not_found(job_id).model_dump())
png = _preview_path(job_id, page)
if not png.is_file():
# 懒渲染:首次访问补齐(渲染后仍不在 → 页码越界)
written = ppt_preview.render_previews(_pptx_path(job_id), _preview_dir(job_id))
png = _preview_path(job_id, page)
if not png.is_file():
if page < 1:
return JSONResponse(
status_code=200,
content=_error("VALIDATION", "page 必须 ≥1").model_dump(),
)
return JSONResponse(
status_code=200,
content=_error("NOT_FOUND", f"page {page} 不存在(共 {len(written)} 页)").model_dump(),
)
return FileResponse(str(png), media_type="image/png")
@router.delete("/jobs/{job_id}", response_model=PptResponse)
async def delete_job(job_id: str) -> PptResponse:
job = ppt_store.get_job(job_id)
if job is None:
return _not_found(job_id)
ppt_store.delete_job(job_id)
job_dir = get_settings().ppt_jobs_abs / job_id
if job_dir.exists():
shutil.rmtree(job_dir, ignore_errors=True)
return PptResponse(ok=True, meta={"deleted": job_id})
+269
View File
@@ -0,0 +1,269 @@
"""routers/presubmit.py —— 投稿前自审稿(POST /api/review/presubmit)。
流程:上传 PDF → paper_parse(vstore 8094 解析)→ 段落感知装箱 + token 双层保护
→ LLM 结构化审稿(DeepSeek json_object,System Prompt 为植物基因组学六维专属
清单,见 services/self_review.py)→ coerce 归一化 → 关键词扫描器证伪硬性缺项
(services/paper_scanner.py)→ 规则引擎计算 recommendation。
"""
from __future__ import annotations
import logging
import re
from fastapi import APIRouter, File, Form, UploadFile
from pydantic import BaseModel
from ..config import get_settings
from ..services import llm, paper_parse, paper_scanner, self_review
from ..services.recommend_engine import decide_recommendation, resolve_tier
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-presubmit"])
# ---- token 双层保护(2026-10-12)----
# 第一层:估算 token 超阈值 → 自动降到 TOKEN_GUARD_REPACK_CHARS 字符重装箱
# 第二层:LLM 返回 HTTP 400/413 → 降到 LLM_REJECT_RETRY_CHARS 字符重试一次
_EST_TOKENS_DIVISOR = 1.5 # 粗略上界:英文 ~4 字符/token,1.5 偏保守
_TOKEN_GUARD_THRESHOLD = 48000
_TOKEN_GUARD_REPACK_CHARS = 100000
_LLM_REJECT_RETRY_CHARS = 80000
# 无节标题可辨时的尾窗(捞页尾 Data Availability 块)
_TAIL_FALLBACK_CHARS = 5000
# 节标题(行首、允许编号前缀、独占一行);键 → 装箱分组名
_SECTION_HEADER_RE = re.compile(
r"^[ \t]*(?:\d+(?:\.\d+)*\.?\s+)?"
r"(Abstract|Summary|Introduction|Results|Discussion|Methods"
r"|Materials\s+and\s+Methods"
r"|Data\s+(?:and\s+(?:code\s+)?)?[Aa]vailability"
r"|Availability\s+of\s+[Dd]ata"
r"|Data\s+and\s+[Mm]aterials\s+[Aa]vailability"
r"|References|Acknowledg(?:e)?ments?)\s*$",
re.IGNORECASE | re.MULTILINE,
)
_SECTION_GROUP = {
"abstract": "abstract", "summary": "abstract",
"introduction": "introduction",
"results": "results",
"discussion": "discussion",
"methods": "methods", "materials and methods": "methods",
"data availability": "data_availability",
"data and code availability": "data_availability",
"data availability statement": "data_availability",
"availability of data": "data_availability",
"data and materials availability": "data_availability",
"references": "references", "acknowledgements": "acknowledgements",
"acknowledgments": "acknowledgements",
}
# 装箱优先级(首条命中优先;methods 限 60% 预算——六维清单大量依赖 Methods)
_PACK_PRIORITY = ("abstract", "data_availability", "methods", "results", "discussion")
_METHODS_CAP_RATIO = 0.6
def estimate_tokens(text: str) -> int:
"""粗略 token 上界估计(len/1.5),无需 tokenizer 依赖。"""
return int(len(text) / _EST_TOKENS_DIVISOR)
def _split_segments(text: str) -> list[tuple[str, str]] | None:
"""按节标题切段 → [(group, content)...]。无可辨标题 → None。"""
matches = list(_SECTION_HEADER_RE.finditer(text))
if not matches:
return None
segments: list[tuple[str, str]] = []
if matches[0].start() > 0:
segments.append(("preamble", text[:matches[0].start()]))
for i, m in enumerate(matches):
end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
group = _SECTION_GROUP.get(m.group(1).casefold(), m.group(1).casefold())
segments.append((group, text[m.start():end]))
return segments
def pack_feed_text(text: str, budget: int) -> tuple[str, list[str]]:
"""段落感知装箱 → (feed_text, dropped_sections)。
预算内:原文返回。超预算:按优先级保留
① Abstract ② Data Availability 全文 ③ Methods 前 60% ④ Results
⑤ Discussion,其余按原序填满;未收段记入 dropped。
无节标题可辨 → 头 + 尾窗拼接(捞页尾 availability 块),不优于现状亦不劣于。
"""
if len(text) <= budget:
return text, []
segments = _split_segments(text)
if not segments:
sep = "\n[...]\n"
head = max(0, budget - _TAIL_FALLBACK_CHARS - len(sep))
return (
text[:head] + sep + text[-_TAIL_FALLBACK_CHARS:],
["[无节标题回退:保留头+尾窗]"],
)
methods_cap = int(budget * _METHODS_CAP_RATIO)
chosen: dict[int, str] = {}
used = 0
for group in _PACK_PRIORITY:
for i, (g, content) in enumerate(segments):
if g != group or i in chosen:
continue
room = budget - used
if room <= 0:
break
piece = content[:methods_cap] if group == "methods" else content
if len(piece) > room:
piece = piece[:room]
if not piece:
break
chosen[i] = piece
used += len(piece)
break
for i, (_, content) in enumerate(segments):
if i in chosen or used >= budget:
continue
room = budget - used
chosen[i] = content if len(content) <= room else content[:room]
used += len(chosen[i])
feed = "".join(chosen[i] for i in sorted(chosen))
dropped: list[str] = []
for i, (g, _) in enumerate(segments):
if i not in chosen and g not in dropped:
dropped.append(g)
return feed, dropped
def _fit_feed_text(text: str, budget: int) -> tuple[str, list[str]]:
"""装箱 + 第一层 token 保护(估算超阈值自动降到 100K 字符重装)。"""
feed, dropped = pack_feed_text(text, budget)
if estimate_tokens(feed) > _TOKEN_GUARD_THRESHOLD and len(feed) > _TOKEN_GUARD_REPACK_CHARS:
logger.info(
"presubmit: token 估算 %d > %d,降到 %d 字符重装箱",
estimate_tokens(feed), _TOKEN_GUARD_THRESHOLD, _TOKEN_GUARD_REPACK_CHARS,
)
feed, dropped = pack_feed_text(text, _TOKEN_GUARD_REPACK_CHARS)
return feed, dropped
class PresubmitResponse(BaseModel):
ok: bool
report: self_review.SelfReviewReport | None = None
error: dict[str, object] | None = None
meta: dict[str, object] | None = None
def _error(code: str, message: str, *, degraded: bool = False) -> PresubmitResponse:
return PresubmitResponse(
ok=False, report=None,
error={"code": code, "message": message, "degraded": degraded},
)
@router.post("/presubmit", response_model=PresubmitResponse)
async def presubmit_review(
file: UploadFile = File(..., description="待审稿 PDF"),
target_journal: str = Form("", description="目标期刊(可选,影响格式类意见)"),
) -> PresubmitResponse:
settings = get_settings()
filename = file.filename or "paper.pdf"
if not filename.lower().endswith(".pdf"):
return _error("VALIDATION", "仅支持 PDF 文件")
content = await file.read()
if not content:
return _error("VALIDATION", "上传文件为空")
if len(content) > settings.max_pdf_bytes:
return _error("VALIDATION", "PDF 超过 50MB 上限")
# ---- 解析(vstore 8094)----
try:
text = await paper_parse.parse_pdf_bytes(content, filename=filename)
except paper_parse.ParseError as exc:
logger.warning("presubmit: 解析失败:%s", exc)
return _error("PARSE_FAILED", str(exc), degraded=True)
truncated = len(text) > settings.max_text_chars
feed_text, dropped_sections = _fit_feed_text(text, settings.max_text_chars)
# ---- LLM 结构化自审(植物基因组学六维清单)----
# 第二层保护:HTTP 400/413(疑似超长/形态拒绝)→ 降 80K 字符重试一次
user_prompt = self_review.build_user_prompt(feed_text, (target_journal or "").strip())
try:
try:
payload = await llm.complete_json(
self_review.SYSTEM_PROMPT,
user_prompt,
temperature=0.2,
max_tokens=8000,
timeout=settings.presubmit_read_timeout,
)
except llm.LLMError as exc:
msg = str(exc)
if ("HTTP 400" in msg or "HTTP 413" in msg) and len(feed_text) > _LLM_REJECT_RETRY_CHARS:
logger.warning("presubmit: LLM 拒绝输入(%s),降到 %d 字符重试",
msg[:120], _LLM_REJECT_RETRY_CHARS)
feed_text, dropped_sections = pack_feed_text(text, _LLM_REJECT_RETRY_CHARS)
truncated = True
payload = await llm.complete_json(
self_review.SYSTEM_PROMPT,
self_review.build_user_prompt(feed_text, (target_journal or "").strip()),
temperature=0.2,
max_tokens=8000,
timeout=settings.presubmit_read_timeout,
)
else:
raise
except llm.LLMError as exc:
logger.error("presubmit: LLM 审稿失败:%s", exc)
return _error("UPSTREAM_DOWN", f"审稿模型调用失败:{exc}", degraded=True)
report = self_review.coerce_report(payload)
# ---- 关键词扫描器:证伪 LLM 的硬性缺项(有 X 却报缺 X → 真降级至 minor)----
# A2(2026-10-12):被冲突的 major 不再留在 major 列表里只剥标签,而是移出
# major、append 到 minor(scanner_downgraded=True)——major_count/引擎输入真实
# 减少;fatal 条目例外(只记冲突不降级,须人工过目)。
signals = paper_scanner.scan_paper_signals(feed_text)
kept_majors, demoted_minors, scanner_conflicts = paper_scanner.reconcile_hard_rules(
report.major_weaknesses, signals
)
report.major_weaknesses = kept_majors
if demoted_minors:
report.minor_weaknesses = list(report.minor_weaknesses) + list(demoted_minors)
# recommendation 由规则引擎计算(LLM 输出的同名字段已被 coerce 忽略)
tier, tier_source = resolve_tier((target_journal or "").strip())
decision = decide_recommendation(
majors=report.major_weaknesses,
minors=report.minor_weaknesses,
presentations=report.presentation_format,
tier=tier,
)
rationale = decision["rationale"]
rationale["scanner_conflicts"] = scanner_conflicts
if scanner_conflicts:
rationale["explanation"] += (
f" 关键词扫描器与审稿意见存在 {len(scanner_conflicts)} 处冲突,"
f"已降级为小修级问题,请人工核对。"
)
report.recommendation = decision["recommendation"]
report.recommendation_rationale = rationale
return PresubmitResponse(
ok=True,
report=report,
meta={
"filename": filename,
"target_journal": (target_journal or "").strip(),
"text_chars": len(feed_text),
"truncated": truncated,
"dropped_sections": dropped_sections,
"scanner": {
"accession_found": bool((signals.get("accession") or {}).get("found")),
"busco_level": (signals.get("busco") or {}).get("level"),
"hic_found": bool((signals.get("hic") or {}).get("found")),
},
"model": llm._llm_config()["model"],
"tier": tier,
"tier_label": decision["rationale"]["tier_label"],
"tier_source": tier_source,
},
)
+288
View File
@@ -0,0 +1,288 @@
"""routers/rebuttal.py —— Rebuttal Phase 1(审稿回复助手,10 端点)。
信封与其余 review 端点一致:业务错误 = HTTP 200 + {ok:false,error:{code,message,degraded}}。
- 文本粘贴主入口 / PDF 次要入口(PDF 经 vstore 8094 解析);
- 逐条 response → draft(校验C前置)→ confirm(后端重算 human_ack)→ letter 拼装;
- letter.md 下载门槛 = human_ack;DELETE 手动清(无 TTL,决策 5)。
"""
from __future__ import annotations
import logging
from typing import Any, Literal
from fastapi import APIRouter, File, UploadFile
from fastapi.responses import PlainTextResponse
from pydantic import BaseModel, Field
from ..config import get_settings
from ..services import llm, paper_parse
from ..services.rebuttal_draft import PART_TYPES, assemble_letter, generate_draft, validate_parts
from ..services.rebuttal_parse import parse_review_text
from ..services import rebuttal_store
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review/rebuttal", tags=["review-rebuttal"])
_MAX_RAW_CHARS = 200_000 # 粘贴文本上限(审稿意见通常 <20K,保守给大)
# ---------- 信封 ----------
class RebuttalResponse(BaseModel):
ok: bool
job: dict[str, Any] | None = None
error: dict[str, Any] | None = None
meta: dict[str, Any] | None = None
def _error(code: str, message: str) -> RebuttalResponse:
return RebuttalResponse(ok=False, error={"code": code, "message": message, "degraded": False})
def _ok(job: dict[str, Any], **meta: Any) -> RebuttalResponse:
return RebuttalResponse(ok=True, job=job, meta=meta or None)
# ---------- 请求体 ----------
class JobCreateBody(BaseModel):
text: str = Field(min_length=1)
class PartBody(BaseModel):
type: Literal["new_data", "reanalysis", "location", "clarification", "rebuttal"]
content: str = ""
location_hint: str | None = None
class ResponseBody(BaseModel):
parts: list[PartBody] = Field(default_factory=list)
class StructureBody(BaseModel):
action: Literal["merge", "split"]
# merge: comment_ids(含当前 cid,≥2 条);split: items [{title, body}]
comment_ids: list[str] | None = None
items: list[dict[str, str]] | None = None
class ConfirmBody(BaseModel):
confirmed: bool
# ---------- 内部工具 ----------
async def _messages_llm(messages: list[dict[str, str]], *, temperature: float = 0.3) -> dict[str, Any]:
"""适配 rebuttal_parse/draft 的 messages 形态 → llm.complete_json。"""
system = next((m["content"] for m in messages if m.get("role") == "system"), "")
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
return await llm.complete_json(system, user, temperature=temperature)
def _find_comment(job: dict[str, Any], cid: str) -> tuple[dict, dict] | None:
for r in job["reviewers"]:
for c in r["comments"]:
if c["id"] == cid:
return r, c
return None
async def _build_job(raw_text: str, *, kind: str, filename: str | None) -> RebuttalResponse:
raw_text = (raw_text or "").strip()
if not raw_text:
return _error("VALIDATION", "审稿意见文本为空")
if len(raw_text) > _MAX_RAW_CHARS:
return _error("VALIDATION", f"文本超过 {_MAX_RAW_CHARS} 字符上限")
reviewers = await parse_review_text(raw_text, _messages_llm)
if not reviewers:
return _error("PARSE_FAILED", "未能从文本中解析出任何审稿意见条目")
job = {
"id": rebuttal_store.new_id("rbt_"),
"status": "ready",
"source": {"kind": kind, "filename": filename, "raw_text": raw_text},
"template": "generic",
"reviewers": reviewers,
"letter_md": None,
"validation": None,
"human_ack": False,
}
rebuttal_store.insert_job(job)
n_comments = sum(len(r["comments"]) for r in reviewers)
return _ok(job, reviewers=len(reviewers), comments=n_comments)
# ---------- 端点 ----------
@router.post("/jobs", response_model=RebuttalResponse)
async def create_job(body: JobCreateBody) -> RebuttalResponse:
"""文本粘贴主入口。"""
return await _build_job(body.text, kind="text", filename=None)
@router.post("/jobs/pdf", response_model=RebuttalResponse)
async def create_job_pdf(file: UploadFile = File(...)) -> RebuttalResponse:
"""PDF 次要入口(批注版审稿 PDF;经 vstore 8094 解析)。"""
settings = get_settings()
filename = file.filename or "review.pdf"
if not filename.lower().endswith(".pdf"):
return _error("VALIDATION", "仅支持 PDF 文件(推荐直接粘贴文本)")
data = await file.read()
if not data:
return _error("VALIDATION", "上传文件为空")
if len(data) > settings.max_pdf_bytes:
return _error("VALIDATION", f"PDF 超过 {settings.max_pdf_bytes // (1024 * 1024)}MB 上限")
try:
text = await paper_parse.parse_pdf_bytes(data, filename=filename)
except Exception as exc: # noqa: BLE001
return _error("UPSTREAM_DOWN", f"PDF 解析失败:{exc}")
return await _build_job(text, kind="pdf", filename=filename)
@router.get("/jobs/{job_id}", response_model=RebuttalResponse)
async def get_job(job_id: str) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
return _ok(job)
@router.patch("/jobs/{job_id}/comments/{cid}/response", response_model=RebuttalResponse)
async def patch_response(job_id: str, cid: str, body: ResponseBody) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
found = _find_comment(job, cid)
if found is None:
return _error("NOT_FOUND", f"comment {cid} 不存在")
_, comment = found
for p in body.parts:
if p.type not in PART_TYPES:
return _error("VALIDATION", f"未知 part.type: {p.type}")
comment["response"] = {"parts": [p.model_dump() for p in body.parts]}
# 改回复 → 旧 draft/validation 作废(保守:不保留过期草稿)
comment["draft"] = None
comment["validation"] = None
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
return _ok(job)
@router.post("/jobs/{job_id}/comments/{cid}/draft", response_model=RebuttalResponse)
async def post_draft(job_id: str, cid: str) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
found = _find_comment(job, cid)
if found is None:
return _error("NOT_FOUND", f"comment {cid} 不存在")
_, comment = found
parts = (comment.get("response") or {}).get("parts") or []
if not parts or not any(str(p.get("content") or "").strip() for p in parts):
# 决策 4:生成前必须填 user_response;LLM 调用次数=0
return _error("MISSING_RESPONSE", "请先填写回复要点(user_response)再生成草稿")
validation = validate_parts(parts)
comment["validation"] = validation
if not validation["passed"]:
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
return RebuttalResponse(
ok=False, job=job,
error={"code": "VALIDATION_FAILED", "message": "反幻觉校验未通过",
"degraded": False},
meta={"validation": validation},
)
comment["draft"] = await generate_draft(_messages_llm, comment)
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
return _ok(job, validation=validation)
@router.patch("/jobs/{job_id}/comments/{cid}/structure", response_model=RebuttalResponse)
async def patch_structure(job_id: str, cid: str, body: StructureBody) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
found = _find_comment(job, cid)
if found is None:
return _error("NOT_FOUND", f"comment {cid} 不存在")
reviewer, comment = found
comments = reviewer["comments"]
idx = comments.index(comment)
if body.action == "merge":
ids = body.comment_ids or []
if cid not in ids:
ids = [cid] + ids
if len(ids) < 2:
return _error("VALIDATION", "merge 至少需要 2 条 comment")
targets = [c for c in comments if c["id"] in ids]
if len(targets) < 2:
return _error("VALIDATION", "merge 目标中有效 comment 不足 2 条")
merged_body = "\n\n".join(c["body"] for c in targets)
comment["body"] = merged_body
comment["title"] = f"{comment['title']} (+{len(targets) - 1} merged)"
reviewer["comments"] = [
c for c in comments
if c is comment or c["id"] not in ids
]
else: # split
items = body.items or []
items = [i for i in items if str(i.get("body") or "").strip()]
if len(items) < 2:
return _error("VALIDATION", "split 至少需要 2 个非空子项")
group_id = comment.get("group_id") or f"g_{cid}"
new_comments = []
for n, it in enumerate(items, start=1):
c = dict(comment)
c["id"] = rebuttal_store.new_id(f"{cid}s{n}")
c["group_id"] = group_id
c["title"] = str(it.get("title") or f"{comment['title']} ({n})")
c["body"] = str(it.get("body") or "").strip()
c["sub_items"] = []
c["response"] = {"parts": []}
c["draft"] = None
c["validation"] = None
c["human_confirmed"] = False
new_comments.append(c)
comments[idx:idx + 1] = new_comments
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
return _ok(job)
@router.patch("/jobs/{job_id}/comments/{cid}/confirm", response_model=RebuttalResponse)
async def patch_confirm(job_id: str, cid: str, body: ConfirmBody) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
found = _find_comment(job, cid)
if found is None:
return _error("NOT_FOUND", f"comment {cid} 不存在")
_, comment = found
comment["human_confirmed"] = bool(body.confirmed)
# human_ack 由 store 重算(决策 4:全勾才置位,不信前端)
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
return _ok(job, human_ack=job["human_ack"])
@router.post("/jobs/{job_id}/letter", response_model=RebuttalResponse)
async def post_letter(job_id: str) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
job = rebuttal_store.update_job(job_id, letter_md=assemble_letter(job)) or job
return _ok(job)
@router.get("/jobs/{job_id}/letter.md")
async def download_letter(job_id: str) -> PlainTextResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return PlainTextResponse(
"job not found", status_code=404, media_type="text/plain; charset=utf-8")
if not job.get("human_ack"):
# 信封口径与下载端点适配:GET 下载返回纯文本,业务拒绝用 400 + 明文
return PlainTextResponse(
"HUMAN_ACK_REQUIRED: 请逐条勾选确认后再下载 letter.md",
status_code=400, media_type="text/plain; charset=utf-8")
md = job.get("letter_md") or assemble_letter(job)
return PlainTextResponse(md, media_type="text/markdown; charset=utf-8")
@router.delete("/jobs/{job_id}", response_model=RebuttalResponse)
async def delete_job(job_id: str) -> RebuttalResponse:
if not rebuttal_store.delete_job(job_id):
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
return RebuttalResponse(ok=True, job=None, meta={"deleted": job_id})
+165
View File
@@ -0,0 +1,165 @@
"""routers/reformat.py —— 论文自动排版 P1(.tex 源码级重排版,不编译)。
端点:
POST /api/review/reformat multipart 上传 .tex + target_journal →
修改后源码 + 逐行 diff 报告(job 落盘)
GET /api/review/reformat/{id} 取完整报告(重启后仍在)
GET /api/review/reformat/{id}/file 下载修改后的 .tex
信封与其余 review 端点一致:业务错误 = HTTP 200 + {ok:false,error:{code,message,degraded:false}},
前端经 review.js unwrap 归一为 ApiError。本次改动无上游依赖(零 LLM/零 vstore),
degraded 恒 false。
"""
from __future__ import annotations
import hashlib
import json
import logging
import re
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from fastapi import APIRouter, File, Form, UploadFile
from fastapi.responses import FileResponse, JSONResponse
from pydantic import BaseModel
from ..config import get_settings
from ..services.reformat import ReformatError, reformat_tex, resolve_rules
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-reformat"])
_JOB_ID_RE = re.compile(r"^[0-9a-f]{12}$")
_ENGINE = "reformat-p1/1.0"
class ReformatResponse(BaseModel):
ok: bool
job_id: str | None = None
modified_tex: str | None = None
changes: list[dict[str, Any]] | None = None
warnings: list[dict[str, Any]] | None = None
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
def _error(code: str, message: str) -> ReformatResponse:
return ReformatResponse(
ok=False,
error={"code": code, "message": message, "degraded": False},
)
def _jobs_dir() -> Path:
d = get_settings().reformat_dir_abs
d.mkdir(parents=True, exist_ok=True)
return d
@router.post("/reformat", response_model=ReformatResponse)
async def reformat(
file: UploadFile = File(..., description="LaTeX 源码(.tex 单文件)"),
target_journal: str = Form("", description="目标期刊 id/名称,或 default=通用规则"),
) -> ReformatResponse:
settings = get_settings()
# ---- 入参校验(拍板 5:只收 .tex,docx/pdf 走 format-check 审核不重排)----
filename = file.filename or "paper.tex"
if not filename.lower().endswith(".tex"):
return _error("VALIDATION", "仅支持 .tex 文件(docx/pdf 请走格式审核)")
target = (target_journal or "").strip()
if not target:
return _error("VALIDATION", "必须指定目标期刊(或选择「通用规则」)")
content = await file.read()
if not content:
return _error("VALIDATION", "上传文件为空")
if len(content) > settings.max_tex_bytes:
return _error("VALIDATION", "源码超过 2MB 上限")
try:
source = content.decode("utf-8")
except UnicodeDecodeError:
return _error("VALIDATION", "源码必须为 UTF-8 编码")
# ---- 规则解析(journals.json / format_cache / journal_directory 只读回链)----
resolved = await resolve_rules(target)
# ---- 变换(纯规则引擎;结构畸形 → PARSER_FAILED)----
try:
result = reformat_tex(source, resolved["rules"])
except ReformatError as exc:
logger.warning("reformat: 结构守卫拒收 %s: %s", filename, exc)
return _error(exc.code, str(exc))
# ---- 落盘(.tex + .json 报告;重启后 GET 仍可用)----
job_id = uuid.uuid4().hex[:12]
base = _jobs_dir()
(base / f"{job_id}.tex").write_text(result["modified_tex"], encoding="utf-8")
meta: dict[str, Any] = {
"journal_id": resolved["journal_id"],
"journal_name": resolved["journal_name"],
"rules": resolved["rules"],
"rules_source": resolved["rules_source"],
"rules_note": resolved["rules_note"],
"filename": filename,
"original_sha256": hashlib.sha256(content).hexdigest(),
"engine": _ENGINE,
"generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
}
payload = {
"ok": True,
"job_id": job_id,
"modified_tex": result["modified_tex"],
"changes": result["changes"],
"warnings": result["warnings"],
"meta": meta,
}
(base / f"{job_id}.json").write_text(
json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
)
logger.info(
"reformat: %s → job %s(%d 处修改, %d 条提示, 规则 %s)",
filename, job_id,
sum(1 for c in result["changes"] if c["action"] == "modified"),
sum(1 for c in result["changes"] if c["action"] == "warning"),
resolved["rules_source"],
)
return ReformatResponse(**payload)
def _load_job(job_id: str) -> dict[str, Any] | None:
if not _JOB_ID_RE.fullmatch(job_id or ""):
return None
path = _jobs_dir() / f"{job_id}.json"
try:
payload = json.loads(path.read_text("utf-8"))
except (OSError, ValueError):
return None
return payload if isinstance(payload, dict) and payload.get("ok") else None
_NOT_FOUND = {"ok": False, "error": {"code": "NOT_FOUND",
"message": "任务不存在或已过期,请重新上传源码",
"degraded": False}}
@router.get("/reformat/{job_id}", response_model=ReformatResponse)
async def get_reformat_job(job_id: str) -> ReformatResponse:
payload = _load_job(job_id)
if payload is None:
return _error("NOT_FOUND", "任务不存在或已过期,请重新上传源码")
return ReformatResponse(**payload)
@router.get("/reformat/{job_id}/file", response_model=None)
async def download_reformat_file(job_id: str) -> FileResponse | JSONResponse:
"""下载修改后的 .tex;缺件/非法 id → 200 信封(前端单一错误口径)。"""
valid = bool(_JOB_ID_RE.fullmatch(job_id or ""))
tex = _jobs_dir() / f"{job_id}.tex" if valid else None
if tex is None or not tex.is_file():
return JSONResponse(status_code=200, content=_NOT_FOUND)
return FileResponse(tex, media_type="application/x-tex", filename=f"{job_id}.tex")
+1
View File
@@ -0,0 +1 @@
"""review 服务服务层包。"""
+788
View File
@@ -0,0 +1,788 @@
"""services/format_check.py —— 版式与规范规则引擎(⑤ 论文排版与规范)。
纯规则、无 LLM、无 vstore、同步执行(通常 <1s):
- PDF → pymupdf 本地解档(AGENTS.md 约束 6 例外:vstore 只回文本,字体/边距/
行距/页数等版式元数据拿不到;正文提取仍走 vstore,本服务不碰正文语义);
- Word → python-docx(页边距/行距精确,页数取 docProps/app.xml 的 <Pages>);
- 规则集 → data/journals/journals.json:
顶层 format_rules_default(通用默认值,rules_source="default")
每条目可选 format_rules(按期刊覆盖,必须带 rules_source: placeholder|official|default)
⚠ 诚实原则:不知道期刊真实投稿指南 → 该条只写 {"rules_source": "placeholder"},
所有具体数字回退 default 通用值,绝不编造期刊专属数字。
输出:{checks[{name,key,status,detail,suggestion}], summary{pass,warn,fail}, meta{...}}
status ∈ pass|warn|fail。参考文献样式宽容化:仅"明确不符"(如 [J] 标引但规则要求 APA)
才 fail,"疑似但不确定/检测不到"一律 warn,避免误伤。
"""
from __future__ import annotations
import io
import json
import re
import statistics
import zipfile
from pathlib import Path
from typing import Any
import fitz # pymupdf(venv 已装 1.28.x)
from docx import Document
from docx.shared import Length
# review_service/app/services/ → parents[2] = review_service
_JOURNALS_PATH = Path(__file__).resolve().parents[2] / "data" / "journals" / "journals.json"
_MAX_SCAN_PAGES = 80 # 版式扫描页数上限(正文/字体/边距;稿件通常远小于此)
_MAX_REFS_SCAN = 40 # 从文末向前找参考文献标题的页数窗口
_MAX_PARAS = 8000 # docx 段落扫描上限
_PT_TO_MM = 25.4 / 72.0
_EMU_TO_MM = 1.0 / 36000.0
_HEADING_RE = re.compile(
r"^\s*(references|bibliography|literature cited|works cited|参考文献|文献)\s*[::]?\s*$",
re.IGNORECASE,
)
_FIG_RE = re.compile(r"(?im)^\s*(figure|fig\.?|图)\s*\d+")
_TAB_RE = re.compile(r"(?im)^\s*(table|表)\s*\d+")
_GB_TAG_RE = re.compile(r"\[[A-Z]{1,3}\d*\]") # [J] [M] [D] [C] …(GB/T 7714 标引)
_PAREN_YEAR_RE = re.compile(r"\((?:19|20)\d{2}\)") # (2024) 作者-年份制
_BRACKET_NUM_RE = re.compile(r"^\[\d+\]") # [1] 编号制
_DOI_RE = re.compile(r"doi:?\.?org|doi:\s*1\.", re.IGNORECASE)
_STYLE_LABEL = {
"gb7714": "GB/T 7714",
"author_year": "作者-年份制(类 APA)",
"numbered": "编号制(顺序编码)",
"unknown": "无法确定",
}
class FormatCheckError(Exception):
"""解档/规则加载失败(路由器转 PARSE_FAILED)。"""
# ---------------------------------------------------------------- 规则集加载
def load_ruleset(target_journal: str) -> tuple[dict[str, Any], dict[str, Any]]:
"""返回 (rules, ruleset_meta)。target_journal 传 id 或 name 均可;未知 → default。"""
try:
data = json.loads(_JOURNALS_PATH.read_text("utf-8"))
except Exception as exc: # noqa: BLE001
raise FormatCheckError(f"规则集加载失败:{exc}") from exc
default = data.get("format_rules_default") or {}
jid = (target_journal or "").strip()
entry = None
if jid:
for j in data.get("journals") or []:
if j.get("id") == jid or j.get("name") == jid:
entry = j
break
if entry is None:
rules = dict(default)
meta = {
"journal_id": None,
"journal_name": None,
"ruleset_source": "default",
"rules_source": str(default.get("rules_source", "default")),
}
else:
journal_rules = entry.get("format_rules") or {}
rules = {**default, **journal_rules}
meta = {
"journal_id": entry.get("id"),
"journal_name": entry.get("name"),
"ruleset_source": "journal",
# 占位规则(无官方核验)→ 前端黄字提示
"rules_source": str(journal_rules.get("rules_source", "placeholder")),
}
rules.setdefault("rules_source", "default")
return rules, meta
# ---------------------------------------------------------------- PDF 解档(pymupdf)
def _norm_font(name: str) -> str:
n = (name or "").split("+")[-1].strip()
return n or "unknown"
def extract_profile_pdf(data: bytes) -> dict[str, Any]:
try:
doc = fitz.open(stream=data, filetype="pdf")
except Exception as exc: # noqa: BLE001
raise FormatCheckError(f"PDF 解档失败:{exc}") from exc
if getattr(doc, "needs_pass", False):
doc.close()
raise FormatCheckError("PDF 已加密,无法解析")
pages_total = doc.page_count
scan = min(pages_total, _MAX_SCAN_PAGES)
margin_rows: list[tuple[float, float, float, float]] = [] # t r b l (mm)
font_w: dict[str, int] = {}
size_w: dict[float, int] = {}
ratios: list[float] = []
img_ids: set[int] = set()
fig_caps = tab_caps = 0
for pno in range(scan):
page = doc[pno]
rect = page.rect
blocks = page.get_text("blocks")
content = [b for b in blocks if b[4].strip()]
if content:
left = min(b[0] for b in content)
top = min(b[1] for b in content)
right = rect.width - max(b[2] for b in content)
bottom = rect.height - max(b[3] for b in content)
margin_rows.append(
(
max(0.0, top) * _PT_TO_MM,
max(0.0, right) * _PT_TO_MM,
max(0.0, bottom) * _PT_TO_MM,
max(0.0, left) * _PT_TO_MM,
)
)
for img in page.get_images(full=True):
img_ids.add(img[0])
text = page.get_text()
fig_caps += len(_FIG_RE.findall(text))
tab_caps += len(_TAB_RE.findall(text))
d = page.get_text("dict")
for block in d.get("blocks", []):
if block.get("type") != 0:
continue
line_meta: list[tuple[float, float]] = [] # (基线 y, 字号)
for line in block.get("lines", []):
spans = [s for s in line.get("spans", []) if s.get("text", "").strip()]
if not spans:
continue
row_size = max(s.get("size", 0.0) for s in spans)
if row_size <= 0:
continue
for s in spans:
size = round(float(s.get("size", 0.0)), 1)
chars = len(s.get("text", ""))
if size > 0:
norm = _norm_font(s.get("font", ""))
font_w[norm] = font_w.get(norm, 0) + chars
size_w[size] = size_w.get(size, 0) + chars
# 行距基准用 span.origin 基线 y,不用 bbox 顶缘:bbox 顶随字形高低
# 漂移,表格/图注等同基线并列文本会产出假性小 delta,混入中位数会把
# 行距拉成 0.1x 的不可能值。
origin = spans[0].get("origin")
line_meta.append((origin[1] if origin else line["bbox"][1], row_size))
for i in range(1, len(line_meta)):
delta = line_meta[i][0] - line_meta[i - 1][0]
size = line_meta[i][1]
# 行距倍数 = 相邻行基线 y 差 / 当前行字号;
# 过滤物理不可能值:<0.5(行重叠/同基线并列表格)、>4.0(异常结构)
ratio = delta / size
if 0.5 <= ratio <= 4.0:
ratios.append(ratio)
refs_text, refs_found = _find_refs_pdf(doc, pages_total)
doc.close()
if margin_rows:
med = [statistics.median(col) for col in zip(*margin_rows)]
margins = {"top": med[0], "right": med[1], "bottom": med[2], "left": med[3]}
else:
margins = None
return {
"source_kind": "pdf",
"pages": pages_total,
"margins_mm": margins,
"font_weights": font_w,
"size_weights": size_w,
"line_spacing_median": statistics.median(ratios) if ratios else None,
"figures": max(len(img_ids), fig_caps),
"tables": tab_caps,
"refs_text": refs_text,
"refs_found": refs_found,
}
def _find_refs_pdf(doc: "fitz.Document", pages_total: int) -> tuple[str | None, bool]:
"""从文末向前找参考文献标题行;找到 → 标题行至文末文本;找不到 → (None, False)。"""
buf: list[str] = []
start = max(0, pages_total - _MAX_REFS_SCAN)
for pno in range(pages_total - 1, start - 1, -1):
page_text = doc[pno].get_text()
lines = page_text.splitlines()
for idx in range(len(lines) - 1, -1, -1):
if _HEADING_RE.match(lines[idx].strip()):
tail = "\n".join(lines[idx + 1:])
joined = "\n".join([tail] + buf).strip()
return (joined if len(joined) >= 50 else None), bool(joined)
buf.insert(0, page_text)
return None, False
# ---------------------------------------------------------------- Word 解档(python-docx)
def _docx_app_pages(data: bytes) -> int | None:
try:
with zipfile.ZipFile(io.BytesIO(data)) as zf:
xml = zf.read("docProps/app.xml").decode("utf-8", "ignore")
m = re.search(r"<Pages>(\d+)</Pages>", xml)
return int(m.group(1)) if m else None
except Exception: # noqa: BLE001
return None
def extract_profile_docx(data: bytes) -> dict[str, Any]:
try:
doc = Document(io.BytesIO(data))
except Exception as exc: # noqa: BLE001
raise FormatCheckError(f"Word 文件解析失败:{exc}") from exc
side_lists: dict[str, list[float]] = {"top": [], "right": [], "bottom": [], "left": []}
for sec in doc.sections:
for k, v in (
("top", sec.top_margin),
("right", sec.right_margin),
("bottom", sec.bottom_margin),
("left", sec.left_margin),
):
if v is not None:
side_lists[k].append(float(v) * _EMU_TO_MM)
margins = None
if all(side_lists.values()):
margins = {k: statistics.median(v) for k, v in side_lists.items()}
try:
normal = doc.styles["Normal"].font
default_name = normal.name
default_size = float(normal.size.pt) if normal.size is not None else None
except Exception: # noqa: BLE001
default_name, default_size = None, None
font_w: dict[str, int] = {}
size_w: dict[float, int] = {}
ratios: list[float] = []
fig_caps = tab_caps = 0
paras = doc.paragraphs
scan_count = min(len(paras), _MAX_PARAS)
for i in range(scan_count):
para = paras[i]
text = para.text
if not text.strip():
continue
ls = para.paragraph_format.line_spacing
if isinstance(ls, Length):
# 精确行距(exactly)以 EMU 存储,Length 是 int 子类——直接当倍数会得出
# 十万级的假值;用正文默认字号换算成倍数
if default_size:
r = float(ls.pt) / default_size
if 0.5 <= r <= 4.0:
ratios.append(r)
elif isinstance(ls, (int, float)) and not isinstance(ls, bool) and ls > 0:
r = float(ls) # multiple 行距本身就是倍数
if 0.5 <= r <= 4.0:
ratios.append(r)
if _FIG_RE.match(text):
fig_caps += 1
elif _TAB_RE.match(text):
tab_caps += 1
for run in para.runs:
if not run.text:
continue
name = run.font.name or default_name or "unknown"
pt = run.font.size.pt if run.font.size is not None else default_size
chars = len(run.text)
font_w[str(name)] = font_w.get(str(name), 0) + chars
if pt:
size_w[round(float(pt), 1)] = size_w.get(round(float(pt), 1), 0) + chars
# 参考文献节:标题段落之后全部文本
refs_text, refs_found = None, False
for i, para in enumerate(paras):
if _HEADING_RE.match((para.text or "").strip()):
tail = "\n".join(p.text for p in paras[i + 1:] if (p.text or "").strip())
if len(tail) >= 50:
refs_text, refs_found = tail, True
break
return {
"source_kind": "docx",
"pages": _docx_app_pages(data), # Word 未记录 → None(检查项降级为 warn)
"margins_mm": margins,
"font_weights": font_w,
"size_weights": size_w,
"line_spacing_median": statistics.median(ratios) if ratios else None,
"figures": max(len(doc.inline_shapes), fig_caps),
"tables": max(len(doc.tables), tab_caps),
"refs_text": refs_text,
"refs_found": refs_found,
}
# ---------------------------------------------------------------- LaTeX 解档(正则+状态机,零新依赖)
#
# 决策前提:只做"解析 + 检查 + 建议代码片段",不自动改写 LaTeX(用户可能有自定义宏,
# 自动改写会引错;LaTeX 应由用户自己编辑)。不引入 pylatexenc/texsoup。
# 会相互覆盖的字体类宏包:≥2 个同时引入 → warn(times + mathptmx 类冲突)
_TEX_FONT_PKGS = {"times", "mathptmx", "newtx", "newpx", "txfonts", "pxfonts", "mathpazo"}
def _strip_tex_comments(src: str) -> str:
r"""去行内 % 注释(r-string,避免 \% 被解释为转义序列)。"""
return re.sub(r"(?<!\\)%.*", "", src)
def _parse_geometry(src: str) -> dict[str, float] | None:
"""\\usepackage[...]{geometry} 或 \\geometry{...} → {边: mm}(可缺边,检查端容缺)。"""
m = re.search(r"\\usepackage\s*\[([^\]]*)\]\s*\{\s*geometry\s*\}", src)
if not m:
m = re.search(r"\\geometry\s*\{([^}]*)\}", src)
if not m:
return None
opts = m.group(1)
def to_mm(val: str) -> float | None:
mm = re.match(r"\s*([\d.]+)\s*(mm|cm|in|pt)?\s*$", val)
if not mm:
return None
x = float(mm.group(1))
unit = mm.group(2) or "pt" # geometry 裸数字按 TeX 惯例为 pt
return {"mm": x, "cm": x * 10, "in": x * 25.4, "pt": x * 25.4 / 72.0}[unit]
kv = {}
for pair in opts.split(","):
if "=" in pair:
k, v = pair.split("=", 1)
kv[k.strip().lower()] = v.strip()
out: dict[str, float] = {}
if "margin" in kv:
v = to_mm(kv["margin"])
if v is not None:
return {"top": v, "right": v, "bottom": v, "left": v}
for k in ("top", "right", "bottom", "left"):
if k in kv:
v = to_mm(kv[k])
if v is not None:
out[k] = v
if "inner" in kv:
v = to_mm(kv["inner"])
if v is not None:
out.setdefault("left", v)
if "outer" in kv:
v = to_mm(kv["outer"])
if v is not None:
out.setdefault("right", v)
return out or None
def extract_tex_profile(data: bytes) -> dict[str, Any]:
"""正则 + 轻量状态机解析 .tex 源文件 → 与 DocProfile 同构(+tex 扩展字段)。"""
try:
text = data.decode("utf-8")
except UnicodeDecodeError:
text = data.decode("latin-1", errors="replace")
src = _strip_tex_comments(text)
# \documentclass[10pt,a4paper]{article}
m_cls = re.search(r"\\documentclass\s*(?:\[[^\]]*\])?\s*\{\s*([\w*]+)\s*\}", src)
cls = m_cls.group(1) if m_cls else None
m_opt = re.search(r"\\documentclass\s*\[([^\]]*)\]", src)
cls_opts = m_opt.group(1) if m_opt else ""
m_pt = re.search(r"(\d+)\s*pt", cls_opts)
# 标准类未写选项时 LaTeX 默认 10pt;无 documentclass → None(检查项降级 warn)
fontsize = int(m_pt.group(1)) if m_pt else (10 if cls else None)
paper = "a4paper" if "a4paper" in cls_opts else ("letterpaper" if "letterpaper" in cls_opts else None)
# \usepackage 列表(支持 {a,b} 逗号多包)
packages: list[str] = []
for m in re.finditer(r"\\usepackage\s*(?:\[[^\]]*\])?\s*\{\s*([^}]+?)\s*\}", src):
packages += [p.strip().lower() for p in m.group(1).split(",") if p.strip()]
# 行距:\linespread{...} 或 \setstretch{...}(值直接按"行距倍数"口径评估——
# 按步骤4 规格:低于 min_line_spacing 即 fail;LaTeX 实际倍数还含 baselinestretch
# 的字号比,本引擎取显式设置值做保守判定)
line_spread: float | None = None
m = re.search(r"\\linespread\s*\{\s*([\d.]+)\s*\}", src)
if not m:
m = re.search(r"\\setstretch\s*\{\s*([\d.]+)\s*\}", src)
if m:
line_spread = float(m.group(1))
# 页边距(geometry)
margins = _parse_geometry(src)
# 计数
figures = len(re.findall(r"\\begin\s*\{\s*figure\*?\s*\}", src))
tables = len(re.findall(r"\\begin\s*\{\s*table\*?\s*\}", src))
cites = len(re.findall(r"\\cite\w*\s*\{", src))
# 参考文献:\bibliography{...}(BibTeX)/ \printbibliography(biblatex)/ thebibliography
has_bibtex = bool(re.search(r"\\bibliography\s*\{", src))
has_printbib = bool(re.search(r"\\printbibliography", src))
has_thebib = bool(re.search(r"\\begin\s*\{\s*thebibliography\s*\}", src))
has_bibstyle = bool(re.search(r"\\bibliographystyle\s*\{", src)) # BibTeX 必需;biblatex 不需要
has_bib = has_bibtex or has_printbib or has_thebib
refs_text: str | None = None
m_bib = re.search(r"\\begin\s*\{\s*thebibliography\s*\}[\s\S]*?\\end\s*\{\s*thebibliography\s*\}", src)
if m_bib:
refs_text = m_bib.group(0)
# 宏表(\newcommand{\xxx} / \renewcommand)——用于解析准确性(识别自定义宏)
macros = re.findall(r"\\(?:new|re)command\s*\{\s*\\(\w+)\s*\}", src)
# 匿名评审命令探测(规则集 anonymous:true 时检查)
m_anon = re.search(r"\\(?:anonymi[sz]e|anon\b|hideauthor|ifdoubleblind)", src, re.IGNORECASE)
return {
"source_kind": "tex",
"pages": None, # LaTeX 源文件页数需编译后才有 → 检查项降级 warn(宽容化)
"margins_mm": margins,
"font_weights": {},
"size_weights": {float(fontsize): 100} if fontsize else {},
"line_spacing_median": line_spread,
"figures": figures,
"tables": tables,
"refs_text": refs_text,
"refs_found": has_bib,
"tex": {
"class": cls,
"fontsize_pt": fontsize,
"paper": paper,
"packages": packages,
"cites": cites,
"macros": macros,
"anon_cmd": m_anon.group(0) if m_anon else None,
"bib_bibtex": has_bibtex,
"bib_style": has_bibstyle,
},
}
# ---------------------------------------------------------------- 参考文献样式检测
def detect_refs_style(text: str | None) -> tuple[str, str]:
"""宽容化检测:返回 (detected, sample_line)。含糊 → unknown(后续只 warn)。"""
if not text:
return "unknown", ""
lines = [l.strip() for l in text.splitlines() if l.strip()][:40]
if not lines:
return "unknown", ""
n = len(lines)
gb = sum(1 for l in lines if _GB_TAG_RE.search(l))
year = sum(1 for l in lines if _PAREN_YEAR_RE.search(l))
num = sum(1 for l in lines if _BRACKET_NUM_RE.match(l))
doi = sum(1 for l in lines if _DOI_RE.search(l))
sample = lines[0][:80]
if gb / n >= 0.4:
return "gb7714", sample
if num / n >= 0.4 and year / n < 0.3:
return "numbered", sample
if year / n >= 0.3 or doi / n >= 0.3:
return "author_year", sample
return "unknown", sample
# ---------------------------------------------------------------- 检查项构建
def _chk(key: str, name: str, status: str, detail: str, suggestion: str = "") -> dict[str, str]:
return {"key": key, "name": name, "status": status, "detail": detail, "suggestion": suggestion}
def _check_pages(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
limit = int(rules.get("max_pages", 10))
pages = profile.get("pages")
kind = profile.get("source_kind") or "doc"
if pages is None:
# 宽容化:Word/LaTeX 页数缺失绝不 fail
if kind == "tex":
return _chk(
"pages", "页数", "warn",
f"LaTeX 源无法直接统计页数(规则上限 {limit} 页),需编译后核对",
"编译后确认(如 xelatex main.tex)总页数 ≤ 上限",
)
return _chk(
"pages", "页数", "warn",
"Word 未记录页数,请自行核对",
f"在 Word 中按 Ctrl+End 查看总页数,或导出 PDF 后重查(规则上限 {limit} 页)",
)
if pages <= limit:
return _chk("pages", "页数", "pass", f"共 {pages} 页,规则上限 {limit} 页")
return _chk(
"pages", "页数", "fail", f"共 {pages} 页,超出规则上限 {limit} 页",
f"压缩至 ≤{limit} 页,或将方法/图表移入补充材料",
)
def _check_fonts(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
min_font = float(rules.get("min_font_pt", 9))
# LaTeX:字号来自 \documentclass 选项(10/11/12pt,粒度粗)
if profile.get("source_kind") == "tex":
tex = profile.get("tex") or {}
cls = tex.get("class") or "documentclass"
fs = tex.get("fontsize_pt")
pkgs = tex.get("packages") or []
font_pkgs = sorted(p for p in pkgs if p in _TEX_FONT_PKGS or p in ("helvet", "courier", "charter", "fourier"))
detail = (
f"\\documentclass 选项字号 {fs}pt(类 {cls});"
f"字体宏包:{'、'.join(font_pkgs) if font_pkgs else '未引入'}"
+ (f";纸张 {tex['paper']}" if tex.get("paper") else "")
)
if fs is None:
return _chk("fonts", "字体字号", "warn", detail + "\n建议:\n \\documentclass[11pt]{article}", "补充 \\documentclass 声明")
if fs < min_font:
# LaTeX 粒度粗 → 按步骤4 规格降为 warn(非 fail)
return _chk(
"fonts", "字体字号", "warn", detail + f"\n建议:\n \\documentclass[11pt]{{{cls}}}",
f"正文字号低于下限 {min_font:.0f}pt,请改用 11pt/12pt 选项",
)
return _chk("fonts", "字体字号", "pass", detail)
size_w: dict[float, int] = profile.get("size_weights") or {}
font_w: dict[str, int] = profile.get("font_weights") or {}
if not size_w:
return _chk("fonts", "字体字号", "warn", "未能提取字体信息", "确认文档含可选中文本(扫描件无法检查字体)")
min_font = float(rules.get("min_font_pt", 9))
max_fonts = int(rules.get("max_fonts", 6))
total = sum(size_w.values())
body_pt = max(size_w, key=size_w.get)
body_ratio = int(size_w[body_pt] / total * 100)
distinct = len(font_w)
top_fonts = sorted(font_w.items(), key=lambda kv: -kv[1])[:3]
top_text = "、".join(f"{k}({v}字)" for k, v in top_fonts)
detail = f"主要字号 {body_pt}pt(约 {body_ratio}%),共 {distinct} 种字体;高频字体:{top_text}"
if body_pt < min_font:
return _chk(
"fonts", "字体字号", "fail", detail,
f"将正文字号提高至 ≥{min_font}pt(当前主字号 {body_pt}pt)",
)
if distinct > max_fonts:
return _chk(
"fonts", "字体字号", "warn", detail,
f"字体种类偏多({distinct} > {max_fonts}),建议全文统一为 1–2 种正文字体",
)
return _chk("fonts", "字体字号", "pass", detail)
def _check_margins(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
limit = float(rules.get("min_margin_mm", 15))
margins = profile.get("margins_mm")
# LaTeX:页边距来自 geometry;低于下限按步骤4 规格给 warn(非 fail),附建议片段
if profile.get("source_kind") == "tex":
if not margins:
return _chk(
"margins", "页边距", "warn",
"未通过 geometry 设置页边距(按类默认评估)"
+ f"\n建议:\n \\usepackage[margin=25mm]{{geometry}}",
"在导言区显式声明页边距",
)
lo = min(margins.values())
detail = f"geometry 页边距最小 {lo:.0f}mm(" + "、".join(f"{k}{v:.0f}" for k, v in margins.items()) + f"),规则下限 {limit:.0f}mm"
if lo < limit:
return _chk(
"margins", "页边距", "warn",
detail + f"\n建议:\n \\usepackage[margin={limit:.0f}mm]{{geometry}}",
f"调大页边距至 ≥{limit:.0f}mm",
)
return _chk("margins", "页边距", "pass", detail)
if not margins:
return _chk("margins", "页边距", "warn", "未能提取页边距信息", "检查文档是否含可定位文本")
vals = [margins.get(k) for k in ("top", "right", "bottom", "left")]
vals = [v for v in vals if v is not None]
if not vals:
return _chk("margins", "页边距", "warn", "未能提取页边距信息", "检查文档是否含可定位文本")
t, r = margins.get("top"), margins.get("right")
b, l = margins.get("bottom"), margins.get("left")
lo = min(vals)
sides = "、".join(
f"{name}{v:.0f}" for name, v in (("上", t), ("右", r), ("下", b), ("左", l)) if v is not None
)
detail = f"四边最小 {lo:.0f}mm({sides}),规则下限 {limit:.0f}mm"
if lo >= limit:
return _chk("margins", "页边距", "pass", detail)
if lo >= limit * 0.8:
return _chk("margins", "页边距", "warn", detail, f"将页边距调大至 ≥{limit:.0f}mm")
return _chk("margins", "页边距", "fail", detail, f"页边距过小,请调至 ≥{limit:.0f}mm(版式设置)")
def _check_spacing(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
lo = float(rules.get("min_line_spacing", 1.15))
ratio = profile.get("line_spacing_median")
# LaTeX:\linespread/\setstretch 显式设置值;低于下限按步骤4 规格直接 fail
if profile.get("source_kind") == "tex":
if ratio is None:
return _chk(
"spacing", "行距", "pass",
"未显式设置 \\linespread/\\setstretch,LaTeX 类默认约 1.2 倍(如 10pt→12pt baselineskip)",
)
detail = f"\\linespread/\\setstretch = {ratio:.2f},规则下限 {lo:.2f}"
if ratio < lo:
return _chk(
"spacing", "行距", "fail",
detail + f"\n建议:\n \\linespread{{{lo:.2f}}} % 或 \\setstretch{{{lo:.2f}}}(导言区)",
"在导言区调大行距设置",
)
return _chk("spacing", "行距", "pass", detail)
if ratio is None:
return _chk("spacing", "行距", "warn", "未能提取行距信息", "确认段落使用固定/多倍行距设置")
lo = float(rules.get("min_line_spacing", 1.15))
detail = f"正文中位行距约 {ratio:.2f} 倍,规则下限 {lo:.2f} 倍"
if ratio >= lo:
return _chk("spacing", "行距", "pass", detail)
if ratio >= lo * 0.9:
return _chk("spacing", "行距", "warn", detail, f"段落行距设为 ≥{lo:.2f} 倍(如 1.5 倍行距)")
return _chk("spacing", "行距", "fail", detail, f"行距过密,请设为 ≥{lo:.2f} 倍行距")
def _check_figures(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
figures = int(profile.get("figures") or 0)
tables = int(profile.get("tables") or 0)
min_f = int(rules.get("min_figures", 1))
detail = f"检测到图 {figures} 张、表 {tables} 个(图注/表注正则 + 内嵌对象启发式)"
if figures >= min_f:
return _chk("figures", "图表数量", "pass", detail)
# 启发式检测不可靠 → 低于阈值只 warn 不 fail
return _chk(
"figures", "图表数量", "warn", detail,
f"常规研究论文建议 ≥{min_f} 图;若为综述/短文可忽略(本项为启发式检测)",
)
# 仅"明确不符"才 fail 的 (expected, detected) 组合(宽容化:其余含糊组合一律 warn)
_REF_FAIL_PAIRS = {
("apa", "gb7714"), # 找到 [J] 标引但规则要求 APA
("gb7714", "author_year"), # 无 [J] 标引的作者-年份制,但规则要求 GB/T 7714
("apa", "numbered"), # 编号制引文但规则要求 APA
}
def _check_refs(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
expected = str(rules.get("refs_style", "apa")).lower()
exp_label = _STYLE_LABEL.get(expected, expected)
if not profile.get("refs_found"):
return _chk(
"refs", "参考文献格式", "warn",
"未找到参考文献节(References/参考文献 / \\bibliography / thebibliography)",
"确认存在参考文献声明后重查",
)
if profile.get("source_kind") == "tex":
tex = profile.get("tex") or {}
# BibTeX:\bibliography 存在但缺 \bibliographystyle → warn(样式引擎未知)
if tex.get("bib_bibtex") and not tex.get("bib_style"):
return _chk(
"refs", "参考文献格式", "warn",
f"规则要求 {exp_label};检测到 \\bibliography 但缺少 \\bibliographystyle(BibTeX 无法确定引用样式)"
+ "\n建议:\n \\bibliographystyle{plain} % 按期刊/模板指定实际样式",
"在 \\bibliography 前补 \\bibliographystyle",
)
if not profile.get("refs_text"):
# 条目在 .bbl/编译产物,源文件无文本 → 宽容化只 warn
return _chk(
"refs", "参考文献格式", "warn",
f"规则要求 {exp_label};检测到参考文献声明(条目由编译生成,源文件无条目文本)",
"编译后核对参考文献条目样式",
)
detected, sample = detect_refs_style(profile.get("refs_text"))
det_label = _STYLE_LABEL[detected]
detail = f"规则要求 {exp_label};检测到 {det_label};样例:{sample}"
# 宽容化:仅"明确不符"(_REF_FAIL_PAIRS)才 fail;含糊/不确定一律 warn
if (expected, detected) in _REF_FAIL_PAIRS:
return _chk("refs", "参考文献格式", "fail", detail, f"按 {exp_label} 调整全部文献条目(可用文献管理工具一键转换)")
if detected == "unknown":
return _chk("refs", "参考文献格式", "warn", detail, f"请人工核对是否符合 {exp_label}")
if expected == "gb7714" and detected == "numbered":
return _chk("refs", "参考文献格式", "warn", detail, "GB/T 7714 顺序编码制也用编号,需人工确认条目含 [J] 等类型标引")
if expected == "author-year-journal" and detected in ("gb7714", "numbered"):
return _chk("refs", "参考文献格式", "warn", detail, f"请对照期刊投稿指南核对是否符合 {exp_label}")
return _chk("refs", "参考文献格式", "pass", detail)
def _check_font_packages(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
"""LaTeX 特有:多个相互覆盖的字体宏包(times + mathptmx 类)→ warn。"""
pkgs = set((profile.get("tex") or {}).get("packages") or [])
clash = sorted(pkgs & _TEX_FONT_PKGS)
if len(clash) >= 2:
detail = (
f"检测到多个相互覆盖的字体宏包:{'、'.join(clash)}(后加载者覆盖前者,易产生冲突)"
+ "\n建议:\n %只保留一套字体方案,例如:\n \\usepackage{newtx} % newtxtext + newtxmath 一体"
)
return _chk("fontpkgs", "字体包兼容", "warn", detail, "移除重复/冲突的字体宏包,只保留一套")
return _chk(
"fontpkgs", "字体包兼容", "pass",
"字体宏包:" + ("、".join(clash) if clash else "未引入冲突类字体包"),
)
def _check_anonymous(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
"""LaTeX 特有且规则集 anonymous:true 时才出:源文件需含匿名处理命令。"""
tex = profile.get("tex") or {}
cmd = tex.get("anon_cmd")
if cmd:
return _chk("anonymous", "匿名评审", "pass", f"检测到匿名处理命令 {cmd}")
return _chk(
"anonymous", "匿名评审", "warn",
"规则要求匿名评审,源文件未检测到匿名处理命令"
+ "\n建议:\n %隐去作者/致谢/自引,或定义匿名宏:\n \\newcommand{\\anon}[1]{[ANON]}",
"按期刊要求对作者信息做匿名处理",
)
def build_checks(profile: dict[str, Any], rules: dict[str, Any]) -> list[dict[str, str]]:
checks = [
_check_pages(profile, rules),
_check_fonts(profile, rules),
_check_margins(profile, rules),
_check_spacing(profile, rules),
]
if profile.get("source_kind") == "tex":
checks.append(_check_font_packages(profile, rules))
checks.append(_check_figures(profile, rules))
checks.append(_check_refs(profile, rules))
if profile.get("source_kind") == "tex" and rules.get("anonymous"):
checks.append(_check_anonymous(profile, rules))
return checks
# ---------------------------------------------------------------- 入口
def check_document(data: bytes, kind: str, target_journal: str) -> dict[str, Any]:
"""kind ∈ pdf|docx|tex。返回 {checks, summary, meta}(filename 由路由器补入 meta)。"""
if kind == "pdf":
profile = extract_profile_pdf(data)
elif kind == "docx":
profile = extract_profile_docx(data)
elif kind == "tex":
profile = extract_tex_profile(data)
else:
raise FormatCheckError(f"不支持的文件类型:{kind}")
rules, ruleset_meta = load_ruleset(target_journal)
checks = build_checks(profile, rules)
summary = {"pass": 0, "warn": 0, "fail": 0}
for c in checks:
summary[c["status"]] = summary.get(c["status"], 0) + 1
detected, _ = detect_refs_style(profile.get("refs_text")) if profile.get("refs_found") else ("unknown", "")
meta = {
**ruleset_meta,
"file_type": kind,
"pages": profile.get("pages"),
"refs_style_expected": str(rules.get("refs_style", "apa")).lower(),
"refs_style_detected": detected,
}
if kind == "tex":
tex = profile.get("tex") or {}
meta["tex"] = {
"class": tex.get("class"),
"fontsize_pt": tex.get("fontsize_pt"),
"paper": tex.get("paper"),
"packages": tex.get("packages"),
"cites": tex.get("cites"),
"macros": (tex.get("macros") or [])[:10],
}
return {"checks": checks, "summary": summary, "meta": meta}
@@ -0,0 +1,232 @@
"""services/format_extractor.py —— 投稿指南 → 格式规则 JSON(LLM 抽取,第一批)。
- 调 services/llm.py 的 complete_json(DeepSeek json_object + 修复重试,零新依赖)
- Prompt = 「格式规则抽取器」模板:目标字段与 journals.json format_rules_default
同构(max_pages / min_font_pt / min_margin_mm / min_line_spacing / min_figures /
max_fonts / refs_style / anonymous),强制每字段带 source_quotes + confidence
- 诚实原则:正文没写 → null,绝不编造(与 format_check 诚实原则一致)
- 解析/校验失败 → {"ok": False, "error": ...}
返回:
成功 {"ok": True, "rules": {...}, "source_quotes": {...},
"confidence": {...}, "notes": str, "raw": LLM原始JSON, "error": None}
失败 {"ok": False, "error": "...", 其余键为 None}
source_url 由调用方(scripts/fetch_format_rules.py)在落盘时补入结果文件。
"""
from __future__ import annotations
import logging
import re
from typing import Any
from .llm import LLMError, complete_json
logger = logging.getLogger(__name__)
# 「格式规则抽取器」模板(system prompt)
_SYSTEM = """你是「格式规则抽取器」:从期刊官方投稿指南的 Markdown 正文中抽取格式规则,供投稿排版检查引擎使用。
# 目标字段(rules 对象,严格用以下字段名)
- max_pages:正文页数上限(整数;原文无明确上限 → null)
- min_font_pt:最小字号(数字,单位 pt;原文若用 cm/mm/inch 给出则换算为 pt)
- min_margin_mm:最小页边距(数字,单位 mm;原文若用 inch 给出则 ×25.4 换算)
- min_line_spacing:最小行距倍数(数字;如 "double-spaced" → 2.0,"1.5 spacing" → 1.5)
- min_figures:研究论文最少图数(整数;无明确要求 → null)
- max_fonts:全文最多字体种类(整数;无明确要求 → null)
- refs_style:参考文献样式,枚举 apa | gb7714 | author_year | numbered | vancouver | mla | chicago | other(无法判断 → null)
- anonymous:是否要求匿名/双盲评审(true / false / null)
# 凭证(强制)
- 每个非 null 字段必须给 source_quotes[字段名]:从正文【逐字】摘录的 1-3 句原文,
不改写、不翻译、不丢掉含关键数字的句子。
- source_quotes["_general"]:整体最有凭证价值的 1-3 条原文摘录。
- confidence:{"overall": 0-1, "per_field": {"字段名": 0-1}}。
直接原文命中 ≥0.9;单位换算 0.6-0.9;靠上下文推断 <0.6。
# 诚实原则
- 正文没写到的字段一律 null,绝不用常识或训练记忆补值。
- 单位换算允许做,但该字段 confidence 不得超过 0.9。
- 换算过程、拿不准的点、页面疑似不完整,写进 notes。
# 输出
只输出一个 JSON 对象,不要代码块、不要解释文字:
{"rules": {"max_pages": null, "min_font_pt": null, "min_margin_mm": null,
"min_line_spacing": null, "min_figures": null, "max_fonts": null,
"refs_style": null, "anonymous": null},
"source_quotes": {"字段名": ["原文1", "..."], "_general": ["原文1", "..."]},
"confidence": {"overall": 0.0, "per_field": {"字段名": 0.0}},
"notes": "一句话说明抽取依据与不确定点"}"""
_USER_TMPL = """期刊:{journal_name}
=== 官方投稿指南正文(Markdown,已截断) ===
{markdown}"""
_RULE_INT_KEYS = ("max_pages", "min_figures", "max_fonts")
_REFS_ALLOWED = {"apa", "gb7714", "author_year", "numbered", "vancouver",
"mla", "chicago", "other"}
_REFS_ALIAS = {
"apa-7": "apa", "apa7": "apa", "apa 7th": "apa", "apa_7th": "apa",
"harvard": "author_year", "author-year": "author_year", "authoryear": "author_year",
"gb/t 7714": "gb7714", "gb/t-7714": "gb7714", "gb7714-2015": "gb7714",
"gb/t7714": "gb7714", "chicago author-date": "chicago",
}
# 字符串值带显式单位时的目标单位换算表(LLM 若回传原文单位的兜底;无单位=数值即目标单位)
_UNIT_TO_MM = {"mm": 1.0, "cm": 10.0, "in": 25.4, "inch": 25.4, "inches": 25.4, "pt": 25.4 / 72}
_UNIT_TO_PT = {"pt": 1.0, "mm": 72 / 25.4, "cm": 72 / 2.54, "in": 72.0, "inch": 72.0, "inches": 72.0}
def _to_int(v: Any) -> int | None:
if isinstance(v, bool) or v is None:
return None
if isinstance(v, int):
return v
if isinstance(v, float):
return int(round(v))
if isinstance(v, str):
m = re.search(r"-?\d+", v)
if m:
return int(m.group(0))
return None
def _to_float(v: Any, units: dict[str, float] | None = None) -> float | None:
if isinstance(v, bool) or v is None:
return None
if isinstance(v, (int, float)):
return float(v)
if isinstance(v, str):
m = re.search(r"(-?\d+(?:\.\d+)?)\s*([a-zA-Z]+)?", v)
if not m:
return None
x = float(m.group(1))
unit = (m.group(2) or "").lower()
if units and unit:
factor = units.get(unit)
if factor is None:
return None # 未知单位不猜
x *= factor
return x
return None
def _to_bool(v: Any) -> bool | None:
if isinstance(v, bool):
return v
if isinstance(v, (int, float)) and not isinstance(v, bool):
if v in (0, 1):
return bool(v)
if isinstance(v, str):
low = v.strip().lower()
if low in ("true", "yes", "required"):
return True
if low in ("false", "no", "not required"):
return False
return None
def _norm_refs(v: Any) -> str | None:
if not isinstance(v, str) or not v.strip():
return None
low = v.strip().lower()
low = _REFS_ALIAS.get(low, low)
return low if low in _REFS_ALLOWED else "other"
def _norm_conf(v: Any) -> float | None:
if isinstance(v, bool) or not isinstance(v, (int, float)):
return None
return max(0.0, min(1.0, float(v)))
def _normalize(raw: dict[str, Any]) -> dict[str, Any]:
"""白名单归一:类型错的值置 null(宁缺勿错,raw 全量保留在结果里)。"""
src = raw.get("rules")
src = src if isinstance(src, dict) else {}
rules: dict[str, Any] = {}
for k in _RULE_INT_KEYS:
rules[k] = _to_int(src.get(k))
rules["min_font_pt"] = _to_float(src.get("min_font_pt"), _UNIT_TO_PT)
rules["min_margin_mm"] = _to_float(src.get("min_margin_mm"), _UNIT_TO_MM)
rules["min_line_spacing"] = _to_float(src.get("min_line_spacing"))
rules["refs_style"] = _norm_refs(src.get("refs_style"))
rules["anonymous"] = _to_bool(src.get("anonymous"))
sq_src = raw.get("source_quotes")
sq_src = sq_src if isinstance(sq_src, dict) else {}
quotes: dict[str, list[str]] = {}
for key, val in sq_src.items():
if key != "_general" and key not in rules:
continue
items = val if isinstance(val, list) else [val]
cleaned = [str(x).strip() for x in items if isinstance(x, (str, int, float))
and str(x).strip()]
if cleaned:
quotes[key] = cleaned
conf_src = raw.get("confidence")
conf_src = conf_src if isinstance(conf_src, dict) else {}
pf_src = conf_src.get("per_field")
pf_src = pf_src if isinstance(pf_src, dict) else {}
per_field: dict[str, float] = {}
for k, v in pf_src.items():
c = _norm_conf(v)
if c is not None and (k in rules or k == "_general"):
per_field[k] = c
overall = _norm_conf(conf_src.get("overall"))
if overall is None and per_field:
overall = sum(per_field.values()) / len(per_field)
confidence = {"overall": overall, "per_field": per_field}
notes = raw.get("notes")
return {
"rules": rules,
"source_quotes": quotes,
"confidence": confidence,
"notes": str(notes) if notes else "",
}
def _validation_error(norm: dict[str, Any]) -> str | None:
"""强制凭证:任一非 null 字段缺 source_quotes → 拒收(ok=False)。"""
missing = [k for k, v in norm["rules"].items()
if v is not None and k not in norm["source_quotes"]]
if missing:
return f"LLM 输出缺少非 null 字段的 source_quotes:{', '.join(missing)}"
if not norm["source_quotes"].get("_general"):
return "LLM 输出缺少 source_quotes._general 整体引证"
if any(v is not None for v in norm["rules"].values()):
return None
return "LLM 未抽出任何非 null 字段(rules 全空)"
async def extract_format_rules(markdown: str, journal_name: str) -> dict:
"""markdown(≤30000 字符)+ 期刊名 → 抽取结果(schema 见模块 docstring)。"""
if not markdown or not markdown.strip():
return {"ok": False, "error": "markdown 为空", "rules": None,
"source_quotes": None, "confidence": None, "notes": None, "raw": None}
user = _USER_TMPL.format(journal_name=journal_name, markdown=markdown)
try:
raw = await complete_json(_SYSTEM, user, temperature=0.1, max_tokens=8000)
except LLMError as exc:
return {"ok": False, "error": f"LLM 调用失败:{exc}", "rules": None,
"source_quotes": None, "confidence": None, "notes": None, "raw": None}
if not isinstance(raw, dict):
return {"ok": False, "error": f"LLM 输出非 JSON 对象:{type(raw).__name__}",
"rules": None, "source_quotes": None, "confidence": None,
"notes": None, "raw": None}
norm = _normalize(raw)
err = _validation_error(norm)
if err:
logger.warning("[extract] %s 校验失败:%s", journal_name, err)
return {"ok": False, "error": err, "rules": None,
"source_quotes": norm["source_quotes"], "confidence": norm["confidence"],
"notes": norm["notes"], "raw": raw}
filled = sum(1 for v in norm["rules"].values() if v is not None)
logger.info("[extract] %s → 8 字段中 %d 个非 null", journal_name, filled)
return {"ok": True, "error": None, "rules": norm["rules"],
"source_quotes": norm["source_quotes"],
"confidence": norm["confidence"], "notes": norm["notes"], "raw": raw}
@@ -0,0 +1,356 @@
"""services/format_fetcher.py —— 投稿指南页面抓取(格式规则自动获取·第一批)。
约束(用户拍板 2026-10-10):
- 零新依赖:httpx + beautifulsoup4(venv 已装 4.15.0,用户明确指定使用;
lxml 6.1.2 在 requirements.lock 内,作 bs4 解析器)
- 同域限速 ≥ 2s(robots.txt 与正文请求共用计时器)
- 抓正文前先 GET /robots.txt,命中 Disallow → ok=False 跳过
- UA: "PlantAI-RAG-Bot/1.0 (+contact: admin@example.com)"
- 只提取正文(去导航/页脚/脚本),截断至 30000 字符
返回固定四键:{"ok": bool, "markdown": str, "error": str | None, "fetched_at": str}
"""
from __future__ import annotations
import asyncio
import logging
import re
import time
from datetime import datetime, timezone
from urllib.parse import urlparse
import httpx
from bs4 import BeautifulSoup, NavigableString
logger = logging.getLogger(__name__)
USER_AGENT = "PlantAI-RAG-Bot/1.0 (+contact: admin@example.com)"
TIMEOUT = 30.0
MAX_CHARS = 30_000
MIN_INTERVAL_PER_HOST = 2.0 # 同域请求间隔 ≥ 2s(robots 与正文共用)
_MIN_TEXT_CHARS = 200 # 正文短于此 → 疑似 JS 渲染/反爬页
_HEADERS = {
"User-Agent": USER_AGENT,
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
}
# 同域限速:host → 上次请求完成时刻(monotonic)
_last_hit: dict[str, float] = {}
_throttle_lock = asyncio.Lock()
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat(timespec="seconds")
def _fail(error: str) -> dict:
return {"ok": False, "markdown": "", "error": error, "fetched_at": _now_iso()}
async def _throttle(host: str) -> None:
"""同域串行 + ≥2s 间隔(全局锁:本批抓取量小,简单可靠优先)。"""
async with _throttle_lock:
now = time.monotonic()
wait = _last_hit.get(host, now) + MIN_INTERVAL_PER_HOST - now
if wait > 0:
await asyncio.sleep(wait)
_last_hit[host] = time.monotonic()
# ---------------------------------------------------------------- robots.txt
def _parse_robots(text: str) -> list[tuple[list[str], list[tuple[str, str]]]]:
"""→ [(agents, [(kind, pattern)...]), ...](连续 UA 行同组;规则行后出现 UA=新组)。"""
groups: list[tuple[list[str], list[tuple[str, str]]]] = []
agents: list[str] = []
rules: list[tuple[str, str]] = []
for raw in text.splitlines():
line = raw.split("#", 1)[0].strip()
if not line or ":" not in line:
continue
key, _, value = line.partition(":")
key, value = key.strip().lower(), value.strip()
if key == "user-agent":
if agents and rules: # 上一组已出规则 → 当前 UA 行开新组
groups.append((agents, rules))
agents, rules = [], []
agents.append(value)
elif key in ("allow", "disallow"):
if not agents: # 无前置 UA 的规则按 * 组处理
agents = ["*"]
rules.append((key, value))
if agents:
groups.append((agents, rules))
return groups
def _robots_path_allowed(groups, path: str) -> tuple[bool, str]:
"""最长匹配决胜(同长 Allow 优先;`$` 结尾=精确匹配;空 Disallow 跳过)。"""
def matched_by(pat: str) -> bool:
if pat.endswith("$"):
return path == pat[:-1]
return path.startswith(pat)
token = "plantai-rag-bot"
matched = [rules for agents, rules in groups
if any(token in a.lower() for a in agents)]
if not matched:
matched = [rules for agents, rules in groups
if any(a.lower() == "*" for a in agents)]
if not matched:
return True, "robots.txt 无匹配组(本 bot 不受限)"
best_len, best_allow, best_pat = -1, False, ""
for rules in matched:
for kind, pat in rules:
if not pat: # "Disallow:" 空值 = 放行一切(该组无需处理)
continue
if not matched_by(pat):
continue
length = len(pat)
allow = kind == "allow"
if length > best_len or (length == best_len and allow and not best_allow):
best_len, best_allow, best_pat = length, allow, pat
if best_len < 0:
return True, "robots.txt 允许(无路径匹配)"
if best_allow:
return True, f"robots.txt Allow 命中({best_pat})"
return False, f"robots.txt Disallow 命中({best_pat})"
async def _robots_allows(client: httpx.AsyncClient, url: str) -> tuple[bool, str]:
p = urlparse(url)
robots_url = f"{p.scheme}://{p.netloc}/robots.txt"
await _throttle(p.netloc.lower())
try:
resp = await client.get(robots_url, headers=_HEADERS)
except httpx.HTTPError as exc:
return True, f"robots.txt 获取失败({type(exc).__name__}),按允许处理"
if resp.status_code != 200:
return True, f"robots.txt HTTP {resp.status_code},按允许处理"
groups = _parse_robots(resp.text)
return _robots_path_allowed(groups, p.path or "/")
# ---------------------------------------------------------------- 正文提取
_STRIP_TAGS = ("script", "style", "noscript", "iframe", "form", "nav",
"footer", "aside", "button", "select", "textarea", "input", "svg")
_STRIP_ROLES = ("navigation", "contentinfo", "complementary", "search")
# 正文容器候选(按优先级;首个文本 ≥500 字符者胜出)
_CONTENT_SELECTORS = (
"main", "article", '[role="main"]', "#content", "#main", "#main-content",
".article-body", "#article-body", ".article__body", ".hub-article-body",
".page-content", ".main-content", ".content-area",
)
_CHALLENGE_MARKERS = ("just a moment", "attention required", "captcha",
"access denied", "ddos protection", "verify you are human")
def _decode_html(resp: httpx.Response) -> str:
"""header charset → <meta charset> → utf-8(errors=replace),防 OUP/Wiley 无 header 乱码。"""
enc = None
m = re.search(r"charset=([\w.-]+)", resp.headers.get("content-type", ""), re.I)
if m:
enc = m.group(1)
if not enc:
m = re.search(rb"""charset\s*=\s*["']?([\w.-]+)""", resp.content[:4096], re.I)
if m:
enc = m.group(1).decode("ascii", "ignore")
try:
return resp.content.decode(enc or "utf-8", errors="replace")
except LookupError:
return resp.content.decode("utf-8", errors="replace")
def _squeeze(s: str) -> str:
"""行内空白压缩 + 去空行(保留单换行)。"""
lines = [re.sub(r"[ \t\r\f\v]+", " ", line).strip() for line in s.split("\n")]
return "\n".join(line for line in lines if line)
def _inline(el) -> str:
"""行内渲染:a→md链接 / strong→** / em→* / code→` / br→换行;跳过列表与图片。"""
parts: list[str] = []
for child in el.children:
name = getattr(child, "name", None)
if name is None: # NavigableString
parts.append(str(child))
continue
if name in ("ul", "ol", "img", "picture", "figure"):
continue
if name == "a":
href = (child.get("href") or "").strip()
txt = _inline(child).strip()
if not txt:
continue
parts.append(f"[{txt}]({href})" if href.startswith(("http://", "https://")) else txt)
elif name in ("strong", "b"):
parts.append(f"**{_inline(child)}**")
elif name in ("em", "i"):
parts.append(f"*{_inline(child)}*")
elif name == "code":
parts.append(f"`{_inline(child).strip()}`")
elif name == "br":
parts.append("\n")
else:
parts.append(_inline(child))
return "".join(parts)
def _list_md(el, ordered: bool) -> str:
lines: list[str] = []
idx = 1
for li in el.find_all("li", recursive=False):
# li 文本不含嵌套列表(嵌套列表随后缩进输出)
buf = []
for child in li.children:
if getattr(child, "name", None) in ("ul", "ol"):
continue
buf.append(_inline(child) if getattr(child, "name", None) else str(child))
txt = " ".join("".join(buf).split())
if txt:
lines.append(f"{idx}. {txt}" if ordered else f"- {txt}")
idx += 1
for nested in li.find_all(("ul", "ol"), recursive=False):
sub = _list_md(nested, nested.name == "ol")
if sub:
lines.extend(" " + line for line in sub.split("\n"))
return "\n".join(lines)
def _table_md(table) -> str | None:
rows: list[list[str]] = []
for tr in table.find_all("tr"):
cells = [" ".join(_squeeze(_inline(c)).split())
for c in tr.find_all(("td", "th"))]
if cells:
rows.append(cells)
if not rows:
return None
out = ["| " + " | ".join(rows[0]) + " |"]
if table.find("th"): # 首行含表头 → 补分隔行(合法 GFM 表)
out.append("| " + " | ".join(["---"] * len(rows[0])) + " |")
out += ["| " + " | ".join(r) + " |" for r in rows[1:]]
return "\n".join(out)
def _blocks(el, out: list[str]) -> None:
for child in el.children:
name = getattr(child, "name", None)
if name is None: # 裸文本
txt = _squeeze(str(child))
if txt:
out.append(txt)
continue
if name in _STRIP_TAGS:
continue
if name in ("h1", "h2", "h3", "h4", "h5", "h6"):
txt = " ".join(_squeeze(_inline(child)).split())
if txt:
out.append("#" * int(name[1]) + " " + txt)
elif name == "p":
txt = _squeeze(_inline(child))
if txt:
out.append(txt)
elif name in ("ul", "ol"):
md = _list_md(child, name == "ol")
if md:
out.append(md)
elif name == "table":
md = _table_md(child)
if md:
out.append(md)
elif name == "blockquote":
txt = _squeeze(_inline(child))
if txt:
out.append("\n".join("> " + line for line in txt.split("\n")))
elif name == "pre":
code = child.get_text().strip("\n")
if code.strip():
out.append("```\n" + code + "\n```")
elif name == "hr":
out.append("---")
else: # div/section/article/main/figure/dl/details/… 容器递归
_blocks(child, out)
def _html_to_markdown(html: str) -> tuple[str, str | None]:
"""→ (markdown, error)。选正文容器 → 去壳 → DOM 递归转 Markdown。"""
soup = BeautifulSoup(html, "lxml") # lxml 在 lock 内;缺时 bs4 自动回退 html.parser
title = (soup.title.get_text() if soup.title else "") or ""
for tag in soup.find_all(_STRIP_TAGS):
tag.decompose()
for tag in soup.find_all(attrs={"role": _STRIP_ROLES}):
tag.decompose()
container = None
fallback_hits = []
for sel in _CONTENT_SELECTORS:
for node in soup.select(sel):
text_len = len(node.get_text(" ", strip=True))
fallback_hits.append((text_len, node))
if text_len >= 500:
container = node
break
if container is not None:
break
if container is None:
if fallback_hits:
container = max(fallback_hits, key=lambda x: x[0])[1]
else:
container = soup.body or soup
blocks: list[str] = []
_blocks(container, blocks)
md = "\n\n".join(blocks)
probe = (title + " " + md[:500]).lower()
marker = next((m for m in _CHALLENGE_MARKERS if m in probe), None)
if marker:
return "", f"疑似反爬/挑战页(命中 {marker!r},title={title[:80]!r})"
if len(md) < _MIN_TEXT_CHARS:
return "", f"正文提取过短({len(md)} < {_MIN_TEXT_CHARS} 字符),疑似 JS 渲染或空壳页"
if len(md) > MAX_CHARS:
md = md[:MAX_CHARS - 1] + "…"
return md, None
# ---------------------------------------------------------------- 入口
async def fetch_submission_guidelines(url: str) -> dict:
"""抓投稿指南页面 → 正文 Markdown。
流程:robots.txt 检查(≥2s 限速)→ 禁止即跳过 → GET 正文(≥2s 限速)
→ 状态/类型校验 → 正文提取 → 截断 30000 字符。
"""
parsed = urlparse(url)
if parsed.scheme not in ("http", "https") or not parsed.netloc:
return _fail(f"非法 URL:{url!r}")
host = parsed.netloc.lower()
async with httpx.AsyncClient(follow_redirects=True, timeout=TIMEOUT) as client:
allowed, robots_reason = await _robots_allows(client, url)
if not allowed:
logger.info("[fetch] robots 禁止,跳过 %s(%s)", url, robots_reason)
return _fail(robots_reason)
await _throttle(host)
try:
resp = await client.get(url, headers=_HEADERS)
except httpx.HTTPError as exc:
return _fail(f"请求失败({type(exc).__name__}):{exc}")
if resp.status_code != 200:
return _fail(f"HTTP {resp.status_code}")
ctype = resp.headers.get("content-type", "")
if "html" not in ctype and "xhtml" not in ctype:
return _fail(f"非 HTML 响应:{ctype or '无 Content-Type'}")
markdown, err = _html_to_markdown(_decode_html(resp))
if err:
return _fail(f"{err}(robots: {robots_reason})")
logger.info("[fetch] %s → %d 字符(robots: %s)", url, len(markdown), robots_reason)
return {"ok": True, "markdown": markdown, "error": None, "fetched_at": _now_iso()}
+223
View File
@@ -0,0 +1,223 @@
"""services/journal_kb.py —— 期刊 scope_desc → vstore 入库 + 向量匹配(只读消费 8094)。
链路(全部复用 vstore 既有 API,零改动):
1. ensure_scope_docs:各期刊 scope_desc 以 .txt 入库(category=journal_scope,
name=期刊名),document_id 回写 journals.vstore_doc_id(幂等:已有 id 不重灌)。
2. match:GET /api/vstore/search-chunks?q=<摘要截断>&doc_ids=<id 白名单>&top_k=N
→ 按 doc_id 映射回期刊,取该期刊最高 chunk 分数为 similarity。
3. 自愈:白名单命中 0 条 → GET /documents/{id} 探活 → 失效的重灌(vstore 去重
语义下同内容重灌会换 doc_id,需回写)→ 重试一次 → 仍 0 视为语义低相似。
错误语义:连通性/上游异常抛 MatchError(路由降级为全量候选);
返回 ([], False) 表示检索成功但无有效匹配(路由同样降级)。
"""
from __future__ import annotations
import logging
from typing import Any
import httpx
from ..config import get_settings
from ..db import set_vstore_doc_id
logger = logging.getLogger(__name__)
SCOPE_CATEGORY = "journal_scope"
class MatchError(Exception):
"""vstore 向量匹配失败(不可达/响应异常)。"""
def _timeout() -> httpx.Timeout:
t = get_settings().vstore_match_timeout
return httpx.Timeout(connect=5.0, read=t, write=t, pool=5.0)
def _err_body(resp: httpx.Response) -> str:
try:
payload = resp.json()
except ValueError:
return (resp.text or "").strip()[:500] or "(空响应体)"
if isinstance(payload, dict):
err = payload.get("error")
if isinstance(err, dict) and err.get("message"):
return str(err["message"])
if isinstance(err, str):
return err
return str(payload)[:500]
async def _ingest_scope(client: httpx.AsyncClient, base: str, journal: dict[str, Any]) -> str:
"""单本期刊 scope_desc 入库,返回 document_id。"""
content = str(journal.get("scope_desc") or "").strip()
if not content:
raise MatchError(f"期刊 scope_desc 为空:{journal.get('id')}")
jid = str(journal.get("id") or "")
try:
resp = await client.post(
f"{base}/api/vstore/ingest",
files={"file": (f"{jid}.txt", content.encode("utf-8"), "text/plain")},
data={
"name": str(journal.get("name") or jid),
"category": SCOPE_CATEGORY,
"description": "review_service 期刊范围种子(scope_desc 向量匹配)",
},
)
except httpx.HTTPError as exc:
raise MatchError(f"vstore 不可达({base}):{exc}") from exc
if resp.status_code >= 400:
raise MatchError(f"vstore ingest 失败(HTTP {resp.status_code}):{_err_body(resp)}")
try:
payload = resp.json()
except ValueError as exc:
raise MatchError(f"vstore ingest 响应非 JSON:{exc}") from exc
if not isinstance(payload, dict) or not payload.get("ok"):
raise MatchError(
str((payload.get("error") or {}).get("message") if isinstance(payload, dict) else "") or "vstore ingest 失败"
)
doc_id = str((payload.get("item") or {}).get("document_id") or "")
if not doc_id:
raise MatchError("vstore ingest 响应缺少 document_id")
return doc_id
async def ensure_scope_docs(journals: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""补齐缺失的 scope 入库映射(就地回写 vstore_doc_id 并持久到 DB)。"""
settings = get_settings()
missing = [j for j in journals if not j.get("vstore_doc_id")]
if not missing:
return journals
async with httpx.AsyncClient(timeout=_timeout()) as client:
for j in missing:
try:
doc_id = await _ingest_scope(client, settings.vstore_base, j)
except MatchError as exc:
# 单本失败不阻断其余(该期刊将不进白名单,由路由降级兜底)
logger.warning("journal_kb: scope 入库失败(%s):%s", j.get("id"), exc)
continue
j["vstore_doc_id"] = doc_id
await set_vstore_doc_id(str(j["id"]), doc_id)
return journals
async def _search(
client: httpx.AsyncClient, base: str, query: str, doc_ids: list[str]
) -> list[dict[str, Any]]:
"""chunk 级白名单检索。连通/上游错误抛 MatchError。"""
if not doc_ids:
# doc_ids 为空时 vstore 侧等于不过滤 → 会命中用户私有库,必须拦死
raise MatchError("白名单为空,拒绝全库检索")
try:
resp = await client.get(
f"{base}/api/vstore/search-chunks",
params={
"q": query,
"top_k": get_settings().recommend_scope_top_k,
"doc_ids": ",".join(doc_ids),
},
)
except httpx.HTTPError as exc:
raise MatchError(f"vstore search-chunks 不可达:{exc}") from exc
if resp.status_code >= 400:
raise MatchError(f"vstore search-chunks 失败(HTTP {resp.status_code}):{_err_body(resp)}")
try:
payload = resp.json()
except ValueError as exc:
raise MatchError(f"vstore search-chunks 响应非 JSON:{exc}") from exc
if not isinstance(payload, dict) or not payload.get("ok"):
raise MatchError(
str((payload.get("error") or {}).get("message") if isinstance(payload, dict) else "") or "vstore search-chunks 失败"
)
items = payload.get("items")
return items if isinstance(items, list) else []
async def _doc_alive(client: httpx.AsyncClient, base: str, doc_id: str) -> bool:
try:
resp = await client.get(f"{base}/api/vstore/documents/{doc_id}")
except httpx.HTTPError:
return False
if resp.status_code != 200:
return False
try:
payload = resp.json()
except ValueError:
return False
return bool(isinstance(payload, dict) and payload.get("ok"))
async def _refresh_dead_docs(
client: httpx.AsyncClient, base: str, journals: list[dict[str, Any]]
) -> None:
"""白名单 0 命中时探活:失效文档重灌并回写新 id。"""
for j in journals:
doc_id = str(j.get("vstore_doc_id") or "")
if not doc_id or await _doc_alive(client, base, doc_id):
continue
try:
new_id = await _ingest_scope(client, base, j)
except MatchError as exc:
logger.warning("journal_kb: scope 重灌失败(%s):%s", j.get("id"), exc)
continue
j["vstore_doc_id"] = new_id
await set_vstore_doc_id(str(j["id"]), new_id)
async def match(
abstract: str, journals: list[dict[str, Any]]
) -> tuple[list[dict[str, Any]], bool]:
"""摘要 vs scope_desc 向量匹配。
返回 (candidates, vector_ok);candidates 元素:
{"journal": <journals 行>, "similarity": float}
按 similarity 降序。MatchError → 路由降级;([], False) → 语义低相似。
"""
settings = get_settings()
query = abstract.strip()[: settings.recommend_query_chars]
if not query:
return [], False
await ensure_scope_docs(journals)
whitelisted = [j for j in journals if j.get("vstore_doc_id")]
if not whitelisted:
raise MatchError("无任何期刊 scope 入库成功(vstore_doc_id 为空)")
doc_map = {str(j["vstore_doc_id"]): j for j in whitelisted}
timeout = _timeout()
async with httpx.AsyncClient(timeout=timeout) as client:
items = await _search(client, settings.vstore_base, query, list(doc_map))
if not items:
await _refresh_dead_docs(client, settings.vstore_base, whitelisted)
doc_map = {str(j["vstore_doc_id"]): j for j in whitelisted if j.get("vstore_doc_id")}
if not doc_map:
return [], False
items = await _search(client, settings.vstore_base, query, list(doc_map))
# 聚合:每本期刊取最高 chunk 分
best: dict[str, float] = {}
for item in items:
if not isinstance(item, dict):
continue
j = doc_map.get(str(item.get("doc_id") or ""))
if j is None:
continue
try:
score = float(item.get("score"))
except (TypeError, ValueError):
continue
jid = str(j["id"])
if jid not in best or score > best[jid]:
best[jid] = score
if not best:
return [], False
by_id = {str(j["id"]): j for j in journals}
candidates = [
{"journal": by_id[jid], "similarity": score}
for jid, score in best.items()
if jid in by_id
]
candidates.sort(key=lambda c: c["similarity"], reverse=True)
return candidates, bool(candidates)
+210
View File
@@ -0,0 +1,210 @@
"""services/journal_rank.py —— LLM 重排 + 幻觉防护 + 三档归类。
职责:
1. rerank:把带语义分的候选期刊(来自 journal_kb)+ 用户偏好交给 LLM,
要求按 id 白名单返回 {rankings:[{id, tier, match_reason, risk_note}]}。
2. validate:硬校验——白名单外 id 丢弃(幻觉拦截);tier 非法/LLM 缺漏的
期刊按规则补位(绝不因 LLM 输出丢失候选期刊);期刊名称/指标一律不采信
LLM,最终展示全部用 DB 字段(防止改名型幻觉)。
3. fallback_rankings:LLM 不可用时的确定性规则分档(按 impact_factor)。
三档语义:challenge 挑战型(高影响/慢审稿)、fit 适配型(均衡)、
safe 稳妥型(快速/门槛低)。
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from typing import Any
from ..config import get_settings
from . import llm
TIERS = ("challenge", "fit", "safe")
TIER_LABELS = {
"challenge": "挑战型(高影响因子/慢审稿)",
"fit": "适配型(中等影响/周期适中)",
"safe": "稳妥型(快速发表/门槛较低)",
}
# 规则分档阈值(近似值,可后续配置化)
_CHALLENGE_MIN_IF = 9.0
_SAFE_MAX_IF = 4.0
# tier 同义词归一(LLM 偶发中文标签)
_TIER_ALIASES = {
"挑战": "challenge",
"挑战型": "challenge",
"高风险": "challenge",
"适配": "fit",
"适配型": "fit",
"稳妥": "safe",
"稳妥型": "safe",
"低风险": "safe",
}
SYSTEM_PROMPT = """你是学术投稿推荐专家(植物基因组学方向)。根据给定候选期刊、论文摘要与用户偏好,为**每一个**候选期刊输出推荐分档与理由。
硬规则(违反即视为无效输出):
1. id 白名单:只能使用输入 candidates 中出现的 id;绝对禁止添加、替换或臆造列表之外的任何期刊。
2. 每个输入 id 恰好输出一次,不得遗漏、不得重复;期刊名称等元数据由系统填充,你只输出 id。
3. tier 只能取 challenge | fit | safe:
- challenge(挑战型):高影响因子、审稿较慢或竞争激烈的期刊
- fit(适配型):影响力与发表周期均衡、与论文契合的期刊
- safe(稳妥型):审稿较快、门槛较低、命中概率较高的期刊
4. match_reason:中文,1-2 句,结合摘要的研究主题与该刊 scope_desc 及指标(如有 similarity 一并参考),说明为何匹配/不匹配。
5. risk_note:中文,1-2 句,指出投稿风险(超出用户偏好、IF 门槛偏高、范围匹配偏弱、审稿周期长等);无明显风险写「无」。
6. 用户 preferences 为软约束:不满足偏好的期刊仍必须输出,并在 risk_note 中明确说明。
7. 不确定时宁可保守陈述,禁止编造统计数据(录用率、审稿天数等)。
输出:仅一个 JSON 对象,格式:
{"rankings":[{"id":"...","tier":"challenge","match_reason":"...","risk_note":"..."}]}
不要输出任何 JSON 之外的内容。"""
class RankError(Exception):
"""LLM 输出形态异常(无法按 JSON 解析/结构非法)。"""
@dataclass
class Ranking:
journal_id: str
tier: str
match_reason: str
risk_note: str
def rule_tier(journal: dict[str, Any]) -> str:
"""确定性分档规则(LLM 缺漏/不可用时兜底)。"""
inf = journal.get("impact_factor")
weeks = journal.get("review_time_weeks")
if isinstance(inf, (int, float)):
if inf >= _CHALLENGE_MIN_IF:
return "challenge"
if inf <= _SAFE_MAX_IF:
return "safe"
elif isinstance(weeks, (int, float)) and weeks <= 4:
return "safe"
return "fit"
def build_user_prompt(
abstract: str,
candidates: list[dict[str, Any]],
preferences: dict[str, Any],
) -> str:
"""候选 + 偏好 + 摘要 → 用户消息(JSON)。"""
payload = {
"abstract": abstract.strip()[:4000],
"preferences": preferences or {},
"candidates": [
{
"id": str(c["journal"]["id"]),
"name": str(c["journal"].get("name") or ""),
"impact_factor": c["journal"].get("impact_factor"),
"review_time_weeks": c["journal"].get("review_time_weeks"),
"acceptance_rate": c["journal"].get("acceptance_rate"),
"similarity": c.get("similarity"),
"scope_desc": str(c["journal"].get("scope_desc") or "")[:400],
}
for c in candidates
],
}
return json.dumps(payload, ensure_ascii=False)
def _normalize_tier(raw: Any) -> str | None:
tier = str(raw or "").strip().lower()
tier = _TIER_ALIASES.get(str(raw or "").strip(), tier)
return tier if tier in TIERS else None
def validate_rankings(
payload: Any, candidates: list[dict[str, Any]]
) -> tuple[list[Ranking], int, int]:
"""校验 LLM 输出。
返回 (rankings, dropped_hallucinated, repaired):
- 白名单外 id / 重复 id → 丢弃并计数(repaired 中重复不计);
- tier 非法、LLM 缺漏的 id → 按规则补位(repaired 计数)。
"""
if not isinstance(payload, dict) or not isinstance(payload.get("rankings"), list):
raise RankError("LLM 输出缺少 rankings 数组")
by_id = {str(c["journal"]["id"]): c["journal"] for c in candidates}
input_ids = list(by_id)
seen: set[str] = set()
rankings: list[Ranking] = []
dropped = 0
repaired = 0
for entry in payload["rankings"]:
if not isinstance(entry, dict):
dropped += 1
continue
jid = str(entry.get("id") or "").strip()
if jid in seen:
continue
if jid not in by_id:
dropped += 1 # 幻觉期刊:绝不进输出
continue
seen.add(jid)
tier = _normalize_tier(entry.get("tier"))
if tier is None:
tier = rule_tier(by_id[jid])
repaired += 1
reason = str(entry.get("match_reason") or "").strip() or "基于期刊范围与指标的匹配"
risk = str(entry.get("risk_note") or "").strip() or "无"
rankings.append(Ranking(journal_id=jid, tier=tier, match_reason=reason, risk_note=risk))
for jid in input_ids:
if jid in seen:
continue
rankings.append(
Ranking(
journal_id=jid,
tier=rule_tier(by_id[jid]),
match_reason="LLM 未给出该期刊条目,已按影响因子/周期规则补位",
risk_note="未提供(LLM 输出缺漏,规则兜底)",
)
)
repaired += 1
return rankings, dropped, repaired
async def rerank(
abstract: str,
candidates: list[dict[str, Any]],
preferences: dict[str, Any],
) -> tuple[list[Ranking], int, int]:
"""LLM 重排 + 校验。LLMError / RankError 上抛(路由降级为规则分档)。"""
settings = get_settings()
payload = await llm.complete_json(
SYSTEM_PROMPT,
build_user_prompt(abstract, candidates, preferences),
temperature=0.2,
max_tokens=3000,
timeout=settings.recommend_read_timeout,
)
return validate_rankings(payload, candidates)
def fallback_rankings(candidates: list[dict[str, Any]]) -> list[Ranking]:
"""LLM 不可用时的确定性分档(不丢任何候选期刊)。"""
rankings: list[Ranking] = []
for c in candidates:
j = c["journal"]
inf = j.get("impact_factor")
weeks = j.get("review_time_weeks")
rankings.append(
Ranking(
journal_id=str(j["id"]),
tier=rule_tier(j),
match_reason=(
f"规则分档:影响因子 {inf if inf is not None else '未收录'}、"
f"审稿约 {weeks if weeks is not None else '未收录'} 周"
),
risk_note="LLM 不可用,本条按影响因子/周期规则估算",
)
)
return rankings
+520
View File
@@ -0,0 +1,520 @@
"""app/services/journal_seed.py —— journal_directory 核心导入逻辑(CLI 与启动 seed 共享)。
数据流(schema 仍由本模块独家拥有,app 路由层只读不建表):
openpyxl 非 read_only 加载(读 cell.hyperlink.target 来源链接)
→ 「国际期刊」「中文期刊」两 sheet 按【列名】映射
→ 行级清洗(空→None 绝不填 0;分区数字→"N区";JCR学科→JSON 数组)
→ 按 id 幂等 upsert(ON CONFLICT DO UPDATE,重跑 count 不变)
两个入口:
- scripts/import_journals.py(CLI):extract_rows_from_xlsx + upsert_to_db
+ 对账报告(exit code 语义见脚本 docstring)
- ensure_journal_directory_seeded():服务启动自动 seed(main.py lifespan),
仅当表不存在或为空才导入;任何失败收敛为 dict 返回值,绝不抛异常。
v4 源(2026-10-10):JIF=2025 统计年/2026 JCR 发布;中文 IF=2024 统计年/
年报 2025 版。源 xlsx 路径走 settings.journal_directory_seed_abs
(REVIEW_JOURNAL_DIRECTORY_SEED_PATH 覆盖)。
id 策略:有 ISSN → "{lang}:{ISSN}"; 无 ISSN → "{lang}:{slug(name)}";
同 lang 内冲突 → 抛 JournalSeedError(CLI 中止导入 / startup 返回 error)。
"""
from __future__ import annotations
import json
import re
import sqlite3
import unicodedata
from collections import Counter
from datetime import datetime
from pathlib import Path
from typing import Any
import openpyxl
from ..config import get_settings
SHEET_INTL = "国际期刊"
SHEET_ZH = "中文期刊"
EXPECT_INTL = 779 # 口径 sheet 声明(v4 不变)
EXPECT_ZH = 277
# 官网列(v4 实际列名「官网网址」在首位;其余为历史/备用别名)
OFFICIAL_URL_HEADERS = ("官网网址", "官网", "官方网址", "期刊官网")
# JCR 学科列别名:v4 改名「JCR学科(2024源表)」,v2 为「JCR学科(源表)」
JCR_DISCIPLINE_HEADERS = ("JCR学科(2024源表)", "JCR学科(源表)")
# 既有表缺列时的幂等迁移(缺哪加哪,重复跑不报错)
MIGRATE_COLS = (
("official_url", "TEXT"),
("official_check_status", "TEXT"),
("jcr_release_year", "INTEGER"),
)
SCHEMA_SQL = """
CREATE TABLE IF NOT EXISTS journal_directory (
id TEXT PRIMARY KEY,
lang TEXT NOT NULL,
name TEXT NOT NULL,
research_direction TEXT,
cas_partition TEXT,
cas_partition_note TEXT,
cas_partition_year INTEGER,
jif REAL,
jif_year INTEGER,
jif_note TEXT,
jif_check_status TEXT,
composite_impact_factor REAL,
comprehensive_impact_factor REAL,
metric_year INTEGER,
metric_report_year INTEGER,
metric_caliber TEXT,
annual_report_page INTEGER,
top_flag TEXT,
publisher_series TEXT,
journal_mark TEXT,
article_type TEXT,
issn TEXT,
eissn TEXT,
jcr_disciplines TEXT,
supplement_note TEXT,
evidence_note TEXT,
official_url TEXT,
official_check_status TEXT,
jcr_release_year INTEGER,
partition_source TEXT,
jif_source TEXT,
jif_source_url TEXT,
partition_source_url TEXT,
data_source_label TEXT,
data_source_url TEXT,
imported_at TEXT
);
CREATE INDEX IF NOT EXISTS idx_dir_lang_name ON journal_directory(lang, name);
CREATE INDEX IF NOT EXISTS idx_dir_lang_jif ON journal_directory(lang, jif);
CREATE INDEX IF NOT EXISTS idx_dir_lang_cif ON journal_directory(lang, composite_impact_factor);
CREATE INDEX IF NOT EXISTS idx_dir_lang_part ON journal_directory(lang, cas_partition);
CREATE INDEX IF NOT EXISTS idx_dir_lang_domain ON journal_directory(lang, research_direction);
"""
# upsert 列序(除 id 外全部 ON CONFLICT DO UPDATE)
COLS = (
"lang", "name", "research_direction", "cas_partition", "cas_partition_note",
"cas_partition_year", "jif", "jif_year", "jif_note", "jif_check_status",
"composite_impact_factor", "comprehensive_impact_factor", "metric_year",
"metric_report_year", "metric_caliber", "annual_report_page", "top_flag",
"publisher_series", "journal_mark", "article_type", "issn", "eissn",
"jcr_disciplines", "supplement_note", "evidence_note", "official_url",
"official_check_status", "jcr_release_year",
"partition_source", "jif_source", "jif_source_url", "partition_source_url",
"data_source_label", "data_source_url", "imported_at",
)
UPSERT_SQL = (
"INSERT INTO journal_directory (id, " + ", ".join(COLS) + ") VALUES ("
+ ", ".join("?" for _ in range(len(COLS) + 1)) + ") "
+ "ON CONFLICT(id) DO UPDATE SET "
+ ", ".join(f"{c}=excluded.{c}" for c in COLS)
)
class JournalSeedError(Exception):
"""xlsx 缺失/结构错误/口径不符/id 冲突(不写库)。
exit_code 对齐 CLI 语义:1=id 冲突中止;2=行数与口径不符;3=文件/结构错误。
"""
def __init__(
self,
message: str,
*,
exit_code: int = 3,
stage: str = "读取失败",
conflicts: list[str] | None = None,
) -> None:
super().__init__(message)
self.exit_code = exit_code
self.stage = stage
self.conflicts = conflicts or []
def new_stats() -> dict[str, Any]:
"""对账统计骨架(CLI 报告与解析共享)。"""
return {
"read_intl": 0, "read_zh": 0, "success": 0, "skipped": 0, "failed": 0,
"jif_missing": [], "cif_missing": [], "part_blank": [], "part_notfound": [],
"part_dist": Counter(), "hl_jif": 0, "hl_part": 0, "hl_zh": 0,
"id_issn": 0, "id_slug": 0, "official_url_detected": False,
"migrated_cols": [], "migrated_skipped": [],
"official_total": 0, "official_filled": 0, "official_check_filled": 0,
"jcr_filled": 0, "official_missing": [],
}
def migrate(conn: sqlite3.Connection) -> tuple[list[str], list[str]]:
"""幂等加列:PRAGMA 探测现有列,缺哪个 ALTER ADD 哪个。"""
existing = {r[1] for r in conn.execute("PRAGMA table_info(journal_directory)")}
added: list[str] = []
skipped: list[str] = []
for col, decl in MIGRATE_COLS:
if col in existing:
skipped.append(col)
else:
conn.execute(f"ALTER TABLE journal_directory ADD COLUMN {col} {decl}")
added.append(col)
conn.commit()
return added, skipped
# ---- 清洗原语 ----
def text(v: Any) -> str | None:
"""单元格 → strip 后的文本;空/纯空白 → None(绝不填 0/空串)。"""
if v is None:
return None
s = str(v).strip()
return s or None
def to_float(v: Any) -> float | None:
if v is None or (isinstance(v, str) and not v.strip()):
return None
try:
return float(v)
except (TypeError, ValueError):
return None
def to_int(v: Any) -> int | None:
f = to_float(v)
if f is None:
return None
return int(f)
def slugify(name: str) -> str:
"""小写 + 非词字符折叠为连字符;\\w 含 CJK(中文期刊无 ISSN 时的 id 兜底)。"""
s = unicodedata.normalize("NFC", name.strip().casefold())
s = re.sub(r"[^\w]+", "-", s, flags=re.UNICODE)
s = re.sub(r"-{2,}", "-", s)
return s.strip("-")
def make_id(lang: str, name: str, issn: str | None) -> str:
if issn:
return f"{lang}:{issn}"
return f"{lang}:{slugify(name)}"
def parse_partition(raw: Any) -> tuple[str | None, str | None]:
"""分区 → (cas_partition, note)。数字 1-4 → 'N区';'未检得' → note;空白 → 双 None。"""
if raw is None:
return None, None
if isinstance(raw, (int, float)) and not isinstance(raw, bool):
n = int(raw)
if 1 <= n <= 4:
return f"{n}区", None
return None, None
s = str(raw).strip()
if not s:
return None, None
if "未检得" in s:
return None, "未检得"
m = re.fullmatch(r"([1-4])\s*区?", s)
if m:
return f"{m.group(1)}区", None
return s, None # 未知文本:原样保留(不丢信息)
def split_disciplines(raw: Any) -> str | None:
"""'A(SCIE);B(SCIE)' → JSON 数组字符串。"""
s = text(raw)
if not s:
return None
parts = [p.strip() for p in s.split(";") if p.strip()]
return json.dumps(parts, ensure_ascii=False) if parts else None
def hyperlink_target(cell) -> str | None:
"""超链接单元格 → target URL;无链接 → None(需要非 read_only 模式)。"""
try:
target = cell.hyperlink.target if cell.hyperlink is not None else None
except Exception: # noqa: BLE001
return None
return text(target)
# ---- sheet 解析 ----
def header_index(ws) -> dict[str, int]:
headers = [text(c.value) for c in ws[1]]
return {h: i for i, h in enumerate(headers) if h is not None}
def read_intl(ws, imported_at: str, stats: dict) -> list[tuple]:
"""国际期刊 sheet → 行元组(顺序同 COLS 前置 id)。"""
ix = header_index(ws)
for req in ("期刊名称", "研究方向", "中科院分区", "JIF影响因子"):
if req not in ix:
raise JournalSeedError(
f"{SHEET_INTL} 缺列:{req}(实际列:{list(ix)})",
exit_code=3, stage="结构错误",
)
url_col = next((ix[h] for h in OFFICIAL_URL_HEADERS if h in ix), None)
jcr_hdr = next((h for h in JCR_DISCIPLINE_HEADERS if h in ix), None)
stats["official_url_detected"] = url_col is not None
if url_col is not None:
stats["official_url_headers"] = next(h for h in OFFICIAL_URL_HEADERS if h in ix)
rows: list[tuple] = []
for row in ws.iter_rows(min_row=2):
if not any(c.value is not None for c in row):
continue # 尾部幻影空行:不计入 skipped
name = text(row[ix["期刊名称"]].value) if ix["期刊名称"] < len(row) else None
if not name:
stats["skipped"] += 1
continue
issn = text(row[ix["ISSN"]].value) if "ISSN" in ix else None
jif = to_float(row[ix["JIF影响因子"]].value)
check_status = text(row[ix["JIF核对状态"]].value) if "JIF核对状态" in ix else None
official_check = text(row[ix["官网核对状态"]].value) if "官网核对状态" in ix else None
jcr_year = to_int(row[ix["JCR发布年"]].value) if "JCR发布年" in ix else None
part, part_note = parse_partition(row[ix["中科院分区"]].value)
# JIF 空 → note = 核对状态原文("未检得"/"源表标N/A" 等说明)
jif_note = check_status if jif is None else None
jif_url = hyperlink_target(row[ix["JIF数据来源"]]) if "JIF数据来源" in ix else None
part_url = hyperlink_target(row[ix["分区数据来源"]]) if "分区数据来源" in ix else None
official = None
if url_col is not None and url_col < len(row):
official = text(row[url_col].value) or hyperlink_target(row[url_col])
rows.append((
make_id("en", name, issn), "en", name,
text(row[ix["研究方向"]].value),
part, part_note,
to_int(row[ix["分区版本年"]].value) if "分区版本年" in ix else None,
jif,
to_int(row[ix["JIF统计年"]].value) if "JIF统计年" in ix else None,
jif_note, check_status,
None, None, None, None, None, None,
text(row[ix["Top(源表)"]].value) if "Top(源表)" in ix else None,
text(row[ix["出版系列"]].value) if "出版系列" in ix else None,
text(row[ix["期刊标记"]].value) if "期刊标记" in ix else None,
text(row[ix["文章类型"]].value) if "文章类型" in ix else None,
issn,
text(row[ix["eISSN"]].value) if "eISSN" in ix else None,
split_disciplines(row[ix[jcr_hdr]].value) if jcr_hdr else None,
text(row[ix["补充说明"]].value) if "补充说明" in ix else None,
text(row[ix["补充证据"]].value) if "补充证据" in ix else None,
official,
official_check,
jcr_year,
"official", "official",
jif_url, part_url,
text(row[ix["JIF数据来源"]].value) if "JIF数据来源" in ix else None,
None, # data_source_url(国际表来源即 jif/partition 两链接)
imported_at,
))
# 统计
stats["official_total"] += 1
stats["official_filled"] += 1 if official else 0
stats["official_check_filled"] += 1 if official_check else 0
stats["jcr_filled"] += 1 if jcr_year else 0
if not official:
stats["official_missing"].append(("en", name))
if jif is None:
stats["jif_missing"].append((name, check_status or ""))
if part is None and part_note == "未检得":
stats["part_notfound"].append(name)
elif part is None:
stats["part_blank"].append(name)
stats["hl_jif"] += 1 if jif_url else 0
stats["hl_part"] += 1 if part_url else 0
stats["id_issn" if issn else "id_slug"] += 1
return rows
def read_zh(ws, imported_at: str, stats: dict) -> list[tuple]:
"""中文期刊 sheet → 行元组。"""
ix = header_index(ws)
for req in ("中文期刊名称", "复合影响因子", "综合影响因子"):
if req not in ix:
raise JournalSeedError(
f"{SHEET_ZH} 缺列:{req}(实际列:{list(ix)})",
exit_code=3, stage="结构错误",
)
url_col = next((ix[h] for h in OFFICIAL_URL_HEADERS if h in ix), None)
stats["official_url_detected_zh"] = url_col is not None
rows: list[tuple] = []
for row in ws.iter_rows(min_row=2):
if not any(c.value is not None for c in row):
continue # 尾部幻影空行:不计入 skipped
name = text(row[ix["中文期刊名称"]].value)
if not name:
stats["skipped"] += 1
continue
composite = to_float(row[ix["复合影响因子"]].value)
comprehensive = to_float(row[ix["综合影响因子"]].value)
check_status = text(row[ix["核对状态"]].value) if "核对状态" in ix else None
src_label = text(row[ix["数据来源"]].value) if "数据来源" in ix else None
src_url = hyperlink_target(row[ix["数据来源"]]) if "数据来源" in ix else None
official = None
if url_col is not None and url_col < len(row):
official = text(row[url_col].value) or hyperlink_target(row[url_col])
official_check = text(row[ix["官网核对状态"]].value) if "官网核对状态" in ix else None
rows.append((
make_id("zh", name, None), "zh", name,
text(row[ix["研究方向"]].value),
# 中文分区原样(常量"不按此口径列示"),不分区 → partition_source=None
text(row[ix["中科院分区"]].value) if "中科院分区" in ix else None,
None, # cas_partition_note:中文表无"未检得"语义(口径=不按此口径列示)
None, # cas_partition_year
None, None, None, check_status,
composite, comprehensive,
to_int(row[ix["指标统计年"]].value) if "指标统计年" in ix else None,
to_int(row[ix["年报版本年"]].value) if "年报版本年" in ix else None,
text(row[ix["指标口径"]].value) if "指标口径" in ix else None,
to_int(row[ix["年报PDF页码"]].value) if "年报PDF页码" in ix else None,
None, None, None, None, None, None, None,
text(row[ix["备注"]].value) if "备注" in ix else None,
None, # evidence_note(中文表无此列)
official,
official_check,
None, # jcr_release_year:中文表无 JCR 发布年列(前端中文 Tab 不显示)
None, # partition_source:中文不按中科院口径
"official", # jif_source:国内年报(official)
None, None,
src_label, src_url,
imported_at,
))
stats["official_total"] += 1
stats["official_filled"] += 1 if official else 0
stats["official_check_filled"] += 1 if official_check else 0
if not official:
stats["official_missing"].append(("zh", name))
if composite is None:
stats["cif_missing"].append((name, check_status or ""))
stats["hl_zh"] += 1 if src_url else 0
stats["id_slug"] += 1
return rows
# ---- 共享主流程 ----
def extract_rows_from_xlsx(
source: Path,
*,
imported_at: str | None = None,
stats: dict | None = None,
) -> tuple[list[tuple], list[tuple], dict]:
"""xlsx → (rows_en, rows_zh, stats)。口径不符/id 冲突抛 JournalSeedError(不写库)。"""
stats = stats if stats is not None else new_stats()
imported_at = imported_at or datetime.now().isoformat(timespec="seconds")
try:
wb = openpyxl.load_workbook(source, data_only=True)
except FileNotFoundError as exc:
raise JournalSeedError(
f"源文件不存在: {source}", exit_code=3, stage="读取失败"
) from exc
except Exception as exc: # noqa: BLE001
raise JournalSeedError(
f"无法打开 Excel: {exc}", exit_code=3, stage="读取失败"
) from exc
if SHEET_INTL not in wb.sheetnames or SHEET_ZH not in wb.sheetnames:
raise JournalSeedError(
f"缺少 sheet:需要 {SHEET_INTL}/{SHEET_ZH},实际 {wb.sheetnames}",
exit_code=3, stage="结构错误",
)
rows_en = read_intl(wb[SHEET_INTL], imported_at, stats)
rows_zh = read_zh(wb[SHEET_ZH], imported_at, stats)
stats["read_intl"], stats["read_zh"] = len(rows_en), len(rows_zh)
for r in rows_en:
stats["part_dist"][r[4] or ("未检得" if r[5] else "空白")] += 1
# 口径行数校验(不符 → 抛错,不写库)
if len(rows_en) != EXPECT_INTL or len(rows_zh) != EXPECT_ZH:
raise JournalSeedError(
f"行数与口径不符:国际 {len(rows_en)}/{EXPECT_INTL}、"
f"中文 {len(rows_zh)}/{EXPECT_ZH},中止导入",
exit_code=2, stage="行数校验失败",
)
# id 冲突检测(同 lang 内重复 → 抛错,不写库)
ids = [r[0] for r in rows_en + rows_zh]
dup = [i for i, c in Counter(ids).items() if c > 1]
if dup:
conflicts = [f"{i} × {Counter(ids)[i]}" for i in sorted(dup)]
raise JournalSeedError(
f"发现 {len(conflicts)} 个重复 id,中止导入(未写库)",
exit_code=1, stage="id 冲突中止", conflicts=conflicts,
)
return rows_en, rows_zh, stats
def upsert_to_db(db_path: Path, rows_en: list[tuple], rows_zh: list[tuple]) -> dict[str, int]:
"""建表(IF NOT EXISTS)+幂等加列+upsert → {"en","zh","total"}。"""
db_path = Path(db_path)
db_path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(str(db_path))
try:
conn.executescript(SCHEMA_SQL)
migrate(conn)
conn.executemany(UPSERT_SQL, list(rows_en) + list(rows_zh))
conn.commit()
d = dict(conn.execute(
"SELECT lang, COUNT(*) FROM journal_directory GROUP BY lang").fetchall())
return {"en": d.get("en", 0), "zh": d.get("zh", 0), "total": sum(d.values())}
finally:
conn.close()
def ensure_journal_directory_seeded(
db_path: Path | None = None,
source_xlsx: Path | None = None,
) -> dict:
"""启动自动 seed:空表才导入;任何失败收敛为 dict,绝不抛异常。
返回统一四键:seeded/reason/error/rows(另有 counts 仅成功时给出 en/zh 明细)。
- 已有数据(只认空/非空,不做补差)→ seeded=False, reason="already has N rows, skip"
- 空表导入成功 → seeded=True, rows=1056
- xlsx 缺失/结构错误/口径不符/id 冲突 → seeded=False, error="..."
"""
settings = get_settings()
db = Path(db_path) if db_path is not None else settings.db_path_abs
source = Path(source_xlsx) if source_xlsx is not None else settings.journal_directory_seed_abs
try:
if db.exists():
conn = sqlite3.connect(str(db))
try:
row = conn.execute("SELECT COUNT(*) FROM journal_directory").fetchone()
except sqlite3.OperationalError:
row = None # 表不存在 → 视为空
finally:
conn.close()
else:
row = None
if row is not None and int(row[0]) > 0:
n = int(row[0])
return {
"seeded": False, "reason": f"already has {n} rows, skip",
"error": None, "rows": n,
}
rows_en, rows_zh, _ = extract_rows_from_xlsx(source)
counts = upsert_to_db(db, rows_en, rows_zh)
return {
"seeded": True, "reason": None, "error": None,
"rows": counts["total"], "counts": counts,
}
except JournalSeedError as exc:
return {"seeded": False, "reason": None, "error": str(exc), "rows": None}
except Exception as exc: # noqa: BLE001
return {"seeded": False, "reason": None, "error": str(exc), "rows": None}
@@ -0,0 +1,869 @@
"""services/latex_compile.py —— tectonic 编译 .tex → PDF(P3a + P4 项目 zip)。
隔离与安全(对齐设计清单):
1. 一编译一目录:{reformat_dir}/{anchor_id}/compile/(input.tex/output.pdf/output.log/meta.json)
2. 文件名永不落盘:用户文件名只做后缀校验,写入一律 input.tex
3. tectonic --untrusted(官方旗标:禁用已知不安全特性,含 shell escape 路径)
4. 子进程 start_new_session + 超时 killpg(SIGKILL)(整进程组)
5. 单槽队列:asyncio.Lock + pending 计数,超出 compile_queue_limit → QueueFullError
6. 产物 PDF > max_compile_pdf_bytes → 删文件 + OutputTooLargeError
7. 启动/每次编译前机会性清扫 retention_days 天前的 job 目录(只删 12 位 hex 锚点目录)
P4 项目 zip(两段式 inspect + run,与 P3a 共享单槽队列/清扫/错误类):
- inspect_zip:安全解压(zip_safety)→ 主文件识别 → compile_inspect/{job_id}/(1h TTL)
- compile_project_job:原子 rename 消费 inspect → compile_jobs/{cid}/ → tectonic →
finally 删 project 树(只留 output.pdf/output.log/meta.json)
- 失败也保证 output.log 落盘(补充 3);classify_compile_error 给出用户友好文案
- P3a 单文件路径(prepare_input/compile_tex/_run_once)零改动
tectonic 二进制缺失 → ToolUnavailableError(路由层回安装指引)。
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
import re
import shutil
import signal
import subprocess
import time
import uuid
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from ..config import get_settings
logger = logging.getLogger(__name__)
_ANCHOR_ID_RE = re.compile(r"^[0-9a-f]{12}$")
_ENGINE = "tectonic-p3/0.17.0"
_LOG_TAIL_LINES = 200
# 单槽并发控制(进程内;重启归零可接受——本服务无持久队列需求)
_lock = asyncio.Lock()
_pending = 0
class CompileError(Exception):
"""编译失败(含 log 尾)。code ∈ {COMPILE_FAILED, TIMEOUT, OUTPUT_TOO_LARGE}。"""
# compile_id:P4 项目编译失败时由 compile_project_job 附加(P3a 单文件为 None),
# 路由层据此在失败信封里回 compile_id + log_url(补充 3)。
compile_id: str | None = None
def __init__(self, code: str, message: str, log_tail: str = "") -> None:
super().__init__(message)
self.code = code
self.log_tail = log_tail
class QueueFullError(Exception):
"""等待槽位超限。"""
class ToolUnavailableError(Exception):
"""tectonic 二进制缺失/不可执行。"""
def _tectonic_bin() -> Path:
return get_settings().tectonic_abs
def _ensure_tool() -> Path:
bin_path = _tectonic_bin()
if not (bin_path.is_file() and os.access(bin_path, os.X_OK)):
raise ToolUnavailableError(
f"tectonic 未安装({bin_path})。"
"请执行:cd review_service && bash tools/install_tectonic.sh"
)
return bin_path
def _log_tail(path: Path, lines: int = _LOG_TAIL_LINES) -> str:
try:
text = path.read_text("utf-8", errors="replace")
except OSError:
return ""
return "\n".join(text.splitlines()[-lines:])
def compile_dir(anchor_id: str) -> Path:
"""{reformat_dir}/{anchor_id}/compile/(存在性由调用方按需 mkdir)。"""
return get_settings().reformat_dir_abs / anchor_id / "compile"
def prepare_input(anchor_id: str, source: str) -> Path:
"""写入隔离目录 input.tex;返回其路径。"""
cdir = compile_dir(anchor_id)
cdir.mkdir(parents=True, exist_ok=True)
input_tex = cdir / "input.tex"
input_tex.write_text(source, encoding="utf-8")
return input_tex
async def compile_tex(anchor_id: str) -> dict[str, Any]:
"""同步编译 anchor 的 input.tex(须先 prepare_input)。
→ {compile_id: anchor_id, pdf_path, log_path, seconds, pdf_bytes}
失败抛 CompileError / QueueFullError / ToolUnavailableError。
"""
global _pending
settings = get_settings()
_ensure_tool() # 快速失败:不占队列
sweep_expired()
if _pending >= settings.compile_queue_limit:
raise QueueFullError(
f"编译队列已满({_pending} 个任务排队),请稍后重试"
)
_pending += 1
try:
async with _lock:
return await _run_once(anchor_id)
finally:
_pending -= 1
async def _run_once(anchor_id: str) -> dict[str, Any]:
settings = get_settings()
cdir = compile_dir(anchor_id)
input_tex = cdir / "input.tex"
pdf_path = cdir / "output.pdf"
log_path = cdir / "output.log"
if not input_tex.is_file():
raise CompileError("COMPILE_FAILED", "编译输入不存在,请重新上传源码")
env = dict(os.environ)
env["TECTONIC_CACHE_DIR"] = str(settings.tectonic_cache_abs)
cmd = [
str(_tectonic_bin()), "-X", "compile", input_tex.name,
"--untrusted", "--keep-logs", "--outdir", ".",
]
started = time.monotonic()
try:
proc = await asyncio.create_subprocess_exec(
*cmd, cwd=cdir, env=env,
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
start_new_session=True,
)
try:
await asyncio.wait_for(
proc.communicate(), timeout=settings.compile_timeout_s
)
except asyncio.TimeoutError:
_kill_group(proc)
try:
await asyncio.wait_for(proc.wait(), timeout=5.0)
except asyncio.TimeoutError:
pass
log_path.write_text(_log_tail(log_path) + "\n[timeout] 编译超时,进程已终止",
encoding="utf-8")
raise CompileError(
"TIMEOUT",
f"编译超过 {settings.compile_timeout_s:g} 秒未完成,已终止",
_log_tail(log_path),
) from None
except FileNotFoundError as exc:
raise ToolUnavailableError(f"tectonic 不可执行:{exc}") from exc
finally:
seconds = round(time.monotonic() - started, 2)
# tectonic --keep-logs 把日志写在 cwd 的 input.log
produced_log = cdir / "input.log"
if produced_log.is_file():
shutil.move(str(produced_log), str(log_path))
if not log_path.is_file():
log_path.write_text("", encoding="utf-8")
produced_pdf = cdir / "input.pdf"
if not produced_pdf.is_file():
raise CompileError(
"COMPILE_FAILED",
"编译失败(未产出 PDF),请查看日志定位 LaTeX 错误",
_log_tail(log_path),
)
pdf_bytes = produced_pdf.stat().st_size
if pdf_bytes > settings.max_compile_pdf_bytes:
produced_pdf.unlink(missing_ok=True)
raise CompileError(
"OUTPUT_TOO_LARGE",
f"产出 PDF 超过 {settings.max_compile_pdf_bytes // (1024 * 1024)}MB 上限,已删除",
_log_tail(log_path),
)
produced_pdf.rename(pdf_path)
meta = {
"compile_id": anchor_id,
"engine": _ENGINE,
"seconds": seconds,
"pdf_bytes": pdf_bytes,
"returncode": 0,
"compiled_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
}
(cdir / "meta.json").write_text(
json.dumps(meta, ensure_ascii=False, indent=2), encoding="utf-8"
)
logger.info("compile: %s → PDF %d bytes, %.2fs", anchor_id, pdf_bytes, seconds)
return {
"compile_id": anchor_id,
"pdf_path": pdf_path,
"log_path": log_path,
"seconds": seconds,
"pdf_bytes": pdf_bytes,
}
def _kill_group(proc: asyncio.subprocess.Process) -> None:
try:
os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
except (ProcessLookupError, PermissionError, OSError):
try:
proc.kill()
except ProcessLookupError:
pass
def sweep_expired() -> int:
"""删除 mtime 超 retention_days 的 12-hex 锚点目录(编译产物+P1 落盘一并清)。"""
settings = get_settings()
base = settings.reformat_dir_abs
if not base.is_dir():
return 0
cutoff = time.time() - settings.compile_retention_days * 86400
removed = 0
for child in base.iterdir():
if not child.is_dir() or not _ANCHOR_ID_RE.fullmatch(child.name):
continue
try:
if child.stat().st_mtime < cutoff:
shutil.rmtree(child, ignore_errors=True)
removed += 1
except OSError:
continue
if removed:
logger.info("compile sweep: 清理 %d 个过期任务目录(>%d 天)",
removed, settings.compile_retention_days)
return removed
# ================================================================ P4:项目 zip(inspect + run)
#
# 与 P3a 共用:_lock/_pending 单槽队列、_ensure_tool、CompileError/QueueFullError/
# ToolUnavailableError、_kill_group、_log_tail、12-hex 目录与 mtime sweep 约定。
# P3a 函数(prepare_input/compile_tex/_run_once)零改动。
class InspectError(Exception):
"""inspect/消费阶段业务失败。
code ∈ {VALIDATION, ZIP_UNSAFE, ZIP_BOMB, ZIP_INVALID, ZIP_TOO_LARGE}。
"""
def __init__(self, code: str, message: str) -> None:
super().__init__(message)
self.code = code
self.message = message
_DOCCLASS_RE = re.compile(r"(?<!\\)\\documentclass\s*(\[|\{)")
_IMG_LIKE_EXT = {".png", ".jpg", ".jpeg", ".pdf", ".eps", ".svg"}
def inspect_root() -> Path:
"""data/compile_inspect/(1 小时 TTL 暂存根,存在性由调用方按需 mkdir)。"""
return get_settings().inspect_dir_abs
def compile_job_dir(cid: str) -> Path:
"""data/compile_jobs/{cid}/(产物根;不走 reformat_dir,P4 独立清扫)。"""
return get_settings().compile_jobs_dir_abs / cid
def _strip_tex_comments(text: str) -> str:
"""粗剥注释:行内 % 截断(\\% 误伤可接受,MVP 设计决策)。"""
return "\n".join(ln.split("%", 1)[0] for ln in text.splitlines())
def _read_tex(path: Path) -> str:
"""utf-8 → gb18030 → latin-1 降级读(中文模板常见 GBK)。"""
raw = path.read_bytes()
for enc in ("utf-8", "gb18030", "latin-1"):
try:
return raw.decode(enc)
except UnicodeDecodeError:
continue
return raw.decode("utf-8", errors="replace")
def find_main_tex(project_dir: Path) -> tuple[str | None, list[str], int]:
"""主文件识别(补充 2)→ (main_rel | None, 含 documentclass 的相对路径清单, .tex 总数)。
场景:唯一候选 → main;0 个 .tex / 0 候选 / ≥2 候选 → None + 清单(调用方报错)。
返回的相对路径用 posix 分隔符(与 zip 条目名一致)。
"""
tex_files = sorted(
p for p in project_dir.rglob("*")
if p.is_file() and p.suffix.lower() == ".tex"
)
if not tex_files:
return None, [], 0
candidates: list[str] = []
for p in tex_files:
if _DOCCLASS_RE.search(_strip_tex_comments(_read_tex(p))):
candidates.append(p.relative_to(project_dir).as_posix())
if len(candidates) == 1:
return candidates[0], candidates, len(tex_files)
return None, candidates, len(tex_files)
_MISSING_FILE_RE = re.compile(
r"LaTeX Error: File [`']([^`']+)' not found"
r"|I can't find file [`']([^`']+)'"
)
_IMG_INCLUSION_RE = re.compile(r"""image inclusion failed for ["']([^"']+)["']""")
_EPS_CONVERTED_SUFFIX = "-eps-converted-to.pdf"
_REF_TEXT_EXTS = {".tex", ".cls", ".sty", ".cfg", ".def"}
# EPS 现场转换 v2(约束 1-4):xdvipdfmx 无法内嵌 EPS → 解压后批量转同名 .pdf
_EPS_CONVERT_TIMEOUT_S = 15 # 约束2:单文件转换超时(秒)
_EPS_CONVERT_MAX_FILE = 20 * 1024 * 1024 # 约束3:单文件转换后上限(warning)
_EPS_CONVERT_MAX_TOTAL = 100 * 1024 * 1024 # 约束3:转换产物总上限(中止)
def _eps_tool_commands(eps: Path, pdf: Path) -> list[tuple[str, list[str]]]:
"""转换工具优先级:gs(约束1 必须 -dSAFER)→ epstopdf → ps2pdf;只列 PATH 里可用的。"""
tools: list[tuple[str, list[str]]] = []
gs = shutil.which("gs")
if gs:
tools.append(("gs", [
gs, "-dSAFER", "-dNOPAUSE", "-dBATCH", "-sDEVICE=pdfwrite",
"-dPDFSETTINGS=/prepress",
f"-sOutputFile={pdf}", str(eps),
]))
epstopdf = shutil.which("epstopdf")
if epstopdf:
# epstopdf 默认输出到同目录同名 .pdf(即 pdf 路径)
tools.append(("epstopdf", [epstopdf, str(eps)]))
ps2pdf = shutil.which("ps2pdf")
if ps2pdf:
tools.append(("ps2pdf", [ps2pdf, str(eps), str(pdf)]))
return tools
def _convert_eps_file(eps: Path, pdf: Path) -> tuple[str, str]:
"""逐工具尝试把 eps 转成同名 pdf → ("ok", tool) | ("timeout", tool) | ("fail", "")。
约束2:单工具 15s 超时 → 放弃该文件(继续下一个 eps,不再试后续工具)。
成功判定:退出后 pdf 存在且非空(防半成品)。
"""
tools = _eps_tool_commands(eps, pdf)
if not tools:
return "fail", ""
for tool, cmd in tools:
pdf.unlink(missing_ok=True) # 清上一个工具的半成品
try:
subprocess.run(
cmd, timeout=_EPS_CONVERT_TIMEOUT_S,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
)
except subprocess.TimeoutExpired:
pdf.unlink(missing_ok=True)
return "timeout", tool
except OSError:
continue
if pdf.is_file() and pdf.stat().st_size > 0:
return "ok", tool
pdf.unlink(missing_ok=True)
return "fail", ""
def process_eps_images(project: Path) -> list[str]:
"""EPS 兼容 v2:解压后**批量转换项目内所有 .eps**(不依赖 zip 自带预转换版)。
对每个 X.eps(原 .eps 保留):
1. 已有同名 X.pdf → 跳过转换(引用照常切换为 .pdf);
2. zip 自带 X-eps-converted-to.pdf 预转换版 → 复制为 X.pdf(免转换);
3. 否则现场转换:gs(-dSAFER) → epstopdf → ps2pdf 逐个 fallback(约束1),
单文件 15s 超时 → warning 后继续下一个(约束2)。
转换成功 → 显式 `X.eps` 引用字节替换为 `X.pdf`(.tex/.cls/.sty/.cfg/.def 全扫,
纯字节 ASCII/UTF-8 替换不碰编码);无扩展名引用 `{.../X}` 由 graphicx 直接
命中新生成的 X.pdf(解析顺序 .pdf→…→.eps)。
约束3:单文件转换后 >20MB → warning;转换产物累计 >100MB → 抛
InspectError("ZIP_TOO_LARGE") 中止;约束4:返回值为写入 output.log 的
"[eps-convert]" 日志行(由 _run_project 落盘)。
→ 日志行列表(无 .eps 项目 → 空)。
"""
eps_files = sorted(
p for p in project.rglob("*")
if p.is_file() and p.suffix.lower() == ".eps"
)
if not eps_files:
return []
lines: list[str] = []
handled: list[tuple[bytes, bytes]] = []
total_out = 0
def _rel(p: Path) -> str:
return p.relative_to(project).as_posix()
for eps in eps_files:
pdf = eps.with_suffix(".pdf")
src_rel, dst_rel = _rel(eps), _rel(pdf)
if pdf.is_file():
handled.append((eps.name.encode("utf-8"), pdf.name.encode("utf-8")))
continue
converted = eps.with_name(eps.stem + _EPS_CONVERTED_SUFFIX)
if converted.is_file():
t0 = time.monotonic()
shutil.copyfile(converted, pdf)
elapsed = time.monotonic() - t0
tool = "preset"
else:
t0 = time.monotonic()
status, tool = _convert_eps_file(eps, pdf)
elapsed = time.monotonic() - t0
if status == "timeout":
lines.append(f"[eps-convert] {src_rel} 转换超时") # 约束2
continue
if status != "ok":
lines.append(
f"[eps-convert] 全部工具失败,图片 {src_rel} 未转换")
continue
# 约束4:转换日志(成功路径)
lines.append(
f"[eps-convert] {src_rel} → {dst_rel} "
f"({eps.stat().st_size} → {pdf.stat().st_size}, "
f"{elapsed:.2f}s, {tool})")
size_out = pdf.stat().st_size
if size_out > _EPS_CONVERT_MAX_FILE: # 约束3
lines.append(
f"[eps-convert] 警告:{dst_rel} 转换后 {size_out} 字节"
f"超过 {_EPS_CONVERT_MAX_FILE // (1024 * 1024)}MB 上限")
total_out += size_out
if total_out > _EPS_CONVERT_MAX_TOTAL: # 约束3
raise InspectError(
"ZIP_TOO_LARGE",
f"EPS 转换后总大小超过 {_EPS_CONVERT_MAX_TOTAL // (1024 * 1024)}MB 上限"
f"(已达 {total_out // (1024 * 1024)}MB),请压缩图片后重试")
handled.append((eps.name.encode("utf-8"), pdf.name.encode("utf-8")))
# 显式 .eps 引用 → .pdf(仅对转换成功者;失败的保持 .eps → 编译兜底友好报错)
changed = 0
if handled:
for f in project.rglob("*"):
if not (f.is_file() and f.suffix.lower() in _REF_TEXT_EXTS):
continue
data = f.read_bytes()
new = data
for old_b, new_b in handled:
new = new.replace(old_b, new_b)
if new != data:
f.write_bytes(new)
changed += 1
if changed:
lines.append(f"[eps-convert] {changed} 个文件的显式 .eps 引用已改写为 .pdf")
if lines:
logger.info("eps v2: %s", " | ".join(lines))
return lines
def _append_eps_lines(log_path: Path, lines: list[str] | None) -> None:
"""约束4:把 [eps-convert] 转换日志写入 output.log(引擎 log 之后、chatter 之前)。"""
if not lines:
return
with log_path.open("a", encoding="utf-8") as fh:
fh.write("\n".join(lines) + "\n")
def classify_compile_error(log_tail: str) -> str:
"""COMPILE_FAILED 的日志 → 用户友好文案(Q4/§4.4)。"""
m = _MISSING_FILE_RE.search(log_tail)
if m:
name = next(g for g in m.groups() if g)
ext = Path(name).suffix.lower()
if ext == ".cls":
return f"项目缺少 {name},请从期刊官网下载模板后重新上传"
if ext in (".sty", ".def", ".clo"):
return f"缺少 LaTeX 宏包 {name},可能是模板不完整"
if ext in _IMG_LIKE_EXT:
return f"图片 {name} 未在 zip 里,请检查引用路径与大小写"
if ext == ".bib":
return f"缺少文献库 {name},请确认 .bib 已包含在 zip 中"
return f"编译找不到文件 {name},请检查 zip 内文件路径"
m = _IMG_INCLUSION_RE.search(log_tail)
if m:
name = m.group(1)
if Path(name).suffix.lower() == ".eps":
# EPS v2 已现场转换;仍失败 = 服务器转换没成功(工具缺失/超时/格式怪异)
return "项目包含 EPS 图片,服务器转换失败,请转成 PDF/PNG 后重试"
return f"图片 {name} 无法内嵌,请检查图片格式与路径(建议使用 PDF/PNG)"
if re.search(r"Emergency stop|Undefined control sequence", log_tail):
return "LaTeX 源码有语法错误,请展开日志定位出错行"
return "编译失败,请查看日志定位 LaTeX 错误"
async def inspect_zip(zip_bytes: bytes) -> dict[str, Any]:
"""两段式之①:安全解压 + 主文件识别 + 落盘 inspect 暂存(1h TTL)。
→ {job_id, main, stats{files,tex,images,cls,bib,other,bytes}, skipped[], skipped_count}
失败抛 InspectError(路由层转信封);任何失败 → rmtree 暂存目录,磁盘零残留。
"""
from .zip_safety import ZipSafetyError, safe_extract
settings = get_settings()
sweep_inspect_expired() # 机会性清扫(同 P3a 模式)
job_id = uuid.uuid4().hex[:12]
jdir = inspect_root() / job_id
project = jdir / "project"
try:
try:
result = await asyncio.to_thread(safe_extract, zip_bytes, project, settings)
except ZipSafetyError as exc:
raise InspectError(exc.code, exc.message) from exc
main, candidates, tex_total = find_main_tex(project)
if tex_total == 0:
raise InspectError("VALIDATION", "zip 中没有 .tex 文件")
if not candidates:
raise InspectError(
"VALIDATION", "未找到含 \\documentclass 的 .tex 文件")
if main is None:
shown = ", ".join(Path(c).name for c in candidates)
raise InspectError(
"VALIDATION",
f"检测到多个主文件({shown}),请合并为单一入口后重新上传")
meta = {
"job_id": job_id,
"main": main,
"stats": {
"files": result.files, "tex": result.tex, "images": result.images,
"cls": result.cls, "bib": result.bib, "other": result.other,
"bytes": result.bytes,
},
"skipped": [{"name": s.name, "reason": s.reason} for s in result.skipped],
"skipped_count": result.skipped_count,
"created_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
}
jdir.mkdir(parents=True, exist_ok=True)
(jdir / "inspect.json").write_text(
json.dumps(meta, ensure_ascii=False, indent=2), encoding="utf-8")
return meta
except BaseException:
shutil.rmtree(jdir, ignore_errors=True)
raise
def consume_inspect(job_id: str, cid: str) -> tuple[Path, dict[str, Any]]:
"""两段式之②前置:**原子 rename 消费** inspect 暂存 → compile_jobs/{cid}/project。
并发正确性(防同一 job 双跑):
os.rename(src, dst) 在同一文件系统内是原子的,且 src 不存在 → FileNotFoundError。
两个并发 run 拿同一 job_id 时,只有一个能成功 rename;后到者拿到
FileNotFoundError → 路由层转 VALIDATION"已过期/不存在",**绝不可能双编译**。
inspect 残壳(inspect.json)在 rename 成功后立即删除,也杜绝二跑读到半状态。
"""
settings = get_settings()
if not _ANCHOR_ID_RE.fullmatch(job_id or ""):
raise InspectError("VALIDATION", "job_id 格式非法(应为 12 位 hex)")
src_root = inspect_root() / job_id
meta_path = src_root / "inspect.json"
src_project = src_root / "project"
if not (meta_path.is_file() and src_project.is_dir()):
raise InspectError(
"VALIDATION", "上传任务不存在或已过期(1 小时),请重新上传 zip")
try:
meta = json.loads(meta_path.read_text("utf-8"))
except (OSError, ValueError) as exc:
raise InspectError(
"VALIDATION", "上传任务数据损坏,请重新上传 zip") from exc
dst_root = compile_job_dir(cid)
dst_project = dst_root / "project"
dst_root.mkdir(parents=True, exist_ok=True)
try:
os.rename(src_project, dst_project)
except FileNotFoundError as exc:
raise InspectError(
"VALIDATION", "上传任务不存在或已过期(1 小时),请重新上传 zip") from exc
except OSError as exc: # 跨设备等罕见失败 → 保持 inspect 原样可重试
raise InspectError(
"VALIDATION", "任务消费失败,请重试或重新上传") from exc
shutil.rmtree(src_root, ignore_errors=True) # 消费成功 → 删残壳
return dst_project, meta
async def compile_project_job(job_id: str) -> dict[str, Any]:
"""两段式之②:原子消费 inspect → tectonic 编译 → 产物留 pdf/log/meta。
队列/工具检查**先于消费**(inspect 保留 → 失败可直接重试);
EPS v2:解压后批量转换所有 .eps → 同名 .pdf(gs -dSAFER → epstopdf → ps2pdf,
单文件 15s 超时;产物累计 >100MB → ZIP_TOO_LARGE 中止),转换日志进 output.log;
成功 → 立即删 project/;失败(COMPILE_FAILED/TIMEOUT)→ **保留 project/ 现场 24h**
供诊断(sweep_compile_jobs_expired 按 failed_project_ttl_s 清理;
KEEP_FAILED_PROJECTS=1 永久保留);
失败必落完整 output.log(含 --print 的 XDV→PDF 阶段输出,补充 3)。
→ {compile_id, pdf_path, log_path, seconds, pdf_bytes}
"""
global _pending
settings = get_settings()
_ensure_tool() # 快速失败:不消费 inspect、不占队列
sweep_inspect_expired()
sweep_compile_jobs_expired()
if _pending >= settings.compile_queue_limit:
raise QueueFullError(
f"编译队列已满({_pending} 个任务排队),请稍后重试")
_pending += 1 # 占位先于消费:QUEUE_FULL 时 inspect 未动
cid = uuid.uuid4().hex[:12]
project: Path | None = None
meta: dict[str, Any] = {}
try:
project, meta = await asyncio.to_thread(consume_inspect, job_id, cid)
try:
# EPS v2:解压后批量转换所有 .eps → 同名 .pdf 并切换显式引用
# (不依赖 zip 自带预转换版;失败现场保留转换后的改写结果)
eps_lines = await asyncio.to_thread(process_eps_images, project)
async with _lock:
result = await _run_project(
cid, project, str(meta.get("main", "")), eps_lines)
# 成功 → 立即清理 project 树(失败路径在 finally 保留现场)
shutil.rmtree(project, ignore_errors=True)
project = None
return result
except CompileError as exc:
# 补充 3:失败信封也要能下载 log → 异常携带产物 cid(log 落盘于 compile_jobs/{cid}/)
exc.compile_id = cid
raise
finally:
_pending -= 1
if project is not None:
# 编译失败 → 保留 compile_jobs/{cid}/project/ 现场供诊断;
# touch 目录使 24h 保留窗从失败时刻起算
try:
os.utime(project, None)
except OSError:
pass
async def _run_project(
cid: str,
project: Path,
main_rel: str,
eps_log_lines: list[str] | None = None,
) -> dict[str, Any]:
"""单槽内执行项目编译:cwd=main 所在目录(相对路径解析),outdir=compile_jobs/{cid}。
eps_log_lines:EPS v2 的 [eps-convert] 转换日志(约束4),正常/超时两条路径
都会写进 output.log(引擎 log 之后、chatter/[timeout] 之前)。
"""
settings = get_settings()
outdir = compile_job_dir(cid)
outdir.mkdir(parents=True, exist_ok=True)
main_path = project / main_rel
if not main_rel or not main_path.is_file():
raise InspectError("VALIDATION", "主文件缺失,请重新上传 zip")
cwd = main_path.parent
stem = main_path.stem
env = dict(os.environ)
env["TECTONIC_CACHE_DIR"] = str(settings.tectonic_cache_abs)
# --print:打印引擎完整 chatter(含 XDV→PDF/xdvipdfmx 阶段输出);
# 不加时 xdvipdfmx 报错只进 tectonic stderr、不落 {stem}.log,导致"LaTeX 成功但
# 无 PDF"且 log 无错误行(MDPI 模板实测)。stdout/stderr 统一落盘 output.log。
cmd = [
str(_tectonic_bin()), "-X", "compile", main_path.name,
"--untrusted", "--keep-logs", "--print", "--outdir", str(outdir),
]
timeout = settings.compile_project_timeout_s
log_path = outdir / "output.log"
started = time.monotonic()
try:
proc = await asyncio.create_subprocess_exec(
*cmd, cwd=str(cwd), env=env,
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
start_new_session=True,
)
try:
stdout_bytes, _ = await asyncio.wait_for(
proc.communicate(), timeout=timeout)
except asyncio.TimeoutError:
_kill_group(proc)
try:
await asyncio.wait_for(proc.wait(), timeout=5.0)
except asyncio.TimeoutError:
pass
# 超时后 communicate 已取消,chatter 不可取;.log(--keep-logs)仍完整
_ensure_project_log(outdir, stem, cwd)
# 约束4:转换日志先落(转换发生在编译之前),再追加超时标记
_append_eps_lines(log_path, eps_log_lines)
# 追加而非覆盖:保留完整引擎输出(诊断 XDV→PDF 阶段必需)
with log_path.open("a", encoding="utf-8") as fh:
fh.write(f"\n[timeout] 编译超时({timeout:g}s),进程已终止\n")
raise CompileError(
"TIMEOUT",
f"编译超过 {timeout:g} 秒未完成,已终止"
"(项目首次编译需下载宏包,可能较慢,请重试)",
_log_tail(log_path),
) from None
except FileNotFoundError as exc:
raise ToolUnavailableError(f"tectonic 不可执行:{exc}") from exc
finally:
seconds = round(time.monotonic() - started, 2)
_ensure_project_log(outdir, stem, cwd)
# 产物归一:{stem}.pdf/.log → output.pdf/.log(补充 3:失败也必有 output.log)
produced_pdf = outdir / f"{stem}.pdf"
produced_log = outdir / f"{stem}.log"
if produced_log.is_file():
shutil.move(str(produced_log), str(log_path))
if not log_path.is_file():
log_path.write_text(
"[no log] tectonic 未产出日志(进程异常退出)", encoding="utf-8")
# 约束4:[eps-convert] 转换日志(引擎 log 之后、chatter 之前)
_append_eps_lines(log_path, eps_log_lines)
# --print 的进程输出(含 XDV→PDF 阶段)追加到 output.log 全文之后;
# 信封 detail 仍只取尾 200 行,磁盘保留完整 log 供"查看完整 log"下载
_append_chatter(log_path, stdout_bytes)
if not produced_pdf.is_file():
raise CompileError(
"COMPILE_FAILED",
classify_compile_error(_log_tail(log_path)),
_log_tail(log_path),
)
pdf_bytes = produced_pdf.stat().st_size
if pdf_bytes > settings.max_compile_pdf_bytes:
produced_pdf.unlink(missing_ok=True)
raise CompileError(
"OUTPUT_TOO_LARGE",
f"产出 PDF 超过 {settings.max_compile_pdf_bytes // (1024 * 1024)}MB 上限,已删除",
_log_tail(log_path),
)
produced_pdf.rename(outdir / "output.pdf")
meta = {
"compile_id": cid,
"project": True,
"main": main_rel,
"engine": _ENGINE,
"seconds": seconds,
"pdf_bytes": pdf_bytes,
"returncode": 0,
"compiled_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
}
(outdir / "meta.json").write_text(
json.dumps(meta, ensure_ascii=False, indent=2), encoding="utf-8")
logger.info("compile(project): %s main=%s → PDF %d bytes, %.2fs",
cid, main_rel, pdf_bytes, seconds)
return {
"compile_id": cid,
"pdf_path": outdir / "output.pdf",
"log_path": log_path,
"seconds": seconds,
"pdf_bytes": pdf_bytes,
}
def _ensure_project_log(outdir: Path, stem: str, cwd: Path) -> None:
"""tectonic 落 log 到 outdir 或 cwd → 统一挪到 outdir/output.log;都没有则占位。"""
log_path = outdir / "output.log"
if log_path.is_file():
return
for cand in (outdir / f"{stem}.log", cwd / f"{stem}.log"):
if cand.is_file():
shutil.move(str(cand), str(log_path))
return
def _append_chatter(log_path: Path, stdout_bytes: bytes | None) -> None:
"""把 tectonic --print 的进程 stdout(含 XDV→PDF/xdvipdfmx 阶段)追加到 output.log。
{stem}.log 只有 LaTeX 引擎内容(止于 "Output written on … .xdv");
XDV→PDF 转换错误只出现在进程 chatter 里——不落盘则"无 PDF 且 log 无错误行"。
"""
if not stdout_bytes:
return
chatter = stdout_bytes.decode("utf-8", errors="replace").strip()
if not chatter:
return
with log_path.open("a", encoding="utf-8") as fh:
fh.write("\n\n===== tectonic --print(进程输出,含 XDV→PDF 阶段)=====\n")
fh.write(chatter)
fh.write("\n")
def sweep_inspect_expired() -> int:
"""删 mtime 超 inspect_ttl_s(默认 1h)的 12-hex inspect 暂存目录。"""
settings = get_settings()
base = settings.inspect_dir_abs
if not base.is_dir():
return 0
cutoff = time.time() - settings.inspect_ttl_s
removed = 0
for child in base.iterdir():
if not child.is_dir() or not _ANCHOR_ID_RE.fullmatch(child.name):
continue
try:
if child.stat().st_mtime < cutoff:
shutil.rmtree(child, ignore_errors=True)
removed += 1
except OSError:
continue
if removed:
logger.info("inspect sweep: 清理 %d 个过期 inspect 暂存(>%d 秒)",
removed, settings.inspect_ttl_s)
return removed
def _keep_failed_projects() -> bool:
"""失败现场是否永久保留(调试):REVIEW_KEEP_FAILED_PROJECTS=1 或裸 KEEP_FAILED_PROJECTS=1。"""
if get_settings().keep_failed_projects:
return True
return os.environ.get("KEEP_FAILED_PROJECTS", "").strip() == "1"
def sweep_compile_jobs_expired() -> int:
"""编译产物清扫,两级:
① 失败现场 project/ 子目录:mtime 超 failed_project_ttl_s(默认 24h)→ 删子目录
(compile_jobs/{cid} 本体与 output.log/meta 保留,供"查看完整 log");
KEEP_FAILED_PROJECTS=1 跳过(永久保留,调试用)。
② 整个 compile_jobs/{cid}:mtime 超 compile_retention_days(7 天)→ 整目录删。
"""
settings = get_settings()
base = settings.compile_jobs_dir_abs
if not base.is_dir():
return 0
removed = 0
keep_failed = _keep_failed_projects()
# ① 失败现场 24h 清理(成功编译的 project/ 已在 compile_project_job 立即删)
if not keep_failed:
proj_cutoff = time.time() - settings.failed_project_ttl_s
for child in base.iterdir():
if not child.is_dir() or not _ANCHOR_ID_RE.fullmatch(child.name):
continue
proj = child / "project"
if not proj.is_dir():
continue
try:
if proj.stat().st_mtime < proj_cutoff:
shutil.rmtree(proj, ignore_errors=True)
removed += 1
except OSError:
continue
# ② 整目录 7 天清理
cutoff = time.time() - settings.compile_retention_days * 86400
for child in base.iterdir():
if not child.is_dir() or not _ANCHOR_ID_RE.fullmatch(child.name):
continue
try:
if child.stat().st_mtime < cutoff:
shutil.rmtree(child, ignore_errors=True)
removed += 1
except OSError:
continue
if removed:
logger.info("compile_jobs sweep: 清理 %d 个过期项目产物/失败现场", removed)
return removed
+203
View File
@@ -0,0 +1,203 @@
"""services/llm.py —— LLM 调用(httpx 裸调 DeepSeek,模式同 paper_writer_service)。
- complete_text(system, user, *, temperature=0.3, max_tokens=8000, timeout=None) -> str
- complete_json(...)(response_format json_object + 一次 JSON 修复重试)
- 4 次尝试(1.5s / 3s / 6s 指数退避,429 / 5xx / 超时)
- env 键(优先 REVIEW_LLM_*,回退 DOCGEN_LLM_*,再回退内置默认):
*_LLM_BASE_URL / *_LLM_API_KEY / *_LLM_MODEL / *_LLM_TIMEOUT
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
from typing import Any
import httpx
logger = logging.getLogger(__name__)
DEFAULT_LLM_BASE_URL = "https://api.deepseek.com/v1"
DEFAULT_LLM_MODEL = "deepseek-chat"
DEFAULT_LLM_TIMEOUT = 300.0
BACKOFF_SECONDS = (1.5, 3.0, 6.0)
MAX_ATTEMPTS = 4
class LLMError(Exception):
"""LLM 调用失败(重试尽/响应异常)。"""
def _env(keys: tuple[str, ...], default: str = "") -> str:
for key in keys:
val = os.environ.get(key)
if val:
return val
return default
def _llm_config() -> dict[str, Any]:
return {
"base_url": _env(
("REVIEW_LLM_BASE_URL", "DOCGEN_LLM_BASE_URL"), DEFAULT_LLM_BASE_URL
).rstrip("/"),
"api_key": _env(("REVIEW_LLM_API_KEY", "DOCGEN_LLM_API_KEY"), ""),
"model": _env(("REVIEW_LLM_MODEL", "DOCGEN_LLM_MODEL"), DEFAULT_LLM_MODEL),
"timeout": float(
_env(("REVIEW_LLM_TIMEOUT", "DOCGEN_LLM_TIMEOUT"), str(DEFAULT_LLM_TIMEOUT))
),
}
def _extract_text(payload: dict[str, Any]) -> str:
try:
choices = payload.get("choices") or []
if choices:
content = choices[0].get("message", {}).get("content")
if isinstance(content, str):
return content
if isinstance(content, list):
return "".join(
str(c.get("text", "")) if isinstance(c, dict) else str(c) for c in content
)
except Exception:
pass
return ""
async def _chat_once(
client: httpx.AsyncClient,
cfg: dict[str, Any],
system: str,
user: str,
*,
temperature: float,
max_tokens: int,
json_mode: bool,
timeout: float | None,
) -> str:
headers = {"Content-Type": "application/json"}
if cfg["api_key"]:
headers["Authorization"] = f"Bearer {cfg['api_key']}"
body: dict[str, Any] = {
"model": cfg["model"],
"messages": [
{"role": "system", "content": system},
{"role": "user", "content": user},
],
"temperature": temperature,
"max_tokens": max_tokens,
}
if json_mode:
body["response_format"] = {"type": "json_object"}
resp = await client.post(
f"{cfg['base_url']}/chat/completions",
headers=headers,
json=body,
timeout=timeout if timeout is not None else cfg["timeout"],
)
resp.raise_for_status()
payload = resp.json()
if not isinstance(payload, dict):
raise LLMError("LLM 响应形态异常(非 JSON 对象)")
text = _extract_text(payload)
if not text:
raise LLMError("LLM 响应缺少 choices[0].message.content")
return text
async def _complete(
system: str,
user: str,
*,
temperature: float,
max_tokens: int,
json_mode: bool,
timeout: float | None,
) -> str:
cfg = _llm_config()
last_exc: BaseException | None = None
async with httpx.AsyncClient() as client:
for attempt in range(MAX_ATTEMPTS):
try:
return await _chat_once(
client, cfg, system, user,
temperature=temperature, max_tokens=max_tokens, json_mode=json_mode,
timeout=timeout,
)
except LLMError:
raise
except (httpx.HTTPStatusError, httpx.HTTPError, httpx.TimeoutException) as exc:
last_exc = exc
status = getattr(getattr(exc, "response", None), "status_code", None)
retryable = status is None or status == 429 or status >= 500
if not retryable or attempt == MAX_ATTEMPTS - 1:
raise LLMError(f"LLM 调用失败(HTTP {status or '网络'}):{exc}") from exc
await asyncio.sleep(BACKOFF_SECONDS[min(attempt, len(BACKOFF_SECONDS) - 1)])
raise LLMError(f"LLM 调用失败:{last_exc}")
async def complete_text(
system: str,
user: str,
*,
temperature: float = 0.3,
max_tokens: int = 8000,
timeout: float | None = None,
) -> str:
"""文本补全。失败抛 LLMError。"""
return await _complete(
system, user, temperature=temperature, max_tokens=max_tokens, json_mode=False,
timeout=timeout,
)
def _repair_json(text: str) -> str:
text = text.strip()
if text.startswith("```"):
text = text.strip("`")
if text.lower().startswith("json"):
text = text[4:]
text = text.strip()
if not text:
raise LLMError("LLM 返回空内容")
if text.startswith("{"):
return text
start = text.find("{")
end = text.rfind("}")
if start >= 0 and end > start:
return text[start:end + 1]
raise LLMError("LLM 输出中未找到 JSON 对象")
async def complete_json(
system: str,
user: str,
*,
temperature: float = 0.3,
max_tokens: int = 8000,
timeout: float | None = None,
) -> dict[str, Any]:
"""JSON 补全(json_object + 一次修复重试)。"""
text = await _complete(
system, user, temperature=temperature, max_tokens=max_tokens, json_mode=True,
timeout=timeout,
)
try:
return json.loads(text)
except ValueError:
pass
text2 = await _complete(
system + "\n(重要:你的回答必须是单个合法 JSON 对象,不要输出其他任何内容)",
user,
temperature=temperature,
max_tokens=max_tokens,
json_mode=True,
timeout=timeout,
)
try:
return json.loads(text2)
except ValueError:
repaired = _repair_json(text2)
return json.loads(repaired)
+139
View File
@@ -0,0 +1,139 @@
"""paper_parse.py —— PDF → 纯文本(经 vstore 8094 现有解析接口,零改动复用)。
链路(全部只读消费 vstore 既有 API,不修改 vstore 代码):
1. POST {vstore}/api/vstore/ingest multipart file(vstore 内部 pymupdf 解析)
2. GET {vstore}/api/vstore/documents/{doc_id} → document.paragraphs 拼接纯文本
3. DELETE {vstore}/api/vstore/documents/{doc_id} (临时解析件 best-effort 清理,
REVIEW_PARSE_DELETE_AFTER=0 可关闭——保留文档便于人工排查)
为何不用 methodology 8092:其 /extract-upload 是 MinerU 分钟级异步 job
(MinerU 子进程 + LLM 抽取两阶段),不适配同步自审稿;vstore 段落解析秒级
返回,满足 presubmit 场景。methodology 地址在 config 中预留供后续扩展。
"""
from __future__ import annotations
import logging
from typing import Any
import httpx
from ..config import get_settings
logger = logging.getLogger(__name__)
class ParseError(Exception):
"""PDF 解析失败(上游不可达/响应异常/空文本)。"""
async def parse_pdf(path: str) -> str:
"""接收 PDF 文件路径,返回拼接后的纯文本。失败抛 ParseError。"""
try:
with open(path, "rb") as fh:
file_bytes = fh.read()
except OSError as exc:
raise ParseError(f"读取 PDF 失败:{exc}") from exc
return await parse_pdf_bytes(file_bytes, filename=str(path).rsplit("/", 1)[-1] or "paper.pdf")
async def parse_pdf_bytes(file_bytes: bytes, filename: str = "paper.pdf") -> str:
"""接收 PDF 字节流,返回拼接后的纯文本。失败抛 ParseError。"""
if not file_bytes:
raise ParseError("PDF 内容为空")
settings = get_settings()
base = settings.vstore_base
timeout = httpx.Timeout(
connect=5.0, read=settings.parse_timeout, write=settings.parse_timeout, pool=5.0
)
async with httpx.AsyncClient(timeout=timeout) as client:
# ---- 1) ingest(vstore 内部 pymupdf 解析 + 嵌入 + 加密落盘)----
try:
resp = await client.post(
f"{base}/api/vstore/ingest",
files={"file": (filename, file_bytes, "application/pdf")},
data={
"name": filename,
"category": "review_presubmit",
"description": "review_service 临时解析件",
},
)
except httpx.HTTPError as exc:
raise ParseError(f"vstore 不可达({base}):{exc}") from exc
if resp.status_code >= 400:
# 读上游错误体(信封 JSON 或裸文本),区分“不可达”与“上游报错”
raise ParseError(f"vstore ingest 失败(HTTP {resp.status_code}):{_err_body(resp)}")
try:
payload = resp.json()
except ValueError as exc:
raise ParseError(f"vstore ingest 响应非 JSON:{exc}") from exc
if not isinstance(payload, dict) or not payload.get("ok"):
message = _err_message(payload) or "vstore ingest 失败"
raise ParseError(message)
doc_id = str((payload.get("item") or {}).get("document_id") or "")
if not doc_id:
raise ParseError("vstore ingest 响应缺少 document_id")
# ---- 2) 取明文全文(paragraphs 拼接)+ 3) 清理临时件 ----
try:
text = await _fetch_text(client, base, doc_id)
finally:
if settings.parse_delete_after:
await _best_effort_delete(client, base, doc_id)
text = text.strip()
if not text:
raise ParseError("PDF 解析结果为空(可能是扫描件/图片型 PDF,vstore 无 OCR)")
return text
async def _fetch_text(client: httpx.AsyncClient, base: str, doc_id: str) -> str:
"""取 vstore 文档明文并按段落拼接纯文本。"""
try:
resp = await client.get(f"{base}/api/vstore/documents/{doc_id}")
except httpx.HTTPError as exc:
raise ParseError(f"vstore 取文档失败({doc_id}):{exc}") from exc
if resp.status_code >= 400:
raise ParseError(
f"vstore 取文档失败({doc_id},HTTP {resp.status_code}):{_err_body(resp)}"
)
try:
payload = resp.json()
except ValueError as exc:
raise ParseError(f"vstore 取文档响应非 JSON({doc_id}):{exc}") from exc
if not isinstance(payload, dict) or not payload.get("ok"):
raise ParseError(_err_message(payload) or f"vstore 取文档失败({doc_id})")
document: dict[str, Any] = payload.get("document") or {}
paragraphs = document.get("paragraphs") or []
return "\n".join(str(p) for p in paragraphs if str(p).strip())
async def _best_effort_delete(client: httpx.AsyncClient, base: str, doc_id: str) -> None:
try:
await client.delete(f"{base}/api/vstore/documents/{doc_id}")
except Exception: # noqa: BLE001 - 清理失败不阻断(文档级 secure_delete 已尽力)
logger.warning("paper_parse: vstore 临时文档清理失败(%s)", doc_id, exc_info=True)
def _err_message(payload: Any) -> str:
if isinstance(payload, dict):
err = payload.get("error")
if isinstance(err, dict):
return str(err.get("message") or "")
if isinstance(err, str):
return err
return ""
def _err_body(resp: httpx.Response) -> str:
"""上游错误体:优先 JSON 信封 message,回退裸文本(截断 500 字符)。"""
try:
payload = resp.json()
except ValueError:
return (resp.text or "").strip()[:500] or "(空响应体)"
return _err_message(payload) or str(payload)[:500]
@@ -0,0 +1,226 @@
"""services/paper_scanner.py —— 关键词兜底扫描器(纯 stdlib,零 LLM)。
背景(2026-10-12):hard_rule 标签由 LLM 输出,存在"正文明明有 accession/BUSCO/挂载率
却报缺失"的漏检风险(真实案例 s41588 Nature Genetics:正文有 PRJCA038097 等 +
BUSCO 98.57% + 94.18% anchored,仍被判 busco/accession/hic_missing → 大修)。
本模块做**程序层证伪**:LLM 说缺某项、扫描器在文中找到该项 → 剥掉 hard_rule
降级为非硬性 major,并在 rationale 记录冲突(前端黄色提示,请人工核对)。
边界(刻意保守,不引入新误判):
- 只做"证伪硬性缺项",不做"证实"——扫不到不等于真缺(可能同义表述),不升级任何条目
- accession 只在 Data Availability 段内扫(用户指令):正文引用他人数据的
accession 不得误认为"自己有"
"""
from __future__ import annotations
import re
from typing import Any
# ---- accession(只在 DA 段扫)----
ACCESSION_RE = re.compile(
r"\b(PRJNA|PRJEB|PRJCA|SRP|ERP|DRP|CRA|SRR|ERR|DRR|GCA_|GCF_)\s?(\d[\d._]*)",
re.IGNORECASE,
)
# ---- DA 段定位:标题多变体(行首允许编号前缀)----
_DA_START_RE = re.compile(
r"^[ \t]*(?:\d+(?:\.\d+)*\.?\s+)?"
r"(?:Data\s+(?:and\s+(?:code\s+)?)?[Aa]vailability"
r"|Availability\s+of\s+(?:data|Data)"
r"|Data\s+and\s+[Mm]aterials\s+[Aa]vailability"
r"|Data\s+availability\s+statement)",
re.MULTILINE,
)
# DA 段结束 = 下一个一级节标题
_NEXT_HEADER_RE = re.compile(
r"^[ \t]*(?:\d+(?:\.\d+)*\.?\s+)?"
r"(?:Abstract|Summary|Introduction|Results|Discussion|Methods"
r"|Materials\s+and\s+Methods|References|Acknowledg(?:e)?ments?"
r"|Author\s+[Ii]nformation|Rights\s+and\s+permissions|Change\s+history"
r"|Data\s+(?:and\s+(?:code\s+)?)?[Aa]vailability"
r"|Availability\s+of\s+[Dd]ata|Data\s+and\s+[Mm]aterials\s+[Aa]vailability)"
r"\s*$",
re.MULTILINE,
)
# 回退窗:DA 无行首标题时,短语出现处 + 上下文
_DA_PHRASE_RE = re.compile(r"data\s+availability", re.IGNORECASE)
_DA_FALLBACK_WINDOW = 3000
# ---- BUSCO(全文)----
_BUSCO_MENTION_RE = re.compile(r"\bBUSCO\b", re.IGNORECASE)
_BUSCO_FULL_RE = re.compile(
r"(?:\bS\s*:\s*\d|\bSingle[- ]copy)"
r".{0,200}?"
r"(?:\bD\s*:\s*\d|\bDuplicat)",
re.IGNORECASE | re.DOTALL,
)
# ---- Hi-C 挂载率(全文)----
HIC_RE = re.compile(
r"(anchored\s+to\s+chromosomes?"
r"|anchoring\s+rate"
r"|pseudo[- ]?chromosomes?"
r"|chromosome[- ]level"
r"|%[^.\n]{0,40}anchored)",
re.IGNORECASE,
)
_MAX_HITS = 20
def _extract_da_section(text: str) -> str:
"""定位 Data Availability 段正文(标题后 → 下一个一级节标题)。
无行首标题时回退:短语出现处向后 3000 字符窗。均未命中 → ""。
"""
m = _DA_START_RE.search(text)
if m:
nxt = _NEXT_HEADER_RE.search(text, m.end())
end = nxt.start() if nxt else min(len(text), m.end() + _DA_FALLBACK_WINDOW)
return text[m.start():end]
m2 = _DA_PHRASE_RE.search(text)
if m2:
return text[m2.start():min(len(text), m2.start() + _DA_FALLBACK_WINDOW)]
return ""
def _scan_accession(da_text: str) -> dict[str, Any]:
if not da_text:
return {"found": False, "hits": [], "in_da_section": False}
hits: list[str] = []
seen: set[str] = set()
for m in ACCESSION_RE.finditer(da_text):
token = (m.group(1) + m.group(2)).upper()
if token not in seen:
seen.add(token)
hits.append(token)
if len(hits) >= _MAX_HITS:
break
return {"found": bool(hits), "hits": hits, "in_da_section": True}
def _scan_busco(text: str) -> dict[str, Any]:
m = _BUSCO_MENTION_RE.search(text)
if not m:
return {"level": "none", "excerpt": ""}
excerpt = text[m.start():m.start() + 200]
if _BUSCO_FULL_RE.search(text):
level = "full"
else:
level = "partial" # 只有百分比、无 S/D 分项 → prompt 已定为 minor
return {"level": level, "excerpt": excerpt}
def _scan_hic(text: str) -> dict[str, Any]:
m = HIC_RE.search(text)
if not m:
return {"found": False, "excerpt": ""}
start = max(0, m.start() - 40)
return {"found": True, "excerpt": text[start:m.end() + 80]}
def scan_paper_signals(text: str) -> dict[str, Any]:
"""全文 → 结构化信号。accession 只在 DA 段;busco/hic 全文。"""
if not text:
return {
"accession": {"found": False, "hits": [], "in_da_section": False},
"busco": {"level": "none", "excerpt": ""},
"hic": {"found": False, "excerpt": ""},
}
return {
"accession": _scan_accession(_extract_da_section(text)),
"busco": _scan_busco(text),
"hic": _scan_hic(text),
}
def _get_hr(item: Any) -> Any:
if isinstance(item, dict):
return item.get("hard_rule")
return getattr(item, "hard_rule", None)
def _item_fatal(item: Any) -> bool:
if isinstance(item, dict):
return bool(item.get("fatal"))
return bool(getattr(item, "fatal", False))
def _set_downgraded(item: Any) -> None:
if isinstance(item, dict):
item["scanner_downgraded"] = True
else:
try:
item.scanner_downgraded = True
except Exception: # pydantic 严格模式等极端场景:吞掉,冲突记录仍保留
pass
def _set_hr_none(item: Any) -> None:
if isinstance(item, dict):
item["hard_rule"] = None
else:
try:
item.hard_rule = None
except Exception: # pydantic 严格模式等极端场景:吞掉,冲突记录仍保留
pass
def _item_title(item: Any) -> str:
if isinstance(item, dict):
return str(item.get("title") or "")
return str(getattr(item, "title", "") or "")
# 冲突映射:hard_rule 标签 → (信号检查, 冲突描述)
def _conflict_for(tag: str, signals: dict[str, Any]) -> str | None:
acc = signals.get("accession") or {}
busco = signals.get("busco") or {}
hic = signals.get("hic") or {}
if tag == "accession_missing" and acc.get("found"):
hits = ", ".join(acc.get("hits") or [])[:120]
return f"Data Availability 段已检出 accession({hits})"
if tag == "busco_missing" and busco.get("level") in ("partial", "full"):
return f"正文已提及 BUSCO({busco.get('level')} 格式)"
if tag == "hic_missing" and hic.get("found"):
excerpt = (hic.get("excerpt") or "").strip()[:80]
return f"正文已检出挂载率描述({excerpt})"
return None
def reconcile_hard_rules(
majors: list[Any], signals: dict[str, Any]
) -> tuple[list[Any], list[Any], list[dict[str, Any]]]:
"""对带 hard_rule 的 major 做扫描器证伪:命中 → 剥标签 + **移出 major 降级至
minor**(A2 语义,2026-10-12)→ 记冲突。
返回 (kept_majors, demoted_minors, conflicts):
- kept_majors:未冲突/未处理的 major(原序)
- demoted_minors:被冲突降级的条目(hard_rule 置 None、scanner_downgraded=True)
- conflicts:{hard_rule, title, reason}——fatal 条目只记冲突不降级
(拒稿级问题须人工过目,扫描器证伪不了伦理判断)。
"""
kept: list[Any] = []
demoted: list[Any] = []
conflicts: list[dict[str, Any]] = []
for m in majors:
tag = _get_hr(m)
if not isinstance(tag, str):
kept.append(m)
continue
reason = _conflict_for(tag, signals)
if reason is None:
kept.append(m)
continue
conflicts.append({
"hard_rule": tag,
"title": _item_title(m),
"reason": reason,
})
if _item_fatal(m):
kept.append(m) # fatal 保留原位(含其 hard_rule 供引擎 R1/R0 使用)
continue
_set_hr_none(m)
_set_downgraded(m)
demoted.append(m)
return kept, demoted, conflicts
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,264 @@
"""services/ppt_figure_index.py —— 论文真实图表清单扫描(P5 增强 1)。
问题根因:LLM 不知道论文里真实图编号与位置,suggested_figures 全靠编。
修复:用正则从论文全文扫描 Figure/Fig./图 + Table/表 编号 → figure_index
([{label, para_id, caption, kind}]),注入 reduce prompt 供 LLM「只选不编」,
代码侧再按清单校验:不在清单里的图引用直接丢弃(而非标 unverified)。
para_id 与 ppt_outline.build_index 的分段完全一致(复用同一套分段逻辑),
因此 figure_index 的 para_id 可直接用于证据链与前端展示「Fig 1(论文 S1-P12)」。
"""
from __future__ import annotations
import logging
import re
from pathlib import Path
from typing import Any
logger = logging.getLogger(__name__)
FIG_RE = re.compile(r"(?:Figure|Fig\.?|图)\s*(\d+)")
TABLE_RE = re.compile(r"(?:Table|表)\s*(\d+)")
_CAPTION_MARK = re.compile(r"\s*[::.]") # 紧跟编号的冒号/句点 → caption 形态
_SENT_SPLIT = re.compile(r"(?<=[.!?。!?;;])\s*")
_MAX_CAPTION = 120
# label → 规约键:("fig", n) / ("table", n)("Figure 2"/"Fig. 2"/"图 2"/"Fig 2" 同键)
_LABEL_KIND = re.compile(r"^\s*(figure|fig|图|table|表)", re.IGNORECASE)
# 行首 caption(借 paper-skills pdf_snap 的分隔符全集:::. 与 |;pdf_snap MIT)
_CAPTION_LINE = re.compile(
r"^\s*(Figure|Fig\.?|Table|Tab\.?|图|表|Algorithm|算法)\s*"
r"([0-9]+[a-z]?|[A-Z]\.?[0-9]+)\s*[::.|]\s*(\S.*)$", re.IGNORECASE,
)
def canon_label(label: Any) -> tuple[str, int] | None:
"""图/表 label 归一化 → (kind, number);无法解析返回 None。"""
s = str(label or "").strip()
m = _LABEL_KIND.match(s)
digits = re.search(r"(\d+)", s)
if not m or not digits:
return None
kind = "table" if m.group(1).lower() in ("table", "表") else "fig"
return (kind, int(digits.group(1)))
def figure_map(figure_index: list[dict[str, Any]] | None) -> dict[tuple[str, int], dict[str, Any]]:
"""figure_index → {("fig",2): entry} 查找表(校验用)。"""
out: dict[tuple[str, int], dict[str, Any]] = {}
for entry in figure_index or []:
key = canon_label(entry.get("label"))
if key and key not in out:
out[key] = entry
return out
def _sentence(paragraph: str, span: tuple[int, int]) -> str:
"""取包含命中的句子(截 120 字符)作为 caption。"""
start = 0
best = paragraph
for part in _SENT_SPLIT.split(paragraph):
end = start + len(part)
if start <= span[0] < end:
best = part
break
start = end
return best.strip()[:_MAX_CAPTION]
def _scan(index: dict[str, str]) -> list[dict[str, Any]]:
"""para_id→原文 索引 → 真实图表清单(按 (kind,编号) 去重)。"""
found: dict[tuple[str, int], tuple[int, dict[str, Any]]] = {}
for para_id, para_text in index.items():
for regex, kind in ((FIG_RE, "figure"), (TABLE_RE, "table")):
for m in regex.finditer(para_text):
key = (kind, int(m.group(1)))
# caption 形态("Figure 2: ...")优先于顺带提及;同分先到先得
score = 2 if _CAPTION_MARK.match(para_text[m.end(): m.end() + 4]) else 1
hit = {
"label": f"Fig {m.group(1)}" if kind == "figure" else f"Table {m.group(1)}",
"para_id": para_id,
"caption": _sentence(para_text, m.span()),
"kind": kind,
}
if key not in found or score > found[key][0]:
found[key] = (score, hit)
return [entry for _, entry in found.values()]
def scan_figure_index(index: dict[str, str]) -> list[dict[str, Any]]:
"""para_id→原文 索引 → 图表清单(纯函数,可单测)。"""
return _scan(index)
def build_figure_index(text: str) -> list[dict[str, Any]]:
"""论文全文 → 图表清单(与证据链共用同一套段落分段,para_id 一致)。"""
from .ppt_outline import build_index # 局部导入避免与 ppt_outline 循环依赖
_, index = build_index(text)
return _scan(index)
# ---------- PDF 原图提取(P5 增强 2.3;AGENTS 例外:vstore 只回文本,图片须本地解档) ----------
_BAND_ABOVE_PT = 300.0 # caption 上方搜索带(图底边距 caption 的最大距离,pt)
_BAND_PAD_X_PT = 40.0 # caption 左右扩展(横向重叠判定)
_MIN_PX = (60, 40) # 裁切结果最小像素(过滤图标/装饰线)
_DPI = 300
def extract_figures(pdf_bytes: bytes, out_dir: Path) -> list[dict[str, Any]]:
"""PDF → 与 caption 配对的原图 PNG(out_dir/Fig_1.png…)+ 清单。
策略(自 paper-skills pdf_snap「页渲染+裁图」与 paper-ppt-skill Step5 裁剪规则
自动化而来):同页 caption 行上方 band 内「嵌入位图 ∪ 矢量绘图」联合包围盒
→ clip 渲染 300dpi。Table 跳过(表格不出栅格图,由占位/表格版式承接)。
配对失败/过小 → 跳过(build 侧回落文字占位),绝不抛错阻塞建 job。
"""
out = Path(out_dir)
out.mkdir(parents=True, exist_ok=True)
results: dict[str, dict[str, Any]] = {}
try:
import pymupdf # lock 中已有(format_check 同款)
except ImportError: # pragma: no cover
logger.warning("pymupdf 不可用,跳过图表原图提取")
return []
try:
doc = pymupdf.open(stream=pdf_bytes, filetype="pdf")
except Exception as exc: # noqa: BLE001
logger.warning("PDF 打开失败,跳过图表原图提取:%s", exc)
return []
try:
for pno in range(doc.page_count):
page = doc[pno]
caps = _caption_lines(page)
if not caps:
continue
img_rects = _image_rects(page)
for cap in caps:
if cap["kind"] != "fig":
continue
key = f"Fig {cap['num']}"
if key in results:
continue
box = _figure_box(page, cap, img_rects)
if box is None:
continue
try:
pix = page.get_pixmap(clip=box, dpi=_DPI)
except Exception: # noqa: BLE001 - clip 越界等
continue
if pix.width < _MIN_PX[0] or pix.height < _MIN_PX[1]:
continue
fname = f"Fig_{cap['num']}.png"
try:
pix.save(str(out / fname))
except OSError:
continue
results[key] = {
"label": key, "file": fname, "page": pno + 1,
"caption": cap["text"][:_MAX_CAPTION],
"width": pix.width, "height": pix.height,
}
finally:
doc.close()
ordered = sorted(results.values(), key=lambda r: int(r["label"].split()[1]))
logger.info("图表原图提取:%d 张 → %s", len(ordered), out)
return ordered
def _caption_lines(page: Any) -> list[dict[str, Any]]:
"""页内行首 caption → [{kind, num, text, x0, x1, y0}](每编号取最先一行)。"""
caps: dict[str, dict[str, Any]] = {}
try:
data = page.get_text("dict")
except Exception: # noqa: BLE001
return []
for block in data.get("blocks", []):
if block.get("type") != 0:
continue
for line in block.get("lines", []):
text = "".join(sp.get("text", "") for sp in line.get("spans", []))
m = _CAPTION_LINE.match(text)
if not m:
continue
raw_kind = m.group(1).rstrip(".").lower() # "Fig." → "fig"(v3:剥尾点)
if raw_kind in ("figure", "fig", "图"):
kind = "fig"
elif raw_kind in ("table", "tab", "表"):
kind = "table"
else:
continue # Algorithm 等:清单不含,原图也跳过
num = m.group(2)
digits = re.match(r"(\d+)", num)
if not digits:
continue
key = f"{kind}{digits.group(1)}"
if key in caps:
continue
x0, y0, x1, y1 = line["bbox"]
caps[key] = {
"kind": kind, "num": int(digits.group(1)),
"text": m.group(3).strip(),
"x0": float(x0), "x1": float(x1), "y0": float(y0),
}
return list(caps.values())
def _image_rects(page: Any) -> list[Any]:
"""页内嵌入位图的 rect 列表(失败返回空)。"""
rects = []
try:
for info in page.get_image_info(xrefs=True):
xref = info.get("xref")
if not xref:
continue
for r in page.get_image_rects(xref):
rects.append(r)
except Exception: # noqa: BLE001
return []
return rects
def _figure_box(page: Any, cap: dict[str, Any], img_rects: list[Any]) -> Any | None:
"""caption 上方 band 内「位图 ∪ 矢量绘图」联合包围盒(含 6pt 内边距)。"""
import pymupdf # 局部导入(extract_figures 已保证可用)
page_rect = page.rect
band_y0 = cap["y0"] - _BAND_ABOVE_PT
band_y1 = cap["y0"] - 1.0
bx0 = cap["x0"] - _BAND_PAD_X_PT
bx1 = cap["x1"] + _BAND_PAD_X_PT
def _in_band(r: Any) -> bool:
return band_y0 <= float(r.y1) <= band_y1 and not (float(r.x1) < bx0 or float(r.x0) > bx1)
union = None
def _merge(r: Any) -> None:
nonlocal union
r = r & page_rect
if r.is_empty:
return
union = r if union is None else (union | r)
for r in img_rects:
if _in_band(r):
_merge(r)
try:
drawings = page.get_drawings()
except Exception: # noqa: BLE001
drawings = []
for d in drawings:
r = d.get("rect")
if r is None or r.is_empty:
continue
# 细长规则线(页眉横线/表格线)不进联合,避免整页宽条
if float(r.height) < 1.5 or (float(r.width) < 1.5):
continue
if _in_band(r):
_merge(r)
if union is None:
return None
box = pymupdf.Rect(union.x0 - 6, union.y0 - 6, union.x1 + 6, union.y1 + 6) & page_rect
return None if box.is_empty else box
+917
View File
@@ -0,0 +1,917 @@
"""services/ppt_outline.py —— P5 科研汇报大纲生成(map-reduce + 证据程序化校验)。
管线(全 LLM 只负责引用与措辞,结构/页数/证据状态由代码裁决):
纯文本 → 段落索引(para_id 形如 S4-P1,figure 引用 S4-P1-Fig2)
→ 图表清单扫描(ppt_figure_index 正则扫 Figure/图/Table/表,非 LLM)
→ map:分块(≤5 块)抽 claims/figures(quote 逐字 + para_id 标注,LLM;
figures 先过真实清单,不在清单的丢弃)
→ reduce:按 page_plan 配额生成页大纲(1 或 2 次调用;>14 页分两片,
第二片带上一片页标题维持连贯;deepseek-chat 输出上限 8K,必须分片;
真实图表清单注入 prompt,suggested_figures 只选不编)
→ 后校验(纯函数、可单测):
① 页数:PAGE_TARGETS 决策 2(10→8-10/15→11-14/20→15-18/30→22-26),
超上限从尾部可删页裁掉(section/reference 先删);低于下限不凑页 + warning
② bullets(补充 3):内容页 <2 条 → 一次按页扩充重试;>8 条 → 拆成两页
③ evidence:quote 归一化后必须是某段落的子串 → 反查真实 para_id
(repair_evidence_para_ids,修复 LLM 编造段落ID)→ verified;
找不到段落 → unverified;result/conclusion 页未核实占比 >50% → warning
④ suggested_figures:label 必须在真实图表清单内(否则丢弃而非标
unverified);para_id 以清单扫描出的真实位置为准 → verified
错误:OutlineError(code=VALIDATION|LLM_ERROR|LLM_INVALID) → 路由映射信封。
LLM 调用次数:map ≤5 + reduce 1~2 + expand ≤1 ≈ 4~8 次/篇。
"""
from __future__ import annotations
import asyncio
import json
import math
import re
import unicodedata
from typing import Any
from ..config import get_settings
from . import llm
from .llm import LLMError
from .ppt_figure_index import canon_label, figure_map, scan_figure_index
# ---------- 枚举与决策常量 ----------
SCENARIOS = (
"group_meeting",
"proposal",
"conclusion_report",
"defense",
"conference_talk",
)
SCENARIO_LABELS = {
"group_meeting": "组会",
"proposal": "开题",
"conclusion_report": "结题",
"defense": "答辩",
"conference_talk": "学术会议",
}
# P5 增强 2:风格 3 → 8 → 14(与 data/ppt_styles/*.json 一一对应)。
# 其中 domestic/international/systems-talk/theory-beamer/jp-gothic/editorial
# 移植自开源 paper-skills styles/tokens.json(MIT),色/字号映射进自有 schema。
STYLE_IDS = (
"academic-minimal", "botanic-forest", "classic-defense",
"academic-blue", "nature-green", "tech-dark", "monochrome", "warm-earth",
"domestic", "international", "systems-talk", "theory-beamer",
"jp-gothic", "editorial",
)
# section = 章节分隔页(P5 增强 3:大标题居中 + 色块背景,由代码定骨架、LLM 只起标题)
PAGE_TYPES = (
"title", "outline", "section", "background", "method", "result", "conclusion", "reference",
)
CONTENT_TYPES = ("background", "method", "result", "conclusion")
# 决策 2:时长 → [下限, 上限](含封面/目录,defense 含参考文献页)
PAGE_TARGETS: dict[int, tuple[int, int]] = {
10: (8, 10),
15: (11, 14),
20: (15, 18),
30: (22, 26),
}
MINUTES_CHOICES = tuple(sorted(PAGE_TARGETS))
# 场景骨架:内容页权重(reference 只有答辩要);代码算配额,LLM 只填内容
_SKELETONS: dict[str, dict[str, Any]] = {
"group_meeting": {"weights": {"background": 0.25, "method": 0.40, "result": 0.35}, "reference": False},
"proposal": {"weights": {"background": 0.40, "method": 0.45, "result": 0.15}, "reference": False},
"conclusion_report": {"weights": {"background": 0.20, "method": 0.30, "result": 0.50}, "reference": False},
"defense": {"weights": {"background": 0.25, "method": 0.35, "result": 0.40}, "reference": True},
"conference_talk": {"weights": {"background": 0.25, "method": 0.35, "result": 0.40}, "reference": False},
}
# 索引切块(正文 ≤150K 字符 → 块数 ≤5,控制 map 调用次数)
_MAX_SECTIONS = 5
_PARAS_PER_SECTION = 40
_PARA_CHAR_CAP = 1000 # 单段送 map 的截断(pymupdf 块通常 <1000 字符)
_SECTION_CHAR_CAP = 30000 # 单块送 map 的总预算
_MAX_CLAIMS_TOTAL = 50
_MAX_FIGURES_TOTAL = 15
_REDUCE_PAGE_SLICE = 14 # 单次 reduce 页数上限(>14 页分两片,守住 8K 输出)
_MAP_TIMEOUT = 75.0
_REDUCE_TIMEOUT = 200.0
_EXPAND_TIMEOUT = 75.0
_MAP_CONCURRENCY = 2 # BFF 读超时 360s 硬顶:map 并发压总时长
class OutlineError(Exception):
"""大纲生成失败(VALIDATION | LLM_ERROR | LLM_INVALID)。"""
def __init__(self, code: str, message: str) -> None:
super().__init__(message)
self.code = code
self.message = message
# ---------- 决策 2:页数配额(纯函数) ----------
def page_plan(scenario: str, minutes: int) -> dict[str, tuple[int, int]]:
"""时长目标页数 + 场景骨架 → 每 type 配额 (min, max)。
固定页:title=1、outline=1、conclusion=1(defense +reference=1);
其余按场景权重最大余数法分配(每内容页保底 1)。LLM 只能在配额内填,
总页数另外夹在 PAGE_TARGETS[minutes] 区间内。
"""
if scenario not in _SKELETONS:
raise ValueError(f"未知场景: {scenario}")
if minutes not in PAGE_TARGETS:
raise ValueError(f"不支持的时长: {minutes}(可选 {list(MINUTES_CHOICES)})")
lo, hi = PAGE_TARGETS[minutes]
target = (lo + hi) // 2
weights = _SKELETONS[scenario]["weights"]
n_ref = 1 if _SKELETONS[scenario]["reference"] else 0
# 章节分隔页:≥15 分钟(target≥11)每个内容组一张;10 分钟小 deck 不放分隔页
n_sec = len(weights) if target >= 11 else 0
rest = max(target - (3 + n_ref + n_sec), len(weights)) # 3 = title+outline+conclusion
raw = {t: w * rest for t, w in weights.items()}
alloc = {t: int(v) for t, v in raw.items()}
for t in sorted(raw, key=lambda k: raw[k] - alloc[k], reverse=True)[: rest - sum(alloc.values())]:
alloc[t] += 1
for t in alloc:
alloc[t] = max(1, alloc[t]) # 内容页保底
plan: dict[str, tuple[int, int]] = {
"title": (1, 1), "outline": (1, 1), "conclusion": (1, 1),
}
if n_sec:
plan["section"] = (n_sec, n_sec)
if n_ref:
plan["reference"] = (1, 1)
for t, a in alloc.items():
plan[t] = (a, a + 1)
return plan
# ---------- 段落索引(para_id:S{块}-P{段}) ----------
def build_index(text: str) -> tuple[list[dict[str, Any]], dict[str, str]]:
"""纯文本 → (blocks, index)。blocks 含段落ID标注文本(map 输入),index 为
para_id → 原文(evidence 校验用)。块按固定段数均分(不做标题猜测,
确定性可测;S 编号 = 块序号,P 编号 = 块内段序)。"""
paras = [ln.strip() for ln in text.splitlines() if ln.strip()]
paras = [p[:_PARA_CHAR_CAP] for p in paras]
if not paras:
return [], {}
n_sections = max(1, min(_MAX_SECTIONS, math.ceil(len(paras) / _PARAS_PER_SECTION)))
size = math.ceil(len(paras) / n_sections)
blocks: list[dict[str, Any]] = []
index: dict[str, str] = {}
for si in range(0, len(paras), size):
sec_no = si // size + 1
lines = paras[si : si + size]
paras_out = []
for pi, p in enumerate(lines, 1):
pid = f"S{sec_no}-P{pi}"
paras_out.append({"para_id": pid, "text": p})
index[pid] = p
blocks.append({
"section_id": f"S{sec_no}",
"paras": paras_out,
"text": "\n".join(f"[{v['para_id']}] {v['text']}" for v in paras_out),
})
return blocks, index
# ---------- map:分块抽 claims / figures ----------
_MAP_SYSTEM = """你是植物学论文精读助手。输入是一个论文文本块,每段以 [段落ID] 标注。
只输出单个 JSON 对象:
{"claims": [{"claim": "一句话事实陈述", "para_id": "S1-P2", "quote": "支撑该事实的原文逐字引用"}],
"figures": [{"label": "Fig 2", "para_id": "S1-P5-Fig2"}]}
硬性要求:
- claim 必须能被同条 quote 直接支撑;quote 必须逐字来自对应段落(≤60字、单句、不得跨段拼接、不得改写);
- para_id 必须照抄输入标注;figure 的 para_id = 其 caption 所在段落ID + "-FigN"(N 为图序号);
- 每块最多 8 条 claims、4 个 figures;没有实质内容就返回空数组;严禁编造引用、数据、图号;
- 不要输出 JSON 以外的任何文字。"""
async def _map_block(block: dict[str, Any]) -> tuple[list[dict], list[dict]]:
payload = await llm.complete_json(
_MAP_SYSTEM, _map_block_user(block), temperature=0.2,
max_tokens=4000, timeout=_MAP_TIMEOUT,
)
claims = [c for c in (payload.get("claims") or []) if isinstance(c, dict)][:8]
figures = [f for f in (payload.get("figures") or []) if isinstance(f, dict)][:4]
return claims, figures
def _map_block_user(block: dict[str, Any]) -> str:
body = block["text"]
if len(body) > _SECTION_CHAR_CAP:
body = body[:_SECTION_CHAR_CAP]
return f"论文文本块 {block['section_id']}:\n{body}"
# ---------- reduce:页大纲(配额内填内容) ----------
_REDUCE_SYSTEM = """你是学术汇报策划。根据给出的论文摘要卡片(claims/figures 已带逐字引用)、
论文真实图表清单与页数配额,生成 PPT 大纲。只输出单个 JSON 对象:
{"pages": [{"type": "...", "title": "...", "subtitle": "...", "bullets": ["..."],
"evidence": [{"quote": "..."}],
"speaker_notes": "...", "suggested_figures": [{"label": "Fig 2", "para_id": "S1-P12"}],
"highlight": {"value": "+0.82 cm", "label": "体长增长"},
"table": {"headers": ["处理", "株高(cm)"], "rows": [["对照", "10.2"]]}}]}
硬性要求:
- type 只能取:title, outline, section, background, method, result, conclusion, reference;
- 结构顺序固定:title → outline →(section → 对应内容组)×3 → conclusion(→ reference,若配额里有);
type=section 表示章节分隔页:放在 background、method、result 各组第一张之前,
title 只用组名(如「研究背景」,2~6 字,末尾不得带「是/的/了」等虚词,不要写成句子),
subtitle 必给(分隔页副标题,渲染在主标题下方):照抄该章第一张内容页的 title
(如「黄瓜遗传改良亟需高精度泛基因组」,≤20 字、一句话、与该页标题逐字一致),
bullets 用空数组(副标题走 subtitle 字段,不要再塞进 bullets),
evidence/speaker_notes 可省略;
- 每个 type 的页数必须落在给定配额 [min,max] 内,不得自创结构;
- 标题即结论:内容页 title 写成一句话结论或核心问题(≤20 字,看标题就懂这页讲什么),
「背景/方法/结果」这类栏目词只允许出现在 section 分隔页;
- 叙事:方法页讲机制与做法,结果页说明「证明了什么」而不是罗列做了什么;
- bullets 每页 3-5 条(title 页放作者/单位/来源,outline 页列各部分标题,
reference 页列 5-8 条文献);单条 ≤40 字、名词短语化、不写整句;
关键词强调用 **双星号** 包裹(如 **盐胁迫**);子要点写成独立数组元素并以「- 」开头
(如 "- 子要点",其 evidence 填 null);
- 版面标记(生成端只按此写法,渲染端自动识别,勿在同一页混用多种):
· 并列短要点:方法/结果/结论页若内容就是 2-4 个并列短语,每条 ≤16 字、不带任何标记,
直接写成这样的短句条目(会渲染为卡片网格);
· 关键词行:背景/方法概述页 ≥3 条统一用「关键词:说明」句式(如「靶基因:SOS1」);
· 流程:步骤页 ≥3 条统一以「→ 」开头(如「→ 提取 RNA」),会渲染为流程卡;
· 强调框:全篇最多 1-2 条最核心结论,以「> 」开头单独成条放在该页最后,
其 evidence 填 null(会渲染为底部强调引用框);
· 表格:结果对比页优先用 table 字段(headers ≤5 列、rows ≤4 行、单元格 ≤18 字)
代替 bullets;该页 bullets 最多 2 条导读,表格里的数据引用写进 speaker_notes;
- evidence 与 bullets 等长对齐(一一对齐,没有来源的位置填 null);
每条 evidence 只需要 "quote" 字段:从摘要卡片里逐字照抄 quote 原文;
**不需要写 para_id,后端会按 quote 自动反查真实段落ID**(你写错会干扰校验);
卡片里找不到对应 quote 的一律填 null,严禁改写或编造;
- speaker_notes 是讲稿口吻(80-160 字),可自然转述 quote 但不得出现段落ID;
- suggested_figures 只能从用户消息给出的「论文真实图表清单」里照抄 label 与 para_id,
清单为空则一律给空数组;你只决定"哪个图放哪页",绝不发明图号;
- result 组最后 1-2 页若呈现对比数据,给该页加 "highlight": {"value": "≤12字符的数字",
"label": "≤16字符的说明"}(如 {"value": "+0.82 cm", "label": "体长增长"});
没有可突出的数字对比就省略 highlight 字段;
- 严禁编造引用、数据、图表编号。不要输出 JSON 以外的任何文字。"""
def _plan_table(plan: dict[str, tuple[int, int]], types: tuple[str, ...] | list[str]) -> str:
return "\n".join(f"- {t}: {plan[t][0]}~{plan[t][1]} 页" for t in types)
def _narrow_figure_index(
figure_index: list[dict], available: list[str] | None,
) -> list[dict]:
"""v3 修复2:把注入 prompt 的图表清单收窄到「已提取原图」的 label。
available 为空/None → 不收窄(text 模式保持全清单占位框);
交集为空(提取 label 与扫描不一致等意外)→ 回退全清单,绝不把图清空。
匹配按 canon_label(kind+编号)进行,且把 entry.label 统一成磁盘文件名形态
("Fig. 6b" → "Fig 6",子图后缀无对应文件),同编号去重只留首个。
"""
if not available or not figure_index:
return figure_index
by_key: dict[tuple[str, int], str] = {}
for lbl in available:
key = canon_label(lbl)
if key and key not in by_key:
by_key[key] = str(lbl).strip()
if not by_key:
return figure_index
narrowed: list[dict] = []
seen: set[tuple[str, int]] = set()
for f in figure_index:
key = canon_label(f.get("label"))
if key in by_key and key not in seen:
entry = dict(f)
entry["label"] = by_key[key] # LLM 所见 = 磁盘可嵌形态
narrowed.append(entry)
seen.add(key)
return narrowed or figure_index
def _claims_block(claims: list[dict], figures: list[dict]) -> str:
# para_id 不下发给 reduce/expand(LLM 会照抄编造;quote 由后端反查真实段落ID)
compact = [
{"claim": str(c.get("claim") or "")[:200],
"quote": str(c.get("quote") or "")[:80]}
for c in claims
]
figs = [
{"label": str(f.get("label") or "")[:40], "para_id": str(f.get("para_id") or "")}
for f in figures
]
return json.dumps({"claims": compact, "figures": figs}, ensure_ascii=False)
def _reduce_user(
scenario: str, minutes: int, plan: dict[str, tuple[int, int]],
claims: list[dict], figures: list[dict],
types: list[str], *, prior_titles: list[str] | None = None, total_hint: int | None = None,
figure_index: list[dict] | None = None,
) -> str:
parts = [
f"场景:{SCENARIO_LABELS[scenario]}({scenario}),汇报时长 {minutes} 分钟,"
f"目标总页数 {PAGE_TARGETS[minutes][0]}~{PAGE_TARGETS[minutes][1]} 页。",
f"本次生成的页类型与配额(按顺序):\n{_plan_table(plan, types)}",
]
if total_hint is not None:
parts.append(f"本次需生成 {total_hint} 页。")
if prior_titles:
parts.append("前半部分已生成的页标题(保持连贯,不要重复):" + ";".join(prior_titles))
parts.append(f"摘要卡片:\n{_claims_block(claims, figures)}")
# P5 增强 1:论文真实图表清单注入(LLM 只选不编)
if figure_index:
compact = [
{"label": f.get("label"), "para_id": f.get("para_id"),
"caption": str(f.get("caption") or "")[:80]}
for f in figure_index
]
parts.append(
"论文中真实存在的图表清单(suggested_figures 只能从这里选,label 与 para_id 照抄):\n"
+ json.dumps(compact, ensure_ascii=False)
)
else:
parts.append("论文中未扫描到任何图表编号:suggested_figures 一律给空数组。")
return "\n\n".join(parts)
# ---------- 结构裁决(纯函数) ----------
def coerce_outline(payload: Any) -> list[dict]:
"""LLM 返回 → 规范化 pages(枚举/字段/bullets-evidence 对齐/slide_number)。"""
pages_raw = payload.get("pages") if isinstance(payload, dict) else None
if not isinstance(pages_raw, list):
raise OutlineError("LLM_INVALID", "LLM 输出缺少 pages 列表")
pages: list[dict] = []
for raw in pages_raw:
if not isinstance(raw, dict):
continue
ptype = str(raw.get("type") or "")
if ptype not in PAGE_TYPES:
continue
title = str(raw.get("title") or "").strip() or ptype
if ptype == "section":
# v3 修复3:LLM 偶发把「是」等虚词缀进组名(如「研究方法是」)
title = re.sub(r"[是的了]$", "", title) or title
bullets = [
str(b).strip() for b in (raw.get("bullets") or [])
if isinstance(b, (str, int, float)) and str(b).strip()
]
ev_raw = raw.get("evidence")
evidence: list[dict | None] = []
for i in range(len(bullets)):
item = ev_raw[i] if isinstance(ev_raw, list) and i < len(ev_raw) else None
if isinstance(item, dict) and str(item.get("quote") or "").strip():
evidence.append({
"para_id": str(item.get("para_id") or ""),
"quote": str(item.get("quote") or "").strip(),
"status": "unverified",
})
else:
evidence.append(None)
figs_raw = raw.get("suggested_figures")
figs = []
if isinstance(figs_raw, list):
for f in figs_raw[:6]:
if isinstance(f, dict) and str(f.get("para_id") or "").strip():
figs.append({
"label": str(f.get("label") or "").strip(),
"para_id": str(f.get("para_id") or "").strip(),
"status": "unverified",
})
# P5 增强 3:大数字突出页标记(result 组对比数据页,可选)
hl_raw = raw.get("highlight")
highlight = None
if isinstance(hl_raw, dict) and str(hl_raw.get("value") or "").strip():
highlight = {
"value": str(hl_raw.get("value") or "").strip()[:24],
"label": str(hl_raw.get("label") or "").strip()[:32],
}
# P5 二次开发 2.2:对比表格(可选;rows 超限截断,坏结构直接丢弃)
table = None
tbl_raw = raw.get("table")
if isinstance(tbl_raw, dict) and isinstance(tbl_raw.get("rows"), list):
headers = [
str(h).strip()[:24] for h in (tbl_raw.get("headers") or [])
if isinstance(h, (str, int, float)) and str(h).strip()
][:8]
rows: list[list[str]] = []
for r in tbl_raw["rows"][:12]:
if not isinstance(r, (list, tuple)):
continue
cells = [
str(c).strip()[:60] for c in r
if isinstance(c, (str, int, float))
][:8]
if any(cells):
rows.append(cells)
if rows:
table = {"headers": headers, "rows": rows}
page = {
"slide_number": 0,
"type": ptype,
"title": title,
"bullets": bullets,
"evidence": evidence,
"speaker_notes": str(raw.get("speaker_notes") or "").strip(),
"suggested_figures": figs,
}
if ptype == "section":
# P5 改进 2:分隔页副标题独立字段(LLM 给 subtitle;旧大纲仍在 bullets[0],
# 编辑器改 bullets[0] 也算数 → bullets 优先,避免编辑被旧 subtitle 覆盖)
page["subtitle"] = (bullets[0] if bullets else "") or str(
raw.get("subtitle") or ""
).strip()
if highlight:
page["highlight"] = highlight
if table:
page["table"] = table
pages.append(page)
if not pages:
raise OutlineError("LLM_INVALID", "LLM 输出中没有可用页(type 全部非法)")
return pages
# 超页裁剪优先级:section/reference 可整页删(floor 0),内容页保底 1 页;
# title/outline/conclusion 永不删
_DROP_PRIORITY = ("section", "reference", "background", "method", "result")
_FLOOR_ZERO = ("section", "reference")
def trim_to_max(
pages: list[dict], hi: int, warnings: list[str],
floors: dict[str, int] | None = None,
) -> list[dict]:
"""总页数超上限:按 _DROP_PRIORITY 优先级删该类型的最后一页;不凑下限。
floors(v3 修复3):按配额给类型设保底(如 section=(3,3) → 传 {"section": 3})。
有保底时先删「超配额的内容页/文献页」,section 不再被一刀清空;
全部 floor 都满足却仍超页 → 才回退到 section/reference 整页删(原语义兜底)。
"""
floors = floors or {}
floor_of = lambda t: floors.get(t, 0 if t in _FLOOR_ZERO else 1) # noqa: E731
while len(pages) > hi:
counts: dict[str, int] = {}
for p in pages:
counts[p["type"]] = counts.get(p["type"], 0) + 1
drop_type = next(
(t for t in _DROP_PRIORITY if counts.get(t, 0) > floor_of(t)),
None,
)
if drop_type is None:
# 配额 floor 全满足仍超页 → 兜底砍可整删类型(section/reference 优先)
drop_type = next(
(t for t in _DROP_PRIORITY
if t in _FLOOR_ZERO and counts.get(t, 0) > 0),
None,
)
if drop_type is None:
break
idx = max(i for i, p in enumerate(pages) if p["type"] == drop_type)
removed = pages.pop(idx)
warnings.append(
f"页数超过 {hi} 页上限,已移除 {removed['type']} 页「{removed['title']}」"
)
return pages
def split_oversized(pages: list[dict], warnings: list[str]) -> list[dict]:
"""内容页 bullets >8:拆成两页(续页 title+"(续)"),递归直到全 ≤8。"""
out: list[dict] = []
for page in pages:
todo = [page]
while todo:
cur = todo.pop(0)
bullets = cur.get("bullets") or []
if cur["type"] in CONTENT_TYPES and len(bullets) > 8:
k = math.ceil(len(bullets) / 2)
ev_full = list(cur.get("evidence") or [])
if len(ev_full) < len(bullets):
ev_full += [None] * (len(bullets) - len(ev_full))
nxt = json.loads(json.dumps(cur)) # 深拷贝
nxt["bullets"] = bullets[k:]
nxt["evidence"] = ev_full[k:]
nxt["title"] = f"{cur['title']}(续)"
nxt["suggested_figures"] = [] # 图占位只放首页
nxt["speaker_notes"] = ""
cur["bullets"] = bullets[:k]
cur["evidence"] = ev_full[:k]
warnings.append(f"「{cur['title'].removesuffix('(续)')}」要点超过 8 条,已拆成两页")
todo.insert(0, nxt)
todo.insert(0, cur)
continue
out.append(cur)
return out
def renumber(pages: list[dict]) -> list[dict]:
for i, p in enumerate(pages, 1):
p["slide_number"] = i
return pages
# ---------- 证据程序化校验(纯函数) ----------
_FIG_RE = re.compile(r"^(S\d+-P\d+)(?:-Fig(\d+))?$", re.IGNORECASE)
def _norm(s: str) -> str:
"""归一化:NFKC + 小写 + 去掉全部空白与标点(只留字母数字/CJK)。"""
s = unicodedata.normalize("NFKC", s).lower()
return re.sub(r"[\W_]+", "", s, flags=re.UNICODE)
def _concat_segments(norm_index: dict[str, str]) -> tuple[str, dict[str, tuple[int, int]]]:
"""按 document 顺序把各段归一化文本拼成一条,并给出每段在拼接串中的 [start, end)。
LLM 照抄的 quote 常横跨两个段落边界 → 任何单段都不含它,但整篇文档含。
repair/validate 据此做跨段宽松反查(v3 修复4)。
"""
offsets: dict[str, tuple[int, int]] = {}
chunks: list[str] = []
pos = 0
for pid, nt in norm_index.items():
offsets[pid] = (pos, pos + len(nt))
chunks.append(nt)
pos += len(nt)
return "".join(chunks), offsets
def _locate_span(
nq: str, concat: str, offsets: dict[str, tuple[int, int]],
) -> str | None:
"""跨段定位:nq 起点落在哪个段(取起点所在段为归属)。≥10 归一化字符才启用。"""
if len(nq) < 10:
return None
i = concat.find(nq)
if i < 0:
return None
for pid, (start, end) in offsets.items():
if start <= i < end:
return pid
return None
def repair_evidence_para_ids(pages: list[dict], index: dict[str, str]) -> int:
"""按 quote 归一化子串反查真实 para_id(P5 修复:LLM 常编 para_id,quote 往往逐字正确)。
LLM reduce/expand 只需给 quote(prompt 已声明后端反查);这里按 document
顺序找第一个包含该 quote 的段落填回 para_id,后续 validate_evidence 据此标
verified。找不到 → 不动,留给 validate_evidence 标 unverified(真幻觉)。
返回修复条数。
"""
norm_index = {pid: _norm(t) for pid, t in index.items()}
concat, offsets = _concat_segments(norm_index)
repaired = 0
for page in pages:
for ev in page.get("evidence") or []:
if not isinstance(ev, dict):
continue
quote = str(ev.get("quote") or "")
nq = _norm(quote)
if not nq:
continue
pid = str(ev.get("para_id") or "")
if pid and nq in norm_index.get(pid, ""):
continue # 标注已正确,不动
hit = next((p for p, nt in norm_index.items() if nq in nt), None)
if not hit:
# 跨段 quote:任何单段都不含,但在整篇拼接串里 → 归属起点所在段
hit = _locate_span(nq, concat, offsets)
if hit:
ev["para_id"] = hit
repaired += 1
return repaired
def validate_evidence(
pages: list[dict], index: dict[str, str],
figure_index: list[dict] | None = None,
) -> list[str]:
"""逐条校验 evidence / suggested_figures → status + warnings。
figure_index(论文真实图表清单)非空时,图引用按清单裁决:
- label 不在清单 → 直接丢弃(不是标 unverified,结果里不会出现论文没有的图);
- label 在清单 → 以清单里的真实 para_id 为准,status=verified。
为 None 时保持旧路径(格式 + 原文 fig{N} 子串校验),兼容无清单的调用方。
"""
warnings: list[str] = []
norm_index = {pid: _norm(t) for pid, t in index.items()}
doc_all = "".join(norm_index.values())
concat, offsets = _concat_segments(norm_index)
fmap = figure_map(figure_index) if figure_index is not None else None
for page in pages:
sn = page["slide_number"]
evs = page.get("evidence") or []
ver = unver = 0
for ev in evs:
if ev is None:
continue
quote = str(ev.get("quote") or "")
pid = str(ev.get("para_id") or "")
nq, np_ = _norm(quote), norm_index.get(pid)
if quote and np_ is not None and nq and nq in np_:
ev["status"] = "verified"
ver += 1
elif quote and nq and pid and _locate_span(nq, concat, offsets) == pid:
# 跨段 quote:repair 已把 para_id 指到起点段 → 等同逐字命中
ev["status"] = "verified"
ver += 1
else:
ev["status"] = "unverified"
unver += 1
if nq and nq in doc_all:
warnings.append(
f"第{sn}页:引用真实存在,但段落标注 {pid or '(空)'} 不符(已标 unverified)"
)
total = ver + unver
if page["type"] in ("result", "conclusion") and total and unver / total > 0.5:
warnings.append(
f"第{sn}页({page['type']})证据未核实占比超过 50%({unver}/{total}),请人工复核"
)
kept: list[dict] = []
for fig in page.get("suggested_figures") or []:
if fmap is not None:
key = canon_label(fig.get("label"))
entry = fmap.get(key) if key else None
if entry is None:
warnings.append(
f"第{sn}页:图引用 {fig.get('label') or '(空)'} 不在论文真实图表清单,已丢弃"
)
continue
fig["para_id"] = entry["para_id"] # 以扫描出的真实位置为准
fig["status"] = "verified"
kept.append(fig)
continue
m = _FIG_RE.match(str(fig.get("para_id") or ""))
if not m:
fig["status"] = "unverified"
warnings.append(f"第{sn}页:图引用 {fig.get('para_id')} 格式不合法")
kept.append(fig)
continue
base, num = m.group(1).upper(), m.group(2)
base_norm = norm_index.get(base)
fig_ok = base_norm is not None and (
num is None or f"fig{num}" in base_norm or f"figure{num}" in base_norm
)
fig["status"] = "verified" if fig_ok else "unverified"
if not fig_ok:
warnings.append(
f"第{sn}页:图引用 {fig['para_id']} 未在原文对应段落找到(已标 unverified)"
)
kept.append(fig)
page["suggested_figures"] = kept
return warnings
# ---------- 扩充重试(内容页 bullets <2 → 一次按页补齐) ----------
_EXPAND_SYSTEM = """你是学术汇报编辑。某几页幻灯片的要点不足(少于 2 条)。
基于给出的页信息与摘要卡片(claims 已带逐字引用),为每页补齐到 3-6 条要点。
只输出单个 JSON 对象:
{"pages": [{"index": 0, "bullets": ["..."], "evidence": [{"quote": "..."}]}]}
硬性要求:
- index 是给定页在列表中的下标;bullets 保留原有要点(可精炼改写)并补充同主题新要点;
- evidence 与 bullets 等长对齐;每条 evidence 只需要 "quote" 字段,从卡片里逐字照抄,
**不需要写 para_id,后端会按 quote 自动反查**;没有依据填 null;
- 单条 ≤60 字;严禁编造。不要输出 JSON 以外的任何文字。"""
async def _expand_thin(
pages: list[dict], thin_idx: list[int],
claims: list[dict], figures: list[dict], warnings: list[str],
) -> None:
"""一次 LLM 调用补齐所有薄弱页;失败仅 warning,不阻断。"""
page_info = [
{
"index": i,
"type": pages[i]["type"],
"title": pages[i]["title"],
"bullets": pages[i]["bullets"],
"speaker_notes": pages[i]["speaker_notes"],
}
for i in thin_idx
]
user = (
f"待补齐页(下标即列表位置):\n{json.dumps(page_info, ensure_ascii=False)}\n\n"
f"摘要卡片:\n{_claims_block(claims, figures)}"
)
try:
payload = await llm.complete_json(
_EXPAND_SYSTEM, user, temperature=0.3,
max_tokens=4000, timeout=_EXPAND_TIMEOUT,
)
except LLMError as exc:
warnings.append(f"薄弱页扩充重试失败(保留原状):{exc}")
return
for item in payload.get("pages") or []:
if not isinstance(item, dict):
continue
idx = item.get("index")
if not isinstance(idx, int) or not (0 <= idx < len(pages)) or idx not in thin_idx:
continue
page = pages[idx]
bullets = [
str(b).strip() for b in (item.get("bullets") or [])
if isinstance(b, (str, int, float)) and str(b).strip()
][:8]
if len(bullets) < 2:
continue
ev_raw = item.get("evidence")
evidence: list[dict | None] = []
for i in range(len(bullets)):
e = ev_raw[i] if isinstance(ev_raw, list) and i < len(ev_raw) else None
if isinstance(e, dict) and str(e.get("quote") or "").strip():
evidence.append({
"para_id": str(e.get("para_id") or ""),
"quote": str(e.get("quote") or "").strip(),
"status": "unverified",
})
else:
evidence.append(None)
page["bullets"] = bullets
page["evidence"] = evidence
# ---------- 主编排 ----------
async def generate_outline(
text: str, scenario: str, minutes: int, style_id: str,
*,
available_figures: list[str] | None = None,
) -> tuple[list[dict], list[str], dict[str, Any]]:
"""生成完整大纲。返回 (pages, warnings, meta);失败抛 OutlineError。
available_figures(v3 修复2):PDF 入口已提取到磁盘的原图 label 列表。
非空时只把「已提取」的图注入 reduce/expand(LLM 选了 build 就能 add_picture
嵌入,不再出虚线占位);None/空 → 保持全清单(text 模式占位框不变)。
"""
if scenario not in SCENARIOS:
raise OutlineError("VALIDATION", f"未知场景 scenario={scenario}")
if minutes not in PAGE_TARGETS:
raise OutlineError("VALIDATION", f"不支持的时长 {minutes} 分钟(可选 {list(MINUTES_CHOICES)})")
if style_id not in STYLE_IDS:
raise OutlineError("VALIDATION", f"未知风格 style_id={style_id}")
text = (text or "").strip()
if len(text) < 300:
raise OutlineError("VALIDATION", "正文过短(<300 字),无法生成汇报大纲")
settings = get_settings()
if len(text) > settings.max_text_chars:
text = text[: settings.max_text_chars]
blocks, index = build_index(text)
if not blocks:
raise OutlineError("VALIDATION", "正文解析为空,无法生成汇报大纲")
# P5 增强 1:论文真实图表清单(正则扫描,非 LLM)——供 reduce 只选不编
figure_index = scan_figure_index(index)
# v3 修复2:有已提取原图 → 注入清单收窄到「磁盘上有图」的条目(交集空则回退全清单)
inject_index = _narrow_figure_index(figure_index, available_figures)
fmap_pick = figure_map(inject_index)
plan = page_plan(scenario, minutes)
lo, hi = PAGE_TARGETS[minutes]
warnings: list[str] = []
llm_calls = 0
# ---- map(并发 2,控总时长压在 BFF 360s 内) ----
claims: list[dict] = []
figures: list[dict] = []
sem = asyncio.Semaphore(_MAP_CONCURRENCY)
async def _map_one(block: dict[str, Any]) -> tuple[list[dict], list[dict]]:
async with sem:
return await _map_block(block)
try:
results = await asyncio.gather(*(_map_one(b) for b in blocks))
except LLMError as exc:
raise OutlineError("LLM_ERROR", f"章节摘要失败:{exc}") from exc
llm_calls += len(results)
for cs, fs in results:
for c in cs:
if len(claims) < _MAX_CLAIMS_TOTAL:
claims.append(c)
for f in fs:
if len(figures) < _MAX_FIGURES_TOTAL:
figures.append(f)
# P5 增强 1:map 抽到的图先过一遍注入清单(不在清单 → 丢弃;在 → 用真实 para_id)
if fmap_pick:
filtered: list[dict] = []
for f in figures:
key = canon_label(f.get("label"))
entry = fmap_pick.get(key) if key else None
if entry:
f = dict(f, para_id=entry["para_id"])
filtered.append(f)
if len(filtered) != len(figures):
warnings.append(
f"map 阶段 {len(figures) - len(filtered)} 个图引用不在论文真实图表清单,已丢弃"
)
figures = filtered
else:
if figures:
warnings.append(f"论文未扫描到图表编号,丢弃 map 阶段 {len(figures)} 个图引用")
figures = []
if not claims:
raise OutlineError("LLM_INVALID", "未能从正文抽取到任何可引用事实(claims 为空)")
# ---- reduce(>14 页分两片,第二片带上一片标题连贯) ----
full_types = [t for t in PAGE_TYPES if t in plan]
split_at = len(full_types)
acc = 0
for i, t in enumerate(full_types):
acc += plan[t][1]
if acc >= _REDUCE_PAGE_SLICE:
split_at = i + 1
break
first_types, second_types = full_types[:split_at], full_types[split_at:]
def _types_quota(ts: list[str]) -> int:
return sum(plan[t][0] for t in ts)
pages: list[dict] = []
try:
payload = await llm.complete_json(
_REDUCE_SYSTEM,
_reduce_user(scenario, minutes, plan, claims, figures, first_types,
figure_index=inject_index),
temperature=0.4, max_tokens=8000, timeout=_REDUCE_TIMEOUT,
)
llm_calls += 1
pages = coerce_outline(payload)
if second_types:
prior_titles = [p["title"] for p in pages]
payload2 = await llm.complete_json(
_REDUCE_SYSTEM,
_reduce_user(
scenario, minutes, plan, claims, figures, second_types,
prior_titles=prior_titles, total_hint=_types_quota(second_types),
figure_index=inject_index,
),
temperature=0.4, max_tokens=8000, timeout=_REDUCE_TIMEOUT,
)
llm_calls += 1
pages.extend(coerce_outline(payload2))
except LLMError as exc:
msg = str(exc)
if "JSON" in msg or "json" in msg:
raise OutlineError("LLM_INVALID", f"LLM 大纲输出无法解析:{msg}") from exc
raise OutlineError("LLM_ERROR", f"大纲生成失败:{msg}") from exc
# ---- 后校验:拆页 → 夹页数 → 扩充 → 证据 → 重编号 ----
pages = split_oversized(pages, warnings)
renumber(pages)
# v3 修复3:按配额给全类型设保底(先删超配额的页;section/reference 不再被一刀清空)
floors = {t: mn for t, (mn, _mx) in plan.items() if mn}
pages = trim_to_max(pages, hi, warnings, floors=floors)
renumber(pages)
if len(pages) < lo:
warnings.append(
f"生成 {len(pages)} 页,低于 {minutes} 分钟目标下限 {lo} 页(不凑页,请人工补足)"
)
# 类型顺序归一:保持 LLM 输出顺序即可,但至少保证 title 在首位
pages.sort(key=lambda p: (0 if p["type"] == "title" else 1))
renumber(pages)
thin_idx = [
i for i, p in enumerate(pages)
if p["type"] in CONTENT_TYPES and len(p["bullets"]) < 2
]
if thin_idx:
await _expand_thin(pages, thin_idx, claims, figures, warnings)
for i in thin_idx:
if len(pages[i]["bullets"]) < 2:
warnings.append(f"第{pages[i]['slide_number']}页要点不足 2 条(扩充后仍不足)")
# P5 修复:LLM 只给 quote(或编 para_id)→ 按 quote 反查真实段落ID,再进校验
repaired = repair_evidence_para_ids(pages, index)
if repaired:
warnings.append(f"{repaired} 条 evidence 的段落标注已按 quote 反查修正")
warnings.extend(validate_evidence(pages, index, inject_index))
renumber(pages)
unverified_total = sum(
1 for p in pages for e in (p.get("evidence") or []) if e and e.get("status") == "unverified"
)
meta = {
"scenario": scenario,
"minutes": minutes,
"style_id": style_id,
"paragraphs": len(index),
"sections": len(blocks),
"claims": len(claims),
"figures": len(figure_index), # 真实图表清单规模(扫描所得)
"figures_used": sum( # 大纲里实际引用且通过校验的图数
1 for p in pages for f in (p.get("suggested_figures") or [])
if f.get("status") == "verified"
),
"llm_calls": llm_calls,
"target_pages": [lo, hi],
"unverified_evidence": unverified_total,
}
return pages, warnings, meta
+230
View File
@@ -0,0 +1,230 @@
"""services/ppt_preview.py —— pptx → PNG 页预览(PIL 自渲染,读回 python-pptx 对象)。
移植自交付脚本 /tmp/opencode/render_pptx_preview.py(本项目自有验证脚本),
用途:build 后给前端缩略图条 / 大图预览(浏览器内直接目检版式)。
非 PowerPoint 截图:几何/颜色/字体色忠实还原,字形以 Noto CJK 近似(预览够用)。
依赖仅 Pillow + python-pptx(均已在 tooling/requirements.lock),零新依赖。
约定:输出 out_dir/slide_{i:02d}.png(i 从 1 起),1280×720 @96dpi;
损坏/空 pptx → 返回 [](调用方降级,不抛)。
"""
from __future__ import annotations
import logging
import tempfile
from io import BytesIO
from pathlib import Path
from PIL import Image, ImageDraw, ImageFont
from pptx import Presentation
from pptx.enum.shapes import MSO_SHAPE_TYPE
from pptx.util import Emu
logger = logging.getLogger(__name__)
DPI = 96
SLIDE_W_IN, SLIDE_H_IN = 13.333, 7.5
# 字体 fallback 链:Noto CJK(中英混排)→ DejaVu(兜底,缺 CJK 也不崩)
_FONT_CANDIDATES_REG = (
"/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc",
"/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
)
_FONT_CANDIDATES_BOLD = (
"/usr/share/fonts/opentype/noto/NotoSansCJK-Bold.ttc",
"/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf",
)
def _first_existing(paths: tuple[str, ...]) -> str:
for p in paths:
if Path(p).is_file():
return p
return paths[-1]
_FONT_REG = _first_existing(_FONT_CANDIDATES_REG)
_FONT_BOLD = _first_existing(_FONT_CANDIDATES_BOLD)
def px(emu: int) -> int:
return int(Emu(emu).inches * DPI)
def font_for(pt: float, bold: bool) -> ImageFont.FreeTypeFont:
path = _FONT_BOLD if bold else _FONT_REG
return ImageFont.truetype(path, max(int(pt * DPI / 72), 6))
def run_style(run) -> tuple[float, bool, tuple[int, int, int]]:
f = run.font
pt = (f.size.pt if f.size else 18)
bold = bool(f.bold)
color = (0, 0, 0)
try:
if f.color and f.color.rgb is not None:
color = tuple(f.color.rgb)
except Exception: # noqa: BLE001 - 主题色等非直接 RGB,回退黑
pass
return pt, bold, color
def wrap_text(draw: ImageDraw.ImageDraw, text: str, font: ImageFont.ImageFont, max_w: float) -> list[str]:
lines: list[str] = []
cur = ""
for ch in text:
if ch == "\n":
lines.append(cur)
cur = ""
continue
if draw.textlength(cur + ch, font=font) > max_w and cur:
lines.append(cur)
cur = ch
else:
cur += ch
if cur:
lines.append(cur)
return lines or [""]
def draw_text_frame(draw, tf, box, *, v_anchor: str = "top") -> None:
"""box = (x, y, w, h);按段落逐 run 画(粗体分段 / level 缩进 / 居中)。"""
x, y, w, h = box
lines_all: list[tuple[str, ImageFont.ImageFont, tuple, int, str]] = []
for p in tf.paragraphs:
runs = p.runs
if not runs:
lines_all.append(("", font_for(14, False), (0, 0, 0), 0, str(p.alignment or "left")))
continue
segs = []
for r in runs:
pt, bold, col = run_style(r)
segs.append((r.text, pt, bold, col))
# 预览够用:整段拼接,取最大字号 + 任一粗体,色随首 run
joined = "".join(s[0] for s in segs)
max_pt = max(s[1] for s in segs)
any_bold = any(s[2] for s in segs)
col = segs[0][3]
indent = (p.level or 0) * 24
align = str(p.alignment or "left")
fnt = font_for(max_pt, any_bold)
for line in wrap_text(draw, joined, fnt, max(w - indent - 8, 20)):
lines_all.append((line, fnt, col, indent, align))
total_h = sum(int(f.size * 1.35) for _, f, _, _, _ in lines_all)
if v_anchor == "middle":
y = y + max((h - total_h) // 2, 0)
for line, fnt, col, indent, align in lines_all:
lw = draw.textlength(line, font=fnt)
if "CENTER" in align:
lx = x + (w - lw) / 2
elif "RIGHT" in align:
lx = x + w - lw - 6
else:
lx = x + indent + 4
draw.text((lx, y), line, font=fnt, fill=col)
y += int(fnt.size * 1.35)
def slide_bg_color(slide) -> tuple[int, int, int]:
try:
fill = slide.background.fill
if fill.type is not None and fill.fore_color and fill.fore_color.rgb is not None:
return tuple(fill.fore_color.rgb)
except Exception: # noqa: BLE001 - 无背景填充 → 白底
pass
return (255, 255, 255)
def render_slide(slide) -> Image.Image:
img = Image.new("RGB", (int(SLIDE_W_IN * DPI), int(SLIDE_H_IN * DPI)), slide_bg_color(slide))
d = ImageDraw.Draw(img)
for sh in slide.shapes:
try:
x, y, w, h = px(sh.left), px(sh.top), px(sh.width), px(sh.height)
except Exception: # noqa: BLE001 - 无定位形状(组合件等)跳过
continue
if sh.shape_type == MSO_SHAPE_TYPE.AUTO_SHAPE:
# 色块(分隔页横幅 / 封面强调条 / 引用框)
fc = None
try:
if sh.fill.type is not None and sh.fill.fore_color.rgb is not None:
fc = tuple(sh.fill.fore_color.rgb)
except Exception: # noqa: BLE001
fc = None
if fc:
d.rectangle([x, y, x + w, y + h], fill=fc)
else:
d.rectangle([x, y, x + w, y + h], outline=(150, 150, 150), width=2)
if sh.has_text_frame and sh.text_frame.text.strip():
draw_text_frame(d, sh.text_frame, (x, y, w, h), v_anchor="middle")
elif sh.shape_type == MSO_SHAPE_TYPE.PICTURE:
# 真图(Tier B contain 插入的原图)
try:
pic = Image.open(BytesIO(sh.image.blob)).convert("RGB")
img.paste(pic.resize((max(w, 1), max(h, 1))), (x, y))
except Exception: # noqa: BLE001 - 损坏图 → 灰块占位
d.rectangle([x, y, x + w, y + h], fill=(205, 205, 205))
elif getattr(sh, "has_table", False):
# 原生表格
tbl = sh.table
cw = [px(tbl.columns[c].width) for c in range(len(tbl.columns))]
rh = [px(tbl.rows[r].height) for r in range(len(tbl.rows))]
cy = y
for r, row_h in enumerate(rh):
cx = x
for c, col_w in enumerate(cw):
cell = tbl.cell(r, c)
try:
fc = tuple(cell.fill.fore_color.rgb)
except Exception: # noqa: BLE001
fc = (245, 245, 245)
d.rectangle([cx, cy, cx + col_w, cy + row_h], fill=fc, outline=(170, 170, 170))
draw_text_frame(d, cell.text_frame, (cx + 3, cy + 2, col_w - 6, row_h - 4))
cx += col_w
cy += row_h
elif sh.has_text_frame:
# 占位符 / 文本框:虚线图框判定
txt = sh.text_frame.text
if "请插入" in txt:
d.rectangle([x, y, x + w, y + h], outline=(150, 150, 150), width=3)
for i in range(x, x + w, 12): # 近似虚线
d.line([i, y, min(i + 6, x + w), y], fill=(150, 150, 150), width=3)
d.line([i, y + h, min(i + 6, x + w), y + h], fill=(150, 150, 150), width=3)
d.line([x, y, x, y + h], fill=(150, 150, 150), width=3)
d.line([x + w, y, x + w, y + h], fill=(150, 150, 150), width=3)
anchor = "middle" if "请插入" in txt else "top"
draw_text_frame(d, sh.text_frame, (x, y, w, h), v_anchor=anchor)
return img
def render_previews(pptx_path: Path, out_dir: Path) -> list[Path]:
"""draft.pptx → out_dir/slide_{i:02d}.png;失败/空/损坏 → [](绝不抛)。"""
if not pptx_path.is_file():
return []
try:
prs = Presentation(str(pptx_path))
except Exception as exc: # noqa: BLE001 - 坏文件降级(调用方 warning)
logger.warning("预览渲染:pptx 打开失败 %s:%s", pptx_path, exc)
return []
try:
out_dir.mkdir(parents=True, exist_ok=True)
# 清旧页:重建后页数变少时不留 slide_{n}.png 残影(懒渲染 GET 同样受益)
for stale in out_dir.glob("slide_*.png"):
stale.unlink(missing_ok=True)
written: list[Path] = []
for i, slide in enumerate(prs.slides, 1):
img = render_slide(slide)
final = out_dir / f"slide_{i:02d}.png"
# 原子写:先临时文件再 replace,避免并发 GET 读到半张图
with tempfile.NamedTemporaryFile(dir=out_dir, suffix=".png", delete=False) as tmp:
tmp_path = Path(tmp.name)
try:
img.save(tmp_path, format="PNG")
tmp_path.replace(final)
finally:
tmp_path.unlink(missing_ok=True)
written.append(final)
return written
except Exception as exc: # noqa: BLE001 - 渲染异常不阻断构建
logger.warning("预览渲染失败(该 job 无缩略图):%s", exc)
return []

Some files were not shown because too many files have changed in this diff Show More