Author SHA1 Message Date
admin 3c00e1675e feat(bioflow): skill 编排续作 WIP checkpoint(多轮链续接/plan 回填/job_id 提取/历史 blocks 文本提取) 2026-10-14 09:41:44 +08:00
admin 8ab4279b04 fix(bioflow): sendCardResult 从 raw.bioflow 读 active_job_id(BFF 归一化剥离顶层 bioflow 导致订阅没建立) 2026-10-13 14:39:33 +08:00
admin c943e8d363 fix(bioflow): sendCardResult 创建任务后补 connectJobEvents 订阅,恢复文件/参数引导等任务事件触发 2026-10-13 12:34:27 +08:00
admin 9ac36e99ba fix(bioflow): 路径选择卡空状态文案「暂无候选路径」改为「正在检索可用分析路径....」 2026-10-13 11:33:39 +08:00
admin 9cb0bf2e35 feat(bioflow): stage_done 摘要从 manifest.metrics 提取关键指标(命中/比对率等) (#010) 2026-10-13 11:28:18 +08:00
admin 6b4b5e571f feat(bioflow): 就绪卡+版本化卡接线——工具结果经 payload.bioflow 下发前端渲染 (#009) 2026-10-13 11:18:53 +08:00
admin 9797b1b414 feat(bioflow): 交付卡接线——job_done 构造 manifest 调 validate_delivery 下发交付清单 (#009-交付) 2026-10-13 11:12:39 +08:00
admin b9bc90ffb0 feat(bioflow): 诊断卡接线——stage_failed 补 G13 错误对象+guidance,前端诊断卡渲染 (#009-诊断) 2026-10-13 11:10:30 +08:00
admin 821f3c1c4e feat(bioflow): 引导帧——卡片环节补自然语言引导气泡(文件/参数/存档/进度) (#006/#007/#008/#010) 2026-10-13 11:04:32 +08:00
admin da8f531bbc fix(bioflow): check_disk_available 捕获 OSError 返回 PATH_NOT_EXIST,不再抛未捕获异常 (#004) 2026-10-13 11:01:17 +08:00
admin 21c79582bf fix(bioflow): QA 生信流加活跃任务续接分支,续接输入(提供文件/参数等)不再掉普通 QA (#011) 2026-10-13 10:59:54 +08:00
admin 1ecc4cb29e chore(deploy): 新增 bioflow_switch_restart.sh 切分支→重启→验证规范脚本 2026-10-13 10:28:06 +08:00
admin 75eb489fc3 fix(bioflow): main.py 加 load_dotenv 把 .env 注入 os.environ,修复规划 LLM 配置断层 2026-10-13 10:03:00 +08:00
admin e4ed3589d5 fix(bioflow): INCOMPLETE_DELIVERY 错误分类映射到 input/diagnosed (G14 补漏) 2026-10-12 12:20:52 +08:00
admin f27336e961 feat(bioflow): 生信 Skill 化改造 (CLI工具层+MCP薄适配+11分析工具+SKILL, G1-G23)
- A 地基: bioflowctl CLI 抽取 + M0 六条数据保护 gap (G1-G6)
- B: 结构化错误对象 G13 + MCP 薄适配
- D: 契约与就绪 G7-G11/G14/G15 (数据角色/下游直入/单一schema/就绪检查/任务绑定/错误分类/恢复预算)
- E: 版本化 G16-G20 (plan_revision/影响范围/历史不覆盖/防重复/执行所有权)
- F: 交付 G21-G23 (交付清单/报告范围/资源预算)
- C: 11 分析控制工具 + SKILL.md + install/uninstall (G12)
- UI: 4 张新增卡 (就绪检查/诊断/版本化修改/交付清单)
- 测试: 后端 724 + QA 167 + 前端 176 passed, 红线 pipeline.py/answer_composer.py 零修改
2026-10-11 16:03:19 +08:00
admin 963a87855c feat(bioflow): 生信模块对话式改造 + MCP 外挂化 (S0-S6)
S0: bioflow_service 三 MCP 端点(gene-family/rnaseq/extension) + 源文件获取/预处理
S1: QA bioflow_agent 前门 router + capabilities.yaml 三条目 + 熔断
S2: card_result 回填 + 多轮记忆 + session↔job 绑定
S3: 前端 TAB 双信号 + 卡片辅助化 + 进度块只读 + 删除直连
S4: 扩展子系统对话化(第三能力单元)
S5: toggle_bioflow.sh 装卸 + SettingsPanel 三开关
S6: 多配置回归 + 对话剧本测试手册
2026-10-11 10:45:25 +08:00
221 changed files with 13547 additions and 31438 deletions
+3 -13
View File
@@ -1,4 +1,4 @@
# 运行时产物与环境
# 运行时产物与环境
node_modules
dist/
.venv/
@@ -43,17 +43,7 @@ paper_writer_service/paper.db-shm
paper_writer_service/paper.db-wal
paper_writer_service/paper.log
# review_service 运行时产物(不入库)
review_service/review.db
review_service/review.db-wal
review_service/review.db-shm
review_service/review.log
review_service/data/reformat_jobs/
# 期刊种子 journals.json 是源码:顶层 **/data/ 会误伤 review_service/data/,显式放行
!review_service/data/
review_service/tools/tectonic
review_service/data/tectonic-cache/
review_service/data/format_cache/
# 部署方案文档(含主机凭据,不入库)
docs/DEPLOY_PAPER_176.md
# Python 覆盖率运行时产物(coverage.py --cov 生成)
.coveragevstore/vstore.log
-54
View File
@@ -1,54 +0,0 @@
# AGENTS.md —— 协作约束(科研审稿模块)
## 项目背景
本项目(PlantAI-rag-new)是植物基因组学文献知识工作台,拥有自研 RAG 与知识库
(Qdrant + Neo4j + vstore)。2026-10 起新增「科研投稿与智能审稿助手」功能
(审稿周期/期刊分析/期刊推荐/投稿前自审/审稿意见辅助/格式检查)。
## 审稿模块(review_service)约束 —— 必须遵守
1. **所有审稿相关新代码只允许写在 `review_service/` 目录下新增**。
- 服务自身:FastAPI 独立进程,端口 8100(模式同 `paper_writer_service` 8097 / `bioflow_service` 8098)。
- 环境变量前缀统一 `REVIEW_`;LLM 凭据复用 `DOCGEN_LLM_*` 裸键(回退链见 `review_service/app/services/llm.py`)。
2. **BFF(`sdk/`)层只允许条件注册代理路由**:
- 允许的全部改动 = `sdk/app/config.py` 增加 `review_enabled` / `review_service_url` 等开关键
+ `sdk/app/main.py` 按开关 `include_router(review_proxy.router)`
+ 新文件 `sdk/app/routers/review_proxy.py`(反代实现)。
- **严禁**修改 sdk 下其他 router/service/db 的任何既有逻辑。
3. **严禁修改现有 RAG 核心与既有服务代码**,包括但不限于:
- `agent/`(qa 8091 / methodology 8092 / legacy-rag 8093)
- `vstore/`(8094,审稿服务只经 HTTP 只读消费其 `/api/vstore/*` 接口)
- `paper_writer_service/`、`bioflow_service/`、`bioflow_kernel/`
- `ui/`(前端入口另行按 UI_SPEC 流程评审)
- `knowledge/`、`kb-libraries/`、`tooling/`(依赖锁另议)
4. **依赖纪律**:审稿服务只使用统一 venv(`tooling/requirements.lock`)中已有的库
(fastapi / httpx / pydantic-settings 等)。如确需新增全局依赖,
须先更新 `tooling/requirements.lock` 并经人工确认。
5. **上游对接只读**:PDF 解析走 vstore 8094(`POST /api/vstore/ingest` +
`GET/DELETE /api/vstore/documents/{id}`);LLM 走 DeepSeek OpenAI 兼容接口;
后续期刊数据/期刊推荐等子功能同样以新 service + 新路由在 `review_service/` 内扩展。
6. **format_check 例外**:format_check 允许用 pymupdf 本地解档获取版式元数据
(字体/边距/行距/页数),因为 vstore 只回文本不返回版式信息。正文提取仍走 vstore。
## 目录速览(审稿模块)
```
review_service/
├── app/main.py # FastAPI 入口(/health + 挂载业务路由)
├── app/config.py # pydantic-settings(REVIEW_ 前缀)
├── app/routers/presubmit.py # POST /api/review/presubmit(自审稿报告)
├── app/routers/format_check.py # POST /api/review/format-check(版式与规范检查)
├── app/services/llm.py # DeepSeek httpx 封装(complete_text/complete_json)
├── app/services/paper_parse.py # PDF → 纯文本(经 vstore 解析接口)
├── app/services/format_check.py# 版式规则引擎(pymupdf/docx 本地解档 + 规则检查)
├── restart.sh # 8100 重启脚本
└── pytest.ini
```
## 启动
```bash
cd review_service && bash restart.sh # 默认 8100,venv 从 ../plantai-rag-new-venv 推导
# BFF 开关:sdk/.env 加 REVIEW_ENABLED=1 后 bff_restart(反代 /api/review/*)
```
-12
View File
@@ -1,12 +0,0 @@
<!DOCTYPE html>
<html lang="zh-CN" data-theme="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<title>蜂群中枢 · PlantAI 管理后台</title>
</head>
<body>
<div id="app"></div>
<script type="module" src="/src/main.js"></script>
</body>
</html>
-1147
View File
File diff suppressed because it is too large Load Diff
-18
View File
@@ -1,18 +0,0 @@
{
"name": "plantai-admin",
"version": "0.1.0",
"private": true,
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview"
},
"dependencies": {
"vue": "3.5.13"
},
"devDependencies": {
"vite": "5.4.11",
"@vitejs/plugin-vue": "5.2.1"
}
}
-143
View File
@@ -1,143 +0,0 @@
<script setup>
import { ref, onMounted } from 'vue';
import { getToken, setToken, api } from './api/admin.js';
import Login from './views/Login.vue';
import Users from './views/Users.vue';
import Logs from './views/Logs.vue';
import HubSettings from './components/HubSettings.vue';
const authed = ref(false);
const page = ref('overview');
const user = ref(null);
const stats = ref({ users: 0, total: 0, denied: 0, p95: null });
const hub = ref({ hub_id: '', hub_name: '蜂群中枢' });
const settingsOpen = ref(false);
const PAGES = { overview: '概览', users: '用户管理', logs: '日志查看' };
function onUnauthorized() { setToken(''); authed.value = false; }
async function loadHub() {
try {
const r = await api.settings();
hub.value = r.settings;
} catch (e) {
if (e.status === 401) onUnauthorized();
}
}
async function loadStats() {
try {
const [u, l] = await Promise.all([api.users(), api.logs({ page: 1, page_size: 1 })]);
stats.value.users = u.users.length;
stats.value.total = l.total;
stats.value.denied = (l.aggregates && l.aggregates.denied) || 0;
stats.value.p95 = (l.aggregates && l.aggregates.p95_latency_ms) || null;
} catch (e) {
if (e.status === 401) onUnauthorized();
}
}
async function init() {
if (!getToken()) { authed.value = false; return; }
try {
const r = await api.me();
user.value = r.user;
authed.value = true;
loadHub();
loadStats();
} catch (e) {
setToken('');
authed.value = false;
}
}
function onLogin(u) { user.value = u; authed.value = true; page.value = 'overview'; loadHub(); loadStats(); }
async function onLogout() { try { await api.logout(); } catch {} setToken(''); authed.value = false; }
function onSettingsSaved(s) { hub.value = s; settingsOpen.value = false; }
onMounted(init);
</script>
<template>
<Login v-if="!authed" @login="onLogin" />
<div v-else class="shell">
<aside class="sidebar">
<div class="side-brand">
<div class="hex">蜂</div>
<div><div class="brand-name">{{ hub.hub_name }}</div><div class="brand-sub">{{ hub.hub_id.slice(0, 8) }}</div></div>
</div>
<nav class="nav">
<div class="nav-item" :class="{ active: page === 'overview' }" @click="page = 'overview'"><span>◈</span> 概览</div>
<div class="nav-item" :class="{ active: page === 'users' }" @click="page = 'users'"><span>◉</span> 用户管理</div>
<div class="nav-item" :class="{ active: page === 'logs' }" @click="page = 'logs'"><span>≣</span> 日志查看</div>
</nav>
<div class="side-foot">
<div class="who">
<div class="avatar">蜂</div>
<div><div class="who-name">{{ user?.username }}</div><div class="who-role">系统管理员</div></div>
<span class="logout" @click="onLogout">退出</span>
</div>
</div>
</aside>
<main class="main">
<div class="topbar">
<h1>{{ PAGES[page] }}</h1>
<span style="color: var(--text-3); font-size: 13px">{{ hub.hub_name }}</span>
<button class="btn btn-ghost" style="margin-left: auto" @click="settingsOpen = true">⚙ 设置</button>
</div>
<div class="content">
<section v-if="page === 'overview'">
<div class="stats">
<div class="stat"><div class="k">{{ stats.users }}</div><div class="l">注册用户</div></div>
<div class="stat"><div class="k amber">{{ stats.total }}</div><div class="l">累计请求</div></div>
<div class="stat"><div class="k">{{ stats.denied }}</div><div class="l">越权 / 失败</div></div>
<div class="stat"><div class="k">{{ stats.p95 ?? '—' }}</div><div class="l">P95 耗时 ms</div></div>
</div>
</section>
<Users v-else-if="page === 'users'" @unauthorized="onUnauthorized" />
<Logs v-else @unauthorized="onUnauthorized" />
</div>
</main>
<HubSettings :open="settingsOpen" :hub="hub" @close="settingsOpen = false" @saved="onSettingsSaved" />
</div>
</template>
<style>
.shell { display: flex; height: 100vh; }
.sidebar { width: 218px; flex: none; background: var(--bg-2); border-right: 1px solid var(--border); display: flex; flex-direction: column; }
.side-brand { display: flex; align-items: center; gap: 10px; padding: 20px 18px 18px; }
.hex { width: 30px; height: 30px; flex: none; background: linear-gradient(135deg, #f2b95a, #d98f22); clip-path: polygon(50% 0, 100% 25%, 100% 75%, 50% 100%, 0 75%, 0 25%); display: grid; place-items: center; color: #1a1206; font-weight: 700; font-size: 12px; }
.brand-name { font-size: 15px; font-weight: 600; }
.brand-sub { font-size: 11px; color: var(--text-3); }
.nav { padding: 6px 10px; flex: 1; }
.nav-item { display: flex; align-items: center; gap: 9px; padding: 9px 12px; margin-bottom: 3px; border-radius: var(--radius-sm); color: var(--text-2); font-size: 13.5px; cursor: pointer; }
.nav-item:hover { background: var(--surface); color: var(--text); }
.nav-item.active { background: var(--accent-soft); color: var(--accent); }
.side-foot { padding: 14px 18px; border-top: 1px solid var(--border); }
.who { display: flex; align-items: center; gap: 10px; }
.avatar { width: 30px; height: 30px; border-radius: 50%; flex: none; background: linear-gradient(135deg, #3a4a5e, #243146); display: grid; place-items: center; font-size: 12px; color: var(--text); }
.who-name { font-size: 13px; }
.who-role { font-size: 11px; color: var(--text-3); }
.logout { margin-left: auto; color: var(--text-3); cursor: pointer; font-size: 12px; }
.logout:hover { color: var(--red); }
.main { flex: 1; overflow-y: auto; display: flex; flex-direction: column; }
.topbar { display: flex; align-items: center; gap: 14px; padding: 18px 28px; border-bottom: 1px solid var(--border); position: sticky; top: 0; background: rgba(12, 14, 20, 0.9); backdrop-filter: blur(6px); z-index: 5; }
.topbar h1 { font-size: 17px; margin: 0; font-weight: 600; }
.content { padding: 26px 28px 40px; }
.stats { display: grid; grid-template-columns: repeat(4, 1fr); gap: 16px; }
.stat { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius); padding: 18px 20px; }
.stat .k { font-size: 26px; font-weight: 600; letter-spacing: -0.5px; }
.stat .k.amber { color: var(--accent); }
.stat .l { color: var(--text-2); font-size: 12.5px; margin-top: 3px; }
.toolbar { display: flex; align-items: center; gap: 10px; margin-bottom: 16px; }
.toolbar .spacer { flex: 1; }
.search { display: flex; align-items: center; gap: 8px; background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius-sm); padding: 8px 12px; width: 240px; }
.search input { background: none; border: none; outline: none; color: var(--text); font-size: 13px; width: 100%; }
select { background: var(--surface); border: 1px solid var(--border); color: var(--text); border-radius: var(--radius-sm); padding: 8px 10px; font-size: 13px; outline: none; }
.pag { display: flex; align-items: center; gap: 8px; padding: 12px 18px; border-top: 1px solid var(--border); color: var(--text-3); font-size: 12.5px; }
.pag .spacer { flex: 1; }
.pag button { background: var(--surface-2); border: 1px solid var(--border); color: var(--text-2); border-radius: 6px; padding: 5px 11px; font-size: 12.5px; }
.pag button.on { color: var(--accent); border-color: var(--accent); }
.pag button:disabled { opacity: 0.4; cursor: default; }
</style>
-48
View File
@@ -1,48 +0,0 @@
const TOKEN_KEY = 'pa-admin-token';
let token = localStorage.getItem(TOKEN_KEY) || '';
export function getToken() { return token; }
export function setToken(t) {
token = t || '';
if (t) localStorage.setItem(TOKEN_KEY, t);
else localStorage.removeItem(TOKEN_KEY);
}
async function request(path, { method = 'GET', body } = {}) {
const res = await fetch('/api/admin' + path, {
method,
headers: {
...(body ? { 'Content-Type': 'application/json' } : {}),
...(token ? { Authorization: 'Bearer ' + token } : {}),
},
body: body ? JSON.stringify(body) : undefined,
});
let data = null;
try { data = await res.json(); } catch { data = null; }
if (!res.ok) {
const err = new Error((data && data.error && data.error.message) || ('请求失败 HTTP ' + res.status));
err.status = res.status;
err.code = data && data.error && data.error.code;
throw err;
}
return data;
}
export const api = {
login: (username, password) => request('/login', { method: 'POST', body: { username, password } }),
logout: () => request('/logout', { method: 'POST' }),
me: () => request('/me'),
units: () => request('/units'),
users: () => request('/users'),
createUser: (body) => request('/users', { method: 'POST', body }),
updateUser: (id, body) => request('/users/' + id, { method: 'PATCH', body }),
deleteUser: (id) => request('/users/' + id, { method: 'DELETE' }),
logs: (params = {}) => {
const q = new URLSearchParams();
Object.entries(params).forEach(([k, v]) => { if (v !== undefined && v !== null && v !== '') q.set(k, v); });
const qs = q.toString();
return request('/logs' + (qs ? '?' + qs : ''));
},
settings: () => request('/settings'),
updateSettings: (body) => request('/settings', { method: 'PUT', body }),
};
-57
View File
@@ -1,57 +0,0 @@
<script setup>
import { ref, watch } from 'vue';
import { api } from '../api/admin.js';
const props = defineProps({ open: Boolean, hub: Object });
const emit = defineEmits(['close', 'saved']);
const hubId = ref('');
const hubName = ref('');
const busy = ref(false);
const error = ref('');
watch(() => props.open, (open) => {
if (open) {
hubId.value = (props.hub && props.hub.hub_id) || '';
hubName.value = (props.hub && props.hub.hub_name) || '';
error.value = '';
busy.value = false;
}
});
async function save() {
if (busy.value) return;
if (!hubName.value.trim()) { error.value = '中枢名称不能为空'; return; }
busy.value = true;
error.value = '';
try {
const r = await api.updateSettings({ hub_id: hubId.value.trim(), hub_name: hubName.value.trim() });
emit('saved', r.settings);
} catch (e) {
error.value = e.message || '保存失败';
} finally {
busy.value = false;
}
}
</script>
<template>
<div v-if="open" class="overlay" @click.self="emit('close')">
<div class="modal">
<div class="modal-head">中枢设置</div>
<div class="modal-body">
<div class="field"><label>中枢名称</label><input v-model="hubName" placeholder="例如:湘湖实验室中枢"></div>
<div class="field" style="margin-bottom: 0"><label>中枢 ID</label><input v-model="hubId" class="mono" placeholder="唯一标识,留空则由系统生成"></div>
<p v-if="error" class="err">{{ error }}</p>
</div>
<div class="modal-foot">
<button class="btn btn-ghost" @click="emit('close')">取消</button>
<button class="btn btn-primary" :disabled="busy" @click="save">{{ busy ? '保存中…' : '保存' }}</button>
</div>
</div>
</div>
</template>
<style scoped>
.err { color: var(--red); font-size: 12px; margin: 12px 0 0; }
</style>
-101
View File
@@ -1,101 +0,0 @@
<script setup>
import { ref, watch } from 'vue';
import { api } from '../api/admin.js';
const props = defineProps({
open: Boolean,
mode: { type: String, default: 'create' },
user: Object,
});
const emit = defineEmits(['close', 'saved']);
const UNITS = [
['qa', '问答'], ['kg', '知识图谱'], ['extraction', '文献抽取'], ['docgen', '文档生成'],
['figures', '图片图库'], ['private_docs', '私有资料'], ['bioflow', '生信分析'],
['paper', '论文撰写'], ['legacy', '旧版接口'],
];
const username = ref('');
const password = ref('');
const role = ref('researcher');
const authorizations = ref([]);
const busy = ref(false);
const error = ref('');
watch(() => props.open, (open) => {
if (open) {
error.value = '';
busy.value = false;
if (props.mode === 'edit' && props.user) {
username.value = props.user.username;
password.value = '';
role.value = props.user.role === 'admin' ? 'admin' : (props.user.role === 'guest' ? 'guest' : 'researcher');
authorizations.value = [...(props.user.authorizations || [])];
} else {
username.value = '';
password.value = '';
role.value = 'researcher';
authorizations.value = ['qa'];
}
}
});
async function save() {
if (busy.value) return;
if (props.mode === 'create' && (!username.value.trim() || !password.value)) {
error.value = '用户名和密码必填';
return;
}
busy.value = true;
error.value = '';
try {
if (props.mode === 'create') {
await api.createUser({ username: username.value.trim(), password: password.value, role: role.value, authorizations: authorizations.value });
} else {
const body = { role: role.value, authorizations: authorizations.value };
if (password.value) body.password = password.value;
await api.updateUser(props.user.id, body);
}
emit('saved');
} catch (e) {
error.value = e.message || '保存失败';
} finally {
busy.value = false;
}
}
</script>
<template>
<div v-if="open" class="overlay" @click.self="emit('close')">
<div class="modal">
<div class="modal-head">{{ mode === 'create' ? '新建用户' : '编辑用户' }}</div>
<div class="modal-body">
<div class="field"><label>用户名</label><input v-model="username" :disabled="mode === 'edit'"></div>
<div class="field"><label>{{ mode === 'create' ? '初始密码' : '重置密码(留空不改)' }}</label><input v-model="password" type="password"></div>
<div class="field">
<label>角色</label>
<div class="seg">
<button :class="{ on: role === 'researcher' }" @click="role = 'researcher'">研究员</button>
<button :class="{ on: role === 'guest' }" @click="role = 'guest'">访客</button>
<button :class="{ on: role === 'admin' }" @click="role = 'admin'">管理员</button>
</div>
</div>
<div class="field" style="margin-bottom: 0">
<label>授权清单</label>
<div class="checks">
<label v-for="[k, label] in UNITS" :key="k" class="check"><input type="checkbox" :value="k" v-model="authorizations"> {{ label }}</label>
</div>
</div>
<p v-if="error" class="err">{{ error }}</p>
</div>
<div class="modal-foot">
<button class="btn btn-ghost" @click="emit('close')">取消</button>
<button class="btn btn-primary" :disabled="busy" @click="save">{{ busy ? '保存中…' : (mode === 'create' ? '创建用户' : '保存') }}</button>
</div>
</div>
</div>
</template>
<style scoped>
.err { color: var(--red); font-size: 12px; margin: 12px 0 0; }
</style>
-5
View File
@@ -1,5 +0,0 @@
import { createApp } from 'vue';
import App from './App.vue';
import './style.css';
createApp(App).mount('#app');
-96
View File
@@ -1,96 +0,0 @@
:root {
--bg: #0c0e14;
--bg-2: #10131b;
--surface: #151924;
--surface-2: #1a1f2c;
--border: #232a38;
--border-2: #2e3648;
--text: #e9edf5;
--text-2: #9aa4b6;
--text-3: #5f6a7d;
--accent: #e8a33d;
--accent-ink: #14100a;
--accent-soft: rgba(232, 163, 61, 0.14);
--green: #3ecf8e;
--green-soft: rgba(62, 207, 142, 0.14);
--red: #ef6b6b;
--red-soft: rgba(239, 107, 107, 0.14);
--blue: #6ea8ff;
--radius: 10px;
--radius-sm: 7px;
}
* { box-sizing: border-box; }
html, body, #app { margin: 0; height: 100%; }
body {
background: var(--bg);
color: var(--text);
font-family: -apple-system, BlinkMacSystemFont, "PingFang SC", "Microsoft YaHei", "Segoe UI", Roboto, sans-serif;
font-size: 14px;
line-height: 1.5;
-webkit-font-smoothing: antialiased;
}
.mono { font-family: ui-monospace, "SF Mono", "JetBrains Mono", Menlo, Consolas, monospace; }
button { font-family: inherit; cursor: pointer; }
.btn {
display: inline-flex; align-items: center; justify-content: center; gap: 7px;
padding: 9px 15px; font-size: 13.5px; border-radius: var(--radius-sm);
border: 1px solid transparent; transition: filter .15s; background: transparent;
}
.btn:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
.btn-primary { background: var(--accent); color: var(--accent-ink); font-weight: 600; }
.btn-primary:hover { filter: brightness(1.06); }
.btn-ghost { border-color: var(--border-2); color: var(--text-2); }
.btn-ghost:hover { color: var(--text); border-color: var(--text-3); }
.btn-danger { background: var(--red-soft); color: var(--red); }
.btn-block { width: 100%; }
.field { margin-bottom: 15px; }
.field label { display: block; font-size: 12px; color: var(--text-2); margin-bottom: 7px; }
.field input, .field select {
width: 100%; padding: 10px 12px; font-size: 14px; color: var(--text);
background: var(--bg-2); border: 1px solid var(--border); border-radius: var(--radius-sm); outline: none;
}
.field input:focus, .field select:focus { border-color: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
.chip {
display: inline-block; padding: 3px 9px; border-radius: 20px; font-size: 11.5px;
border: 1px solid var(--border-2); color: var(--text-2); margin: 1.5px 3px 1.5px 0;
}
.chip.amber { border-color: transparent; background: var(--accent-soft); color: var(--accent); }
.badge { display: inline-block; padding: 2px 8px; border-radius: 5px; font-size: 11.5px; font-family: ui-monospace, Menlo, monospace; }
.badge.green { background: var(--green-soft); color: var(--green); }
.badge.red { background: var(--red-soft); color: var(--red); }
.badge.amber { background: var(--accent-soft); color: var(--accent); }
.badge.muted { background: var(--surface-2); color: var(--text-3); }
.panel { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius); }
table { width: 100%; border-collapse: collapse; }
th { text-align: left; font-size: 11.5px; font-weight: 500; color: var(--text-3); padding: 10px 18px; border-bottom: 1px solid var(--border); }
td { padding: 13px 18px; border-bottom: 1px solid var(--border); font-size: 13.5px; }
tr:last-child td { border-bottom: none; }
tbody tr:hover { background: rgba(255,255,255,0.015); }
.cell-id { color: var(--text-2); }
.row-actions { display: flex; gap: 6px; }
.row-actions .act { color: var(--blue); cursor: pointer; font-size: 13px; padding: 3px 6px; border-radius: 5px; }
.row-actions .act:hover { background: rgba(110,168,255,0.1); }
.row-actions .act.danger { color: var(--red); }
.row-actions .act.danger:hover { background: var(--red-soft); }
.role { font-weight: 600; }
.role.admin { color: var(--accent); }
.role.researcher { color: var(--blue); }
.role.guest { color: var(--text-2); }
.overlay { position: fixed; inset: 0; background: rgba(0,0,0,0.55); display: flex; align-items: center; justify-content: center; z-index: 50; }
.modal { width: 460px; max-width: calc(100vw - 32px); background: var(--surface); border: 1px solid var(--border-2); border-radius: 14px; }
.modal-head { padding: 18px 22px 0; font-size: 15px; font-weight: 600; }
.modal-body { padding: 18px 22px; }
.modal-foot { padding: 14px 22px 20px; display: flex; justify-content: flex-end; gap: 10px; border-top: 1px solid var(--border); }
.checks { display: grid; grid-template-columns: 1fr 1fr; gap: 9px 14px; margin-top: 8px; }
.check { display: flex; align-items: center; gap: 8px; font-size: 13px; color: var(--text-2); }
.check input { accent-color: var(--accent); width: 15px; height: 15px; }
.seg { display: flex; gap: 6px; }
.seg button { background: transparent; border: 1px solid var(--border-2); color: var(--text-2); border-radius: 7px; padding: 7px 14px; font-size: 13px; }
.seg button.on { background: var(--accent-soft); border-color: transparent; color: var(--accent); }
-53
View File
@@ -1,53 +0,0 @@
<script setup>
import { ref } from 'vue';
import { api, setToken } from '../api/admin.js';
const emit = defineEmits(['login']);
const username = ref('');
const password = ref('');
const error = ref('');
const busy = ref(false);
async function submit() {
if (busy.value) return;
if (!username.value || !password.value) { error.value = '请输入用户名和密码'; return; }
busy.value = true;
error.value = '';
try {
const r = await api.login(username.value, password.value);
setToken(r.token);
emit('login', r.user);
} catch (e) {
error.value = e.message || '登录失败';
} finally {
busy.value = false;
}
}
</script>
<template>
<div class="login">
<div class="card">
<div class="brand">
<div class="hex">蜂</div>
<div><div class="brand-name">蜂群中枢</div><div class="brand-sub">PlantAI 管理后台</div></div>
</div>
<div class="field"><label>用户名</label><input v-model="username" autocomplete="username" @keyup.enter="submit"></div>
<div class="field"><label>密码</label><input v-model="password" type="password" autocomplete="current-password" @keyup.enter="submit"></div>
<button class="btn btn-primary btn-block" :disabled="busy" @click="submit">{{ busy ? '登录中…' : '登录' }}</button>
<p v-if="error" class="err">{{ error }}</p>
<div class="hint">仅限中枢管理员。登录与所有操作均会留痕。</div>
</div>
</div>
</template>
<style scoped>
.login { height: 100vh; display: flex; align-items: center; justify-content: center; background: radial-gradient(1200px 600px at 70% -10%, rgba(232, 163, 61, 0.08), transparent 60%), var(--bg); }
.card { width: 360px; background: var(--surface); border: 1px solid var(--border); border-radius: 14px; padding: 34px 32px 30px; }
.brand { display: flex; align-items: center; gap: 11px; margin-bottom: 26px; }
.hex { width: 34px; height: 34px; flex: none; background: linear-gradient(135deg, #f2b95a, #d98f22); clip-path: polygon(50% 0, 100% 25%, 100% 75%, 50% 100%, 0 75%, 0 25%); display: grid; place-items: center; color: #1a1206; font-weight: 700; font-size: 14px; }
.brand-name { font-size: 16px; font-weight: 600; }
.brand-sub { color: var(--text-3); font-size: 12px; }
.err { color: var(--red); font-size: 12px; margin: 10px 0 0; }
.hint { margin-top: 20px; padding-top: 16px; border-top: 1px solid var(--border); font-size: 12px; color: var(--text-3); }
</style>
-102
View File
@@ -1,102 +0,0 @@
<script setup>
import { ref, onMounted } from 'vue';
import { api } from '../api/admin.js';
const emit = defineEmits(['unauthorized']);
const from = ref('');
const event = ref('');
const q = ref('');
const page = ref(1);
const pageSize = 30;
const items = ref([]);
const total = ref(0);
const aggregates = ref({});
const loading = ref(false);
const EVENT_LABEL = {
http_request: '调用', admin_login_ok: '登录成功', admin_login_fail: '登录失败',
call_denied: '越权', user_created: '建用户', user_updated: '改用户', user_deleted: '删用户',
};
function statusBadge(s) {
if (!s) return 'muted';
if (s >= 200 && s < 300) return 'green';
if (s >= 500) return 'red';
return 'amber';
}
async function load() {
loading.value = true;
try {
const r = await api.logs({
from_date: from.value || undefined,
event: event.value || undefined,
q: q.value || undefined,
page: page.value,
page_size: pageSize,
});
items.value = r.items;
total.value = r.total;
aggregates.value = r.aggregates || {};
} catch (e) {
if (e.status === 401) emit('unauthorized');
} finally {
loading.value = false;
}
}
function search() { page.value = 1; load(); }
function prev() { if (page.value > 1) { page.value -= 1; load(); } }
function next() { if (page.value * pageSize < total.value) { page.value += 1; load(); } }
onMounted(load);
</script>
<template>
<div>
<div class="stats" style="margin-bottom: 16px">
<div class="stat"><div class="k">{{ total }}</div><div class="l">匹配日志</div></div>
<div class="stat"><div class="k amber">{{ aggregates.denied ?? 0 }}</div><div class="l">越权 / 失败</div></div>
<div class="stat"><div class="k">{{ aggregates.avg_latency_ms ?? '—' }}</div><div class="l">平均耗时 ms</div></div>
<div class="stat"><div class="k">{{ aggregates.p95_latency_ms ?? '—' }}</div><div class="l">P95 耗时 ms</div></div>
</div>
<div class="toolbar">
<input class="date" v-model="from" type="date">
<select v-model="event">
<option value="">全部事件</option>
<option v-for="(l, k) in EVENT_LABEL" :key="k" :value="k">{{ l }}</option>
</select>
<div class="search"><input v-model="q" placeholder="按用户 / 路径过滤…" @keyup.enter="search"></div>
<button class="btn btn-ghost" @click="search">查询</button>
</div>
<div class="panel">
<table>
<thead><tr><th>时间</th><th>用户</th><th>事件</th><th>方法</th><th>路径</th><th>状态</th><th>耗时</th></tr></thead>
<tbody>
<tr v-if="loading"><td colspan="7" class="cell-id">加载中…</td></tr>
<tr v-else-if="!items.length"><td colspan="7" class="cell-id">暂无日志</td></tr>
<tr v-for="(e, i) in items" :key="i">
<td class="cell-id mono">{{ (e.ts || '').replace('T', ' ').slice(0, 19) }}</td>
<td class="mono cell-id">{{ e.user_id ? e.user_id.slice(0, 8) : '—' }}</td>
<td>{{ EVENT_LABEL[e.event] || e.event }}</td>
<td class="cell-id mono">{{ e.method || '—' }}</td>
<td class="mono">{{ e.path || '—' }}</td>
<td><span v-if="e.status" class="badge" :class="statusBadge(e.status)">{{ e.status }}</span><span v-else class="badge muted">—</span></td>
<td class="cell-id mono">{{ e.latency_ms != null ? e.latency_ms + 'ms' : '—' }}</td>
</tr>
</tbody>
</table>
<div class="pag">
<span>共 {{ total }} 条</span><span class="spacer"></span>
<button :disabled="page <= 1" @click="prev">上一页</button>
<button class="on">{{ page }}</button>
<button :disabled="page * pageSize >= total" @click="next">下一页</button>
</div>
</div>
</div>
</template>
<style scoped>
input.date { background: var(--surface); border: 1px solid var(--border); color: var(--text); border-radius: var(--radius-sm); padding: 8px 10px; font-size: 13px; outline: none; }
</style>
-72
View File
@@ -1,72 +0,0 @@
<script setup>
import { ref, onMounted } from 'vue';
import { api } from '../api/admin.js';
import UserForm from '../components/UserForm.vue';
const emit = defineEmits(['unauthorized']);
const users = ref([]);
const loading = ref(true);
const formOpen = ref(false);
const formMode = ref('create');
const editing = ref(null);
const ROLE_LABEL = { admin: '管理员', researcher: '研究员', guest: '访客' };
const UNIT_LABEL = { qa: '问答', kg: '知识图谱', extraction: '文献抽取', docgen: '文档生成', figures: '图片图库', private_docs: '私有资料', bioflow: '生信分析', paper: '论文撰写', legacy: '旧版接口' };
async function load() {
loading.value = true;
try {
const r = await api.users();
users.value = r.users;
} catch (e) {
if (e.status === 401) emit('unauthorized');
} finally {
loading.value = false;
}
}
function create() { formMode.value = 'create'; editing.value = null; formOpen.value = true; }
function edit(u) { formMode.value = 'edit'; editing.value = u; formOpen.value = true; }
async function del(u) {
if (!confirm(`删除用户「${u.username}」?此操作不可撤销。`)) return;
try { await api.deleteUser(u.id); load(); } catch (e) { alert(e.message || '删除失败'); }
}
function onSaved() { formOpen.value = false; load(); }
onMounted(load);
</script>
<template>
<div>
<div class="toolbar">
<div class="spacer"></div>
<button class="btn btn-primary" @click="create">+ 新建用户</button>
</div>
<div class="panel">
<table>
<thead><tr><th>用户</th><th>角色</th><th>授权范围</th><th>创建时间</th><th style="width: 120px"></th></tr></thead>
<tbody>
<tr v-if="loading"><td colspan="5" class="cell-id">加载中…</td></tr>
<tr v-else-if="!users.length"><td colspan="5" class="cell-id">暂无用户</td></tr>
<tr v-for="u in users" :key="u.id">
<td>
<div style="display: flex; align-items: center; gap: 10px">
<div class="avatar">{{ u.username[0] }}</div>
<div><div>{{ u.username }}</div><div class="cell-id" style="font-size: 12px">{{ u.id.slice(0, 8) }}</div></div>
</div>
</td>
<td><span class="role" :class="u.role">{{ ROLE_LABEL[u.role] || u.role }}</span></td>
<td>
<span v-if="u.role === 'admin'" class="chip amber">全部权限</span>
<template v-else><span v-for="a in u.authorizations" :key="a" class="chip">{{ UNIT_LABEL[a] || a }}</span></template>
</td>
<td class="cell-id mono">{{ (u.created_at || '').slice(0, 10) }}</td>
<td><div class="row-actions"><span class="act" @click="edit(u)">编辑</span><span class="act danger" @click="del(u)">删除</span></div></td>
</tr>
</tbody>
</table>
</div>
<UserForm :open="formOpen" :mode="formMode" :user="editing" @close="formOpen = false" @saved="onSaved" />
</div>
</template>
-15
View File
@@ -1,15 +0,0 @@
import { defineConfig } from 'vite';
import vue from '@vitejs/plugin-vue';
export default defineConfig({
base: '/admin/',
plugins: [vue()],
server: {
host: '127.0.0.1',
port: 5181,
proxy: {
'/api': { target: 'http://127.0.0.1:8090', changeOrigin: false },
},
},
build: { outDir: 'dist', target: 'es2020' },
});
+67
View File
@@ -16,3 +16,70 @@ capabilities:
endpoint: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi
method: GET
timeout: 8
# ---- 生信能力单元(S1):两个 MCP 端点,各自独立装卸 ----
- name: bioflow-gene-family
provider: mcp
permission: read_only
description: 基因家族分析(路径规划/建任务/参数确认/文件获取与预处理/状态查询),自然语言触发
session_default: true
nl_triggers:
- 基因家族
- 家族成员
- 亚家族
- 系统发育
- 建树
- 进化树
- 保守基序
- motif
- 结构域
- 基因结构
- 共线性
- Ka/Ks
- 选择压力
- 组合证据
mcp:
url: http://127.0.0.1:8098/mcp/gene-family/
prompt_pack:
file: bioflow_gene_family_agent_rules.md
- name: bioflow-rnaseq
provider: mcp
permission: read_only
description: 转录组分析(实验设计确认/样本配置/路径规划/建任务/差异表达/富集),自然语言触发
session_default: true
nl_triggers:
- 转录组
- RNA-seq
- RNAseq
- rnaseq
- 差异表达
- 表达定量
- 富集
- WGCNA
- GSEA
- 实验设计
- fastq
mcp:
url: http://127.0.0.1:8098/mcp/rnaseq/
prompt_pack:
file: bioflow_rnaseq_agent_rules.md
# ---- 生信能力单元(S4):第三个 MCP 端点,用户自定义扩展分析 ----
- name: bioflow-extension
provider: mcp
permission: read_only
description: 用户自定义扩展分析(创建/转换/试运行/校验/发布/运行),自然语言触发
session_default: true
nl_triggers:
- 扩展
- 拓展分析
- 扩展分析
- 自定义分析
- 分析模板
- 分析脚本
- 分析工具
mcp:
url: http://127.0.0.1:8098/mcp/extension/
prompt_pack:
file: bioflow_extension_agent_rules.md
+3
View File
@@ -31,3 +31,6 @@ QA_V1_WEB_SEARCH_MAX_RESULTS=4
QA_V1_WEB_SEARCH_TIMEOUT=8
QA_V1_WEB_SEARCH_MAX_CHARS=1200
QA_V1_ENABLE_SUPERPOWERS=1
# 生信对话式能力熔断开关(S1):默认关(=0)。置 1 才启用 QA 前门 bioflow_agent。
QA_V1_ENABLE_BIOFLOW_AGENT=0
@@ -0,0 +1,31 @@
# 用户自定义扩展分析对话规范(bioflow_extension_agent_rules)
你是用户自定义扩展分析助手。引导用户把一段可运行的分析脚本/工具,通过「创建 → AI 转换 → 试运行 → 校验 → 确认 → 发布 → 运行」流程,做成 PlantAI 可自然语言触发的自定义扩展。
## 对话节奏(自然语言主导 + 卡片辅助)
每次需要用户输入/选择时,遵循固定节奏:
1. 先用自然语言说明并列表展示需要什么;
2. 提示用户可用卡片快捷操作,也可直接打字(两条通道语义一致);
3. 用户回答后,用自然语言复述,确认无误再继续。
## 创建(create_extension)
- 需要用户提供:显示名、运行命令、代码文件(可上传/粘贴文本内容)、运行时镜像、可选测试数据/金标。
- 代码文件以「路径 + 文本内容」提交;引导用户说明脚本是做什么的、依赖什么。
## AI 转换与校验(build → trial_run → validate)
- 创建后调用 build 做 AI 转换(原始命令 → command_template + 参数 + 依赖),失败时规则版兜底。
- 转换结果先向用户说明(识别到的参数/依赖/输入角色),请用户确认参数名与默认值。
- 试运行/校验的结果要转述给用户,失败时说明原因并引导修复(repair 循环)。
## 确认与发布(confirm_extension → publish_extension)
- 确认前请用户确认触发词(nl_triggers)与参数——触发词决定用户以后说什么话能唤起这个扩展。
- 发布后扩展才可被自然语言触发运行;发布是不可逆的对外可用状态,须经用户确认。
## 运行(run_extension)
- 仅已发布扩展可运行;运行参数须在白名单内(参数 schema),未知参数明确追问。
- 运行结果(run_result)转述给用户。
## 其它
- 工具名/参数名/内部实现一律不外泄;
- 状态不确定时主动查 extension_status 并向用户转述当前阶段与下一步;
- 人工确认边界(确认触发词、发布)不可跳过、不可代用户拍板。
@@ -0,0 +1,32 @@
# 基因家族分析对话规范(bioflow_gene_family_agent_rules)
你是基因家族分析助手。用自然语言引导用户从需求到分析完成,全程对话式交互。
## 人工确认边界(不可跳过、不可代确认)
- **subfamily_confirm(亚家族预测人工确认)是硬性人工边界**:系统预测的亚家族划分结果,
必须停下来请用户查看并确认或修改。你绝不能代用户确认,也不能在用户未确认前继续下游分析。
- 用户提出修改(如"把某成员从亚家族 A 移到亚家族 B")时,先复述修改内容,再请用户做最终确认。
## 对话节奏(自然语言主导 + 卡片辅助)
每次需要用户输入/选择时,遵循固定节奏:
1. 先用自然语言说明并**列表展示**选项及含义;
2. 提示用户可用卡片快捷操作,也可直接打字回复(两条通道语义一致);
3. 用户选择后,用自然语言**复述**其选择,确认无误再继续。
## 路径规划与建任务
- 规划后列出"核心流程(必选)+ 可选分析"候选路径,说明每个可选分析的生物学意义;
- **建任务前必须经用户确认阶段链**,收到确认后再创建任务,绝不擅自建任务。
## 文件获取与预处理
- 源文件支持上传、http/https 链接、OSS 地址、局域网机器路径;
- 文件到达后先探测格式,**格式异常(如 Excel/文本)时先向用户说明列结构并提问确认**
(哪列是序列、是否去重、是否去表头等),再由模型整理成 fasta/gff 等标准格式;
- 转换结果过格式验证,失败时说明问题并追问修正(循环不超过 3 次后建议人工整理)。
## 参数确认
- 列"参数名(默认值,含义)"请用户确认或修改;未识别参数时明确指出并给出允许范围。
## 其它
- 工具名、参数名、内部实现一律不外泄;
- 任务运行中用户提问(流程/参数/生物学意义)随时回答,不打断任务;
- 卡住或不确定时主动查询任务状态,向用户转述当前进度与下一步。
@@ -0,0 +1,29 @@
# 转录组分析对话规范(bioflow_rnaseq_agent_rules)
你是转录组分析助手。用自然语言引导用户从实验设计确认到差异表达/富集分析完成。
## 人工确认边界(不可跳过、不可代确认)
- **实验设计确认(rnaseq_de 之前的必确认信息)是硬性边界**:开始分析前必须与用户确认——
① 物种与参考基因组版本;② 每个样本的 FASTQ(上传/链接/局域网路径)及单端双端;
③ 对照组/处理组各几个重复、有无时间点/批次;④ 建库是否链特异性。
- 统计模型正确性依赖上述信息,未确认前不得创建差异表达任务,也不能代用户拍板基线/比较方向。
## 对话节奏(自然语言主导 + 卡片辅助)
每次需要用户输入/选择时,遵循固定节奏:
1. 先用自然语言说明并**列表展示**必确认信息与可选分析;
2. 提示用户可用卡片快捷操作,也可直接打字回复(两条通道语义一致);
3. 用户回答后,用自然语言**复述**其样本设计与选择(如"干旱 3 vs 对照 3,双端,非链特异,对照为基线"),
确认无误再继续。
## 样本配置与文件预处理
- 用户用自然语言描述样本时,可解析生成 samples.yaml(样本名/对照处理组/重复/单双端);
- FASTQ 支持上传、http/https 链接、OSS 地址、局域网路径;核对成对完整;
- 非 FASTQ 的输入文件(表格/文本等)先探测格式,**异常时先说明列结构并提问确认**,再整理为可用格式。
## 参数确认
- 列"参数名(默认值,含义)"请用户确认或修改;未识别参数时明确指出并给出允许范围。
## 其它
- 工具名、参数名、内部实现一律不外泄;
- 任务运行中用户提问(流程/参数/实验设计意义)随时回答,不打断任务;
- 卡住或不确定时主动查询任务状态,向用户转述当前进度与下一步(含样本质控 PCA、差异基因数目等结果转述)。
+29
View File
@@ -13,6 +13,7 @@ from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from . import __version__
from .bioflow_agent import try_bioflow_agent
from .config import QAConfig
from .pipeline import QAAnswerError, QAV1Pipeline
@@ -47,6 +48,9 @@ class QARequest(BaseModel):
# ---- Capability Gateway(S1):纯数据字段,无 validator(Cython 编译安全)----
history: List[Dict[str, str]] = Field(default_factory=list)
capabilities: Dict[str, Any] = Field(default_factory=dict)
# ---- Bioflow 对话式编排(S2):卡片回填协议 + session↔job 绑定 ----
card_result: Optional[Dict[str, Any]] = None
bioflow: Optional[Dict[str, Any]] = None
@lru_cache(maxsize=1)
@@ -150,6 +154,18 @@ async def chat(request: QARequest) -> Dict[str, Any]:
detail={"error_code": "llm_unavailable", "answer": "", "message": "Final-answer LLM is not configured"},
)
try:
# ---- Bioflow Agent 前门(S1):命中则短路返回,否则走原 pipeline ----
bioflow_result = await asyncio.to_thread(
try_bioflow_agent,
request.question,
llm_client=pipeline.llm,
history=request.history,
capabilities=request.capabilities,
card_result=request.card_result,
bioflow=request.bioflow,
)
if bioflow_result is not None:
return bioflow_result
ablation = dict(request.ablation or {})
ablation["use_final_llm"] = True
return await asyncio.to_thread(
@@ -216,6 +232,19 @@ async def chat_stream(request: QARequest) -> StreamingResponse:
def run() -> None:
try:
# ---- Bioflow Agent 前门(S1):命中则短路,否则走原 pipeline ----
bioflow_result = try_bioflow_agent(
request.question,
llm_client=pipeline.llm,
history=request.history,
capabilities=request.capabilities,
card_result=request.card_result,
bioflow=request.bioflow,
progress_callback=emit,
)
if bioflow_result is not None:
events.put(("complete", bioflow_result))
return
ablation = dict(request.ablation or {})
ablation["use_final_llm"] = True
result = pipeline.answer(
@@ -0,0 +1,764 @@
from __future__ import annotations
"""bioflow_agent.py —— 生信 MCP 能力前门路由(S1 + S2)。
S1:三重门(会话开关/注册表/意图预筛)→ ToolRunner 工具循环 → 自然语言转述。
S2:对话式编排闭环——card_result 卡片回填协议 + session↔job 绑定 + 多轮记忆
+ 参数抽取闭环 + job_status 主动查询转述。
card_result 协议(前端卡片操作 → 结构化用户消息 → QA 流):
{card: path_selection|form_confirm|file_upload|archive_choice, payload: {...}}
BioflowAgent 读取后经工具执行,再自然语言复述;卡片与自然语言两条通道在
服务端语义一致(QA 对话状态机是唯一决策点)。
session↔job 绑定:BFF sessions.meta.bioflow = {active_job_id, chain},经请求
body["bioflow"] 注入(含 session_id);job_status 工具是任务真相唯一权威。
金标路径零改动:任何不命中/异常都返回 None 回退原 pipeline。
"""
import json
import logging
import os
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
from .capabilities.manifests import CapabilityManifest
from .capabilities.providers import PromptPackProvider, get_dispatcher
from .capabilities.registry import get_registry
from .tool_runner import ToolRunner, make_llm_complete
LOGGER = logging.getLogger(__name__)
# 三个生信能力单元(与 bioflow_service 三个 MCP 端点一一对应)
BIOFLOW_CAPABILITY_NAMES = ("bioflow-gene-family", "bioflow-rnaseq", "bioflow-extension")
ENV_FUSE = "QA_V1_ENABLE_BIOFLOW_AGENT"
MAX_ROUNDS = 5
# card_result 卡片类型(S2 前端卡片回填协议)
CARD_PATH_SELECTION = "path_selection"
CARD_FORM_CONFIRM = "form_confirm"
CARD_FILE_UPLOAD = "file_upload"
CARD_ARCHIVE_CHOICE = "archive_choice"
CARD_EXTENSION_ACTION = "extension_action" # S4:扩展能力单元操作(build/trial/validate/confirm/publish/run)
CARD_TYPES = (CARD_PATH_SELECTION, CARD_FORM_CONFIRM, CARD_FILE_UPLOAD, CARD_ARCHIVE_CHOICE, CARD_EXTENSION_ACTION)
# chain 名 → 能力名(建任务时选择哪条链的 create 工具)
_CHAIN_TO_CAPABILITY = {
"gene_family": "bioflow-gene-family",
"rnaseq": "bioflow-rnaseq",
}
# 能力名 → chain 名(反向映射,供规划阶段从命中能力回填 chain,续接多轮对话)
_CAPABILITY_TO_CHAIN = {v: k for k, v in _CHAIN_TO_CAPABILITY.items()}
def _chain_to_capability(chain: str) -> Optional[str]:
"""把 chain 名(gene_family/rnaseq)映射为能力名;未知返回 None。"""
return _CHAIN_TO_CAPABILITY.get(str(chain or "").strip().lower())
_SYSTEM_BASE = (
"你是 PlantAI 生信分析助手,用自然语言引导用户完成基因家族或转录组分析。\n"
"你可以调用生信工具完成规划、建任务、确认参数、登记文件、查状态等操作。\n"
"硬性规则:\n"
"1. 回复只使用自然语言,绝不向用户泄露工具名、参数名或内部实现细节。\n"
"2. 需要用户提供文件/参数/做选择时,先自然语言说明并列表展示选项与含义,再请用户回答,"
"绝不擅自替用户做不可逆决策。\n"
"3. 人工确认边界(基因家族 subfamily_confirm、转录组实验设计确认)不可跳过、不可代用户确认。\n"
"4. 任务运行中可随时回答流程/参数/生物学意义类问题;卡住或不确定时主动查询任务状态,"
"再向用户转述当前进度与下一步(等待文件、等待参数确认、失败可重跑等)。\n"
"5. 文件格式异常时先向用户说明并提问确认,再由用户决定如何处理。\n"
"6. 用户用自然语言给出参数(如「用 8 核、padj 用 0.01」)时,抽取为结构化参数后经确认步骤提交;"
"只提交白名单内的参数名,未知参数明确指出并追问(最多 3 次)。\n"
"7. 若用户消息是对上一轮卡片/提问的确认或选择,先复述用户的选择,确认无误再执行对应动作。\n"
)
def _env_enabled() -> bool:
"""门 1:QA_V1_ENABLE_BIOFLOW_AGENT 熔断开关。默认关(未设置 = 关)。"""
raw = os.environ.get(ENV_FUSE)
return raw is not None and str(raw).strip().lower() in {"1", "true", "yes", "on"}
def _intent_match(question: str, triggers: List[str]) -> bool:
"""门 3:意图预筛——问句是否命中任一触发词(大小写不敏感子串匹配)。"""
if not triggers:
return False
q = (question or "").lower()
return any(t and t.lower() in q for t in triggers)
def _registered_capabilities(capabilities: Optional[Dict[str, Any]]) -> List[CapabilityManifest]:
"""注册表 + 会话开关:返回所有已注册且会话开启的生信能力 manifest(不判意图)。"""
try:
registry = get_registry()
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: registry 访问失败,回退原 pipeline: %s", exc)
return []
session_caps = capabilities or {}
out: List[CapabilityManifest] = []
for name in BIOFLOW_CAPABILITY_NAMES:
try:
spec = registry.get(name)
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: registry.get(%s) 失败: %s", name, exc)
continue
if spec is None:
continue # 未注册
manifest = spec.manifest
if not bool(session_caps.get(name, manifest.session_default)):
continue # 会话开关关
out.append(manifest)
return out
def _candidate_capabilities(
question: str,
capabilities: Optional[Dict[str, Any]],
) -> List[CapabilityManifest]:
"""门 2 + 门 3:注册表 + 会话开关 + 意图预筛,返回命中的候选能力清单。"""
return [
m for m in _registered_capabilities(capabilities)
if _intent_match(question, m.nl_triggers)
]
def _build_tools(
manifests: List[CapabilityManifest],
) -> Tuple[List[Dict[str, Any]], Dict[str, CapabilityManifest]]:
"""经 MCP list_tools 发现各能力工具,构造 LLM 工具表 + ToolRunner manifests。
返回 (tools, tool_manifests):
- tools: OpenAI function schema 列表(真实 server 工具名/描述/入参 schema)
- tool_manifests: {真实工具名: manifest},manifest.name=能力名(共享连接),
mcp.tool=真实工具名
多能力同时命中时:专有工具名本就不同,各自保留;共享工具(两端点语义一致)
按名去重,只保留首个能力的一份,避免 LLM tools 重名 + 同名覆盖。
"""
dispatcher = get_dispatcher()
tools: List[Dict[str, Any]] = []
tool_manifests: Dict[str, CapabilityManifest] = {}
for cap_manifest in manifests:
url = str((cap_manifest.config.get("mcp") or {}).get("url") or "").strip()
if not url:
LOGGER.warning("bioflow agent: 能力 %s 缺 mcp.url,跳过", cap_manifest.name)
continue
try:
discovered = dispatcher.list_mcp_tools(cap_manifest)
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: 能力 %s MCP 工具发现失败: %s", cap_manifest.name, exc)
continue
for item in discovered:
tool_name = str(item.get("name") or "").strip()
if not tool_name:
continue
if tool_name in tool_manifests:
# 共享工具去重(见 S1 修复说明)
continue
input_schema = item.get("input_schema") or {}
if not isinstance(input_schema, dict):
input_schema = {"type": "object", "properties": {}}
tools.append(
{
"type": "function",
"function": {
"name": tool_name,
"description": str(item.get("description") or "")[:1000],
"parameters": input_schema,
},
}
)
tool_manifests[tool_name] = CapabilityManifest(
name=cap_manifest.name,
provider="mcp",
permission="read_only",
description=str(item.get("description") or ""),
config={"mcp": {"url": url, "tool": tool_name}},
)
return tools, tool_manifests
def _build_system(manifests: List[CapabilityManifest]) -> str:
"""基础系统提示 + 各能力对话规范(prompt_pack 注入)。"""
qa_root = Path(os.environ.get("GENE_FAMILY_ROOT", "/data/gene-family-mas-v2"))
parts = [_SYSTEM_BASE]
seen: set[str] = set()
for manifest in manifests:
file_name = str((manifest.config.get("prompt_pack") or {}).get("file") or "").strip()
if not file_name or file_name in seen:
continue
seen.add(file_name)
try:
text = PromptPackProvider.load(manifest, qa_root)
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: prompt_pack 加载失败 %s: %s", file_name, exc)
text = ""
if text:
parts.append(text)
return "\n\n".join(parts)
# ---------------------------------------------------------------------------
# S2 工具执行 / 转述 / 参数抽取 辅助
# ---------------------------------------------------------------------------
def _execute_tool(
tool_manifests: Dict[str, CapabilityManifest], tool_name: str, arguments: Dict[str, Any]
) -> str:
"""执行单个工具,返回结果文本(异常转文本,不炸主流程)。"""
manifest = tool_manifests.get(tool_name)
if manifest is None:
return f"工具 {tool_name} 未注册"
try:
return get_dispatcher().execute(manifest, arguments)
except Exception as exc: # noqa: BLE001
return f"工具执行失败: {type(exc).__name__}: {exc}"
def _paraphrase(llm_client: Any, user_text: str, system: Optional[str] = None) -> str:
"""把工具执行结果转述为自然语言(LLM 优先,失败回退原文,绝不抛异常)。"""
system = system or (
"你是 PlantAI 生信分析助手。把下面的工具执行结果转述为对用户的自然语言回复,"
"绝不提及工具名/参数名/内部实现;需要用户下一步操作时明确说明。"
)
try:
text = llm_client.complete(system, user_text, temperature=0.2, max_tokens=600)
if text and str(text).strip():
return str(text).strip()
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: paraphrase LLM 失败,回退原文: %s", exc)
return str(user_text or "").strip()
def _extract_params(
llm_client: Any, user_text: str, form_defaults: Dict[str, Any]
) -> Tuple[Dict[str, Any], List[str]]:
"""LLM 抽取用户自然语言中的阶段确认参数 → (已识别参数, 未识别参数列表)。
白名单 = form_defaults 的键;未识别参数(不在白名单)单独返回供追问(≤3 次)。
参照 extension_llm 的「LLM 出 JSON → 白名单净化」模式,LLM 失败/坏 JSON 返回空。
"""
whitelist = {str(k) for k in (form_defaults or {}).keys()}
if not whitelist:
return {}, []
system = (
"你是参数抽取助手。从用户的话中抽取阶段确认参数,只输出 JSON。\n"
'输出形态:{"params": {"<参数名>": <值>}, "unknown": ["<未识别参数名>"]}。\n'
f"允许的参数名(白名单):{', '.join(sorted(whitelist))}。\n"
"只抽取用户明确提到的参数;不在白名单里的参数名放进 unknown。"
)
try:
raw = llm_client.complete_json(system, user_text, max_tokens=500)
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: 参数抽取 LLM 失败: %s", exc)
return {}, []
if not isinstance(raw, dict):
return {}, []
params = raw.get("params") if isinstance(raw.get("params"), dict) else {}
known = {str(k): v for k, v in params.items() if str(k) in whitelist}
unknown_raw = raw.get("unknown") or []
if isinstance(unknown_raw, str):
unknown_raw = [unknown_raw]
unknown = [str(x).strip() for x in unknown_raw if isinstance(x, str) and str(x).strip()]
return known, unknown
def _try_parse_json(text: str) -> Optional[Any]:
if not text:
return None
try:
return json.loads(text)
except (json.JSONDecodeError, ValueError):
return None
def _extract_job_id(result_text: str) -> Optional[str]:
"""从 create_job 工具结果文本里提取 job_id(结果形如 {"ok":true,"data":{"id":...}})。"""
data = _try_parse_json(result_text)
if isinstance(data, dict):
inner = data.get("data") if isinstance(data.get("data"), dict) else data
jid = inner.get("id") or inner.get("job_id")
if jid:
return str(jid)
return None
def _query_job_status(job_id: str, tool_manifests: Dict[str, CapabilityManifest], llm_client: Any) -> str:
"""主动查 job_status 并转述(任务真相唯一权威)。"""
result = _execute_tool(tool_manifests, "job_status", {"job_id": job_id})
return _paraphrase(llm_client, f"查询任务状态的结果:\n{result}")
def _empty_audit() -> Dict[str, Any]:
return {"rounds": 0, "tool_calls": 0, "used_json_fallback": False, "tool_names": [], "errors": []}
def _map_readiness(data: Dict[str, Any]) -> Dict[str, Any]:
"""check_readiness 结果 → 就绪卡字段 {ready, quadrants:[{name,ok}]}。"""
inner = data.get("result", data) if isinstance(data, dict) else {}
quadrants = []
for key in ("environment", "reference", "space", "budget"):
q = inner.get(key) or {}
quadrants.append({"name": key, "ok": bool(q.get("ok"))})
return {"ready": bool(inner.get("ready")), "quadrants": quadrants}
def _map_versioning(data: Dict[str, Any]) -> Dict[str, Any]:
"""propose_change 结果 → 版本化卡字段 {plan_revision, run_id, affected_stages, based_on_revision}。"""
inner = data.get("result", data) if isinstance(data, dict) else {}
affected = inner.get("affected") or []
return {
"plan_revision": "",
"run_id": "",
"affected_stages": list(affected),
"based_on_revision": None,
}
def _make_payload(
question: str,
answer: str,
*,
audit: Optional[Dict[str, Any]] = None,
capabilities: Optional[List[str]] = None,
active_job_id: Optional[str] = None,
chain: Optional[str] = None,
readiness: Optional[Dict[str, Any]] = None,
versioning: Optional[Dict[str, Any]] = None,
plan: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
payload: Dict[str, Any] = {
"success": True,
"version": "bioflow-agent-s2",
"question": question,
"answer": answer,
"sources": [],
"source_count": 0,
"citation_validation": {"valid": True, "cited_pmids": [], "unknown_pmids": []},
"bioflow": {
"handled": True,
"audit": audit or _empty_audit(),
"capabilities": capabilities or [],
},
}
if active_job_id:
payload["bioflow"]["active_job_id"] = str(active_job_id)
if chain:
payload["bioflow"]["chain"] = str(chain)
if readiness:
payload["bioflow"]["readiness"] = readiness
if versioning:
payload["bioflow"]["versioning"] = versioning
if plan:
payload["bioflow"]["plan"] = plan
return payload
# ---------------------------------------------------------------------------
# card_result 回填处理(S2)
# ---------------------------------------------------------------------------
def _card_path_selection(
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
) -> Optional[Dict[str, Any]]:
"""路径卡确认 → create_*_job → 复述;带 reuse_archive 时先复用存档的阶段链/参数再建任务。"""
chain = str(payload.get("chain") or (bioflow or {}).get("chain") or "").strip()
stages = payload.get("stages")
reuse_name = str(payload.get("reuse_archive") or payload.get("reuse_name") or "").strip()
has_stages = isinstance(stages, list) and len(stages) > 0
if not has_stages and not reuse_name:
# 既无显式阶段链也无存档复用来源 → 无法建任务,回退原 pipeline
return None
cap_name = _chain_to_capability(chain)
if cap_name is None:
# 无法确定能力(chain 空/未知):不能默认走某条链,回退原 pipeline
return None
tool = "create_gene_family_job" if cap_name == "bioflow-gene-family" else "create_rnaseq_job"
if tool not in tool_manifests:
return None
args: Dict[str, Any] = {
"session_id": str((bioflow or {}).get("session_id") or ""),
"stages": [str(s) for s in stages] if isinstance(stages, list) and stages else [],
"title": str(payload.get("title") or ""),
"auto_continue": bool(payload.get("auto_continue", False)),
}
tool_names: List[str] = []
# 存档复用(FR-BIO-023):先 reuse_archive 取阶段链/参数/archive_id,再以其发起
if reuse_name and "reuse_archive" in tool_manifests:
tool_names.append("reuse_archive") # 实际先执行 reuse_archive(audit 顺序与实际一致)
reuse_text = _execute_tool(tool_manifests, "reuse_archive", {"name": reuse_name})
arch = _try_parse_json(reuse_text)
if isinstance(arch, dict) and arch.get("ok"):
data = arch.get("data") if isinstance(arch.get("data"), dict) else {}
if data.get("id"):
args["archive_id"] = str(data["id"])
if not args["stages"] and isinstance(data.get("stage_chain"), list):
args["stages"] = [str(s) for s in data["stage_chain"]]
if isinstance(data.get("params"), dict):
merged = dict(data["params"])
if isinstance(payload.get("params"), dict):
merged.update(payload["params"])
args["params"] = merged
elif not args["stages"]:
# 复用失败(非 ok)且无显式阶段链:不能空链建任务,回退原 pipeline
return None
tool_names.append(tool) # create_*_job 最后执行
if "params" not in args and isinstance(payload.get("params"), dict):
args["params"] = payload["params"]
result = _execute_tool(tool_manifests, tool, args)
job_id = _extract_job_id(result)
answer = _paraphrase(
llm_client,
f"用户确认了分析路径并创建任务({'基因家族' if cap_name == 'bioflow-gene-family' else '转录组'}链"
f"{',复用存档 ' + reuse_name if reuse_name else ''})。工具结果:\n{result}",
)
return _make_payload(
"确认路径并创建任务", answer,
audit={"rounds": 1, "tool_calls": len(tool_names), "used_json_fallback": False, "tool_names": tool_names, "errors": []},
capabilities=[cap_name] if cap_name else [],
active_job_id=job_id,
chain=chain or None,
)
def _card_form_confirm(
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
) -> Optional[Dict[str, Any]]:
"""参数卡确认 → confirm_stage(白名单校验)→ 复述;rerun:true 走 rerun_stage(改参数重跑)。"""
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
stage = str(payload.get("stage") or "").strip()
params = payload.get("params") if isinstance(payload.get("params"), dict) else {}
rerun = bool(payload.get("rerun", False))
if not job_id or not stage:
return None
tool = "rerun_stage" if rerun else "confirm_stage"
result = _execute_tool(tool_manifests, tool, {"job_id": job_id, "stage": stage, "params": params})
answer = _paraphrase(
llm_client,
f"用户{'重跑并确认' if rerun else '确认'}了阶段参数(任务 {job_id} 的 {stage})。工具结果:\n{result}",
)
return _make_payload(
f"{'重跑' if rerun else '确认'}阶段参数:{stage}", answer,
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": [tool], "errors": []},
active_job_id=job_id,
)
def _card_file_upload(
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
) -> Optional[Dict[str, Any]]:
"""文件卡 → 网络源 fetch_source_file / 服务器·局域网路径 register_job_files → 复述。"""
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
raw_files = payload.get("paths") or payload.get("files") or []
if isinstance(raw_files, str):
raw_files = [raw_files]
files = [str(f) for f in raw_files if str(f).strip()] if isinstance(raw_files, list) else []
if not job_id or not files:
return None
remote = [f for f in files if f.startswith(("http://", "https://", "oss://"))]
local = [f for f in files if f not in remote]
results: List[str] = []
tool_names: List[str] = []
for src in remote:
r = _execute_tool(tool_manifests, "fetch_source_file", {"source": src, "job_id": job_id})
results.append(r)
tool_names.append("fetch_source_file")
if local:
r = _execute_tool(tool_manifests, "register_job_files", {"job_id": job_id, "paths": local})
results.append(r)
tool_names.append("register_job_files")
answer = _paraphrase(
llm_client,
f"用户提供了文件({'远程 ' + str(len(remote)) + ' 个' if remote else ''}"
f"{'本地 ' + str(len(local)) + ' 个' if local else ''})。工具结果:\n" + "\n".join(results),
)
return _make_payload(
"登记任务文件", answer,
audit={"rounds": 1, "tool_calls": len(tool_names), "used_json_fallback": False, "tool_names": tool_names, "errors": []},
active_job_id=job_id,
)
def _card_archive_choice(
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
) -> Optional[Dict[str, Any]]:
"""存档卡 → archive_job 工具执行实际存档(或跳过)→ 复述。"""
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
do_archive = bool(payload.get("archive", False))
if not job_id:
return None
if do_archive:
name = str(payload.get("name") or "")
description = str(payload.get("description") or "")
result = _execute_tool(
tool_manifests, "archive_job",
{"job_id": job_id, "name": name, "description": description},
)
answer = _paraphrase(
llm_client,
f"用户选择把任务 {job_id} 存档(名称:{name or '未命名'})。工具结果:\n{result}",
)
return _make_payload(
"存档选择", answer,
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": ["archive_job"], "errors": []},
active_job_id=job_id,
)
answer = _paraphrase(
llm_client,
f"用户选择不存档任务 {job_id}。请用自然语言告知用户:已跳过存档,"
"本次分析结果仍可在报告/结果中查看。",
)
return _make_payload(
"存档选择", answer,
audit=_empty_audit(),
active_job_id=job_id,
)
def _card_extension_action(
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
) -> Optional[Dict[str, Any]]:
"""扩展操作卡(S4):action ∈ build|trial_run|validate|confirm|publish|run → 对应扩展工具。"""
ext_id = str(payload.get("ext_id") or "").strip()
action = str(payload.get("action") or "").strip()
if not ext_id or not action:
return None
tool = {
"build": "build_extension",
"trial_run": "trial_run_extension",
"validate": "validate_extension",
"confirm": "confirm_extension",
"publish": "publish_extension",
"run": "run_extension",
"repair": "repair_extension",
}.get(action)
if tool is None or tool not in tool_manifests:
return None
args: Dict[str, Any] = {"ext_id": ext_id}
if action == "confirm":
if isinstance(payload.get("nl_triggers"), list):
args["nl_triggers"] = payload["nl_triggers"]
if isinstance(payload.get("parameters"), list):
args["parameters"] = payload["parameters"]
if payload.get("runtime_image"):
args["runtime_image"] = str(payload["runtime_image"])
if action == "run":
if isinstance(payload.get("params"), dict):
args["params"] = payload["params"]
if isinstance(payload.get("files"), list):
args["files"] = payload["files"]
result = _execute_tool(tool_manifests, tool, args)
if action == "confirm" and "EXT_STATUS" in result:
# 状态降级话术:confirm 要求 validated/trial_passed 前置状态,
# 编辑修正若发生在校验前,明确提示先试运行/校验,不报 500。
answer = (
"该扩展当前状态还不能确认。请先完成 AI 转换与试运行、通过校验后,"
"再确认参数与触发词。"
)
else:
answer = _paraphrase(
llm_client,
f"用户对扩展 {ext_id} 执行了操作 {action}。工具结果:\n{result}",
)
return _make_payload(
f"扩展操作:{action}", answer,
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": [tool], "errors": []},
)
def _handle_card_result(
card_result: Dict[str, Any],
llm_client: Any,
bioflow: Optional[Dict[str, Any]],
tool_manifests: Dict[str, CapabilityManifest],
) -> Optional[Dict[str, Any]]:
card = str(card_result.get("card") or "").strip()
payload = card_result.get("payload") if isinstance(card_result.get("payload"), dict) else {}
ctx = bioflow or {}
if card == CARD_PATH_SELECTION:
return _card_path_selection(payload, llm_client, ctx, tool_manifests)
if card == CARD_FORM_CONFIRM:
return _card_form_confirm(payload, llm_client, ctx, tool_manifests)
if card == CARD_FILE_UPLOAD:
return _card_file_upload(payload, llm_client, ctx, tool_manifests)
if card == CARD_ARCHIVE_CHOICE:
return _card_archive_choice(payload, llm_client, ctx, tool_manifests)
if card == CARD_EXTENSION_ACTION:
return _card_extension_action(payload, llm_client, ctx, tool_manifests)
return None
# ---------------------------------------------------------------------------
# 前门路由
# ---------------------------------------------------------------------------
def try_bioflow_agent(
question: str,
*,
llm_client: Any,
history: Optional[List[Dict[str, str]]] = None,
capabilities: Optional[Dict[str, Any]] = None,
card_result: Optional[Dict[str, Any]] = None,
bioflow: Optional[Dict[str, Any]] = None,
progress_callback: Optional[Any] = None,
) -> Optional[Dict[str, Any]]:
"""前门路由入口。命中生信能力则返回答案 payload,否则返回 None(走原 pipeline)。
- card_result:非空时走卡片回填协议(直接路由到对应工具,不做意图预筛)。
- 任何异常都不上抛——记日志后返回 None,保证金标路径永不因本模块失败。
"""
try:
return _try_bioflow_agent_impl(
question,
llm_client=llm_client,
history=history,
capabilities=capabilities,
card_result=card_result,
bioflow=bioflow,
progress_callback=progress_callback,
)
except Exception as exc: # noqa: BLE001
LOGGER.warning("bioflow agent: 执行失败,回退原 pipeline: %s", exc)
return None
def _try_bioflow_agent_impl(
question: str,
*,
llm_client: Any,
history: Optional[List[Dict[str, str]]],
capabilities: Optional[Dict[str, Any]],
card_result: Optional[Dict[str, Any]],
bioflow: Optional[Dict[str, Any]],
progress_callback: Optional[Any],
) -> Optional[Dict[str, Any]]:
# 门 1:会话开关(熔断)
if not _env_enabled():
return None
# card_result 卡片操作优先:用户已在卡片上明确选择,不做意图预筛,直接路由
if isinstance(card_result, dict) and str(card_result.get("card") or "").strip() in CARD_TYPES:
return _route_card_result(card_result, llm_client, capabilities, bioflow, progress_callback)
# NL 路径:活跃任务/生信链续接优先,其次意图预筛
active_job_id = (bioflow or {}).get("active_job_id")
chain = str((bioflow or {}).get("chain") or "").strip()
if active_job_id:
# 有活跃任务:跳过意图预筛,按任务 chain 续接(任务链不可变)
manifests = _registered_capabilities(capabilities)
cap_name = _chain_to_capability(chain)
if cap_name:
manifests = [m for m in manifests if m.name == cap_name]
else:
# 无活跃任务:先门 2 + 门 3 意图预筛(新任务入口)
manifests = _candidate_capabilities(question, capabilities)
# 意图未命中但 session 留有生信链(上轮规划)→ 续接该链(如路径选择「1」)
if not manifests and chain:
cap_name = _chain_to_capability(chain)
if cap_name:
registered = _registered_capabilities(capabilities)
manifests = [m for m in registered if m.name == cap_name]
if not manifests:
return None
tools, tool_manifests = _build_tools(manifests)
if not tools:
LOGGER.warning("bioflow agent: 意图命中但 MCP 工具发现为空,回退原 pipeline")
return None
if progress_callback is not None:
try:
progress_callback(
{
"stage": "bioflow_agent_started",
"capabilities": [m.name for m in manifests],
"tool_count": len(tools),
}
)
except Exception: # noqa: BLE001
pass
system = _build_system(manifests)
runner = ToolRunner(make_llm_complete(llm_client), max_rounds=MAX_ROUNDS)
content, audit = runner.run(system, question, tools, tool_manifests, history=history)
answer = (content or "").strip()
if not answer:
return None
tool_results = (audit or {}).get("tool_results") or {}
readiness = None
versioning = None
new_job_id = None
plan = None
for name, result_text in tool_results.items():
if name in ("create_gene_family_job", "create_rnaseq_job"):
jid = _extract_job_id(result_text)
if jid:
new_job_id = jid
data = _try_parse_json(result_text)
if not isinstance(data, dict):
continue
if name.startswith("plan_") and data.get("ok"):
plan = {
"candidates": data.get("candidates") or [],
"auto_filled": data.get("auto_filled") or [],
"reuse_archive": data.get("reuse_archive"),
"reuse_name": data.get("reuse_name"),
"fallback": data.get("fallback"),
}
if name == "check_readiness":
readiness = _map_readiness(data)
elif name == "propose_change":
versioning = _map_versioning(data)
effective_chain = chain or (_CAPABILITY_TO_CHAIN.get(manifests[0].name) if manifests else "")
return _make_payload(
question,
answer,
audit=audit,
capabilities=[m.name for m in manifests],
active_job_id=(bioflow or {}).get("active_job_id") or new_job_id,
chain=effective_chain or None,
readiness=readiness,
versioning=versioning,
plan=plan,
)
def _route_card_result(
card_result: Dict[str, Any],
llm_client: Any,
capabilities: Optional[Dict[str, Any]],
bioflow: Optional[Dict[str, Any]],
progress_callback: Optional[Any],
) -> Optional[Dict[str, Any]]:
"""卡片操作路由:确定相关能力 → 构建工具 → 分发到对应 card handler。"""
card = str(card_result.get("card") or "").strip()
payload = card_result.get("payload") if isinstance(card_result.get("payload"), dict) else {}
chain = str(payload.get("chain") or (bioflow or {}).get("chain") or "").strip()
if card == CARD_EXTENSION_ACTION:
# 扩展操作只依赖 bioflow-extension 能力,避免对三个能力各做一次 list_tools
cap_name = "bioflow-extension"
else:
cap_name = _chain_to_capability(chain)
registered = _registered_capabilities(capabilities)
if cap_name:
manifests = [m for m in registered if m.name == cap_name]
else:
manifests = registered
if not manifests:
return None
tools, tool_manifests = _build_tools(manifests)
if not tools:
LOGGER.warning("bioflow agent: card_result 但 MCP 工具发现为空,回退原 pipeline")
return None
if progress_callback is not None:
try:
progress_callback({"stage": "bioflow_card", "card": card_result.get("card")})
except Exception: # noqa: BLE001
pass
return _handle_card_result(card_result, llm_client, bioflow, tool_manifests)
@@ -128,6 +128,8 @@ class InstallManager:
"allowed_llm_phases": manifest.allowed_llm_phases,
"session_default": manifest.session_default,
}
if manifest.nl_triggers:
raw["nl_triggers"] = manifest.nl_triggers
if manifest.schema:
raw["schema"] = manifest.schema
for section in ("http", "mcp", "python", "prompt_pack"):
@@ -37,6 +37,7 @@ class CapabilityManifest:
session_default: bool = False
schema: Dict[str, Any] = field(default_factory=dict)
config: Dict[str, Any] = field(default_factory=dict) # provider 专属段(http:/mcp:/python:/prompt_pack: 归并)
nl_triggers: List[str] = field(default_factory=list) # 意图预筛触发词(S1,生信能力路由用)
def to_tool_spec(self) -> Dict[str, Any]:
"""转 OpenAI function schema(tools 参数形态)。"""
@@ -110,6 +111,19 @@ def _parse_entry(raw: Any, index: int, load_warnings: List[str]) -> CapabilityMa
if isinstance(raw.get(section), dict):
config[section] = raw[section]
config["session_default"] = bool(raw.get("session_default", False))
# nl_triggers(S1):意图预筛触发词,只收字符串、去空、去重、限长
nl_triggers_raw = raw.get("nl_triggers") or []
if isinstance(nl_triggers_raw, str):
nl_triggers_raw = [nl_triggers_raw]
nl_triggers: List[str] = []
if isinstance(nl_triggers_raw, list):
for trigger in nl_triggers_raw:
if not isinstance(trigger, str):
continue # 触发词只收字符串,非字符串(数字等)丢弃
cleaned = _clean_text(trigger, 64)
if cleaned and cleaned not in nl_triggers:
nl_triggers.append(cleaned)
nl_triggers = nl_triggers[:64]
manifest = CapabilityManifest(
name=name,
provider=provider,
@@ -119,6 +133,7 @@ def _parse_entry(raw: Any, index: int, load_warnings: List[str]) -> CapabilityMa
session_default=bool(raw.get("session_default", False)),
schema=schema,
config=config,
nl_triggers=nl_triggers,
)
# 大小上限:单条目序列化后 ≤8KB
import json
@@ -130,6 +130,7 @@ class _McpConnection:
self._thread: Optional[threading.Thread] = None
self._session: Any = None
self._server_tools: List[str] = []
self._server_tool_schemas: List[Dict[str, Any]] = []
self._started_at = 0.0
self._last_used = 0.0
self._restarts = 0
@@ -200,6 +201,18 @@ class _McpConnection:
raise
self._ctx = ctx
self._server_tools = [t.name for t in tools_result.tools]
self._server_tool_schemas = [
{
"name": t.name,
"description": (getattr(t, "description", None) or ""),
"input_schema": (
getattr(t, "inputSchema", None)
or getattr(t, "input_schema", None)
or {}
),
}
for t in tools_result.tools
]
self._session = session
self._started_at = time.time()
@@ -225,6 +238,7 @@ class _McpConnection:
ctx = getattr(self, "_ctx", None)
self._ctx = None
self._server_tools = []
self._server_tool_schemas = []
if session is not None:
try:
await session.__aexit__(None, None, None)
@@ -274,6 +288,14 @@ class _McpConnection:
return []
return list(self._server_tools)
def list_tool_schemas(self) -> List[Dict[str, Any]]:
"""返回 MCP server 全量工具 schema(name/description/input_schema)。连接失败返回空。"""
try:
self._ensure_session()
except ToolExecutionError:
return []
return list(self._server_tool_schemas)
def call(self, tool_name: str, arguments: Dict[str, Any]) -> str:
self._ensure_session()
try:
@@ -337,6 +359,11 @@ class McpToolProvider:
return tools + [f"<缺失声明工具 {tool_name}>"]
return tools
def list_tool_schemas(self, manifest: CapabilityManifest) -> List[Dict[str, Any]]:
"""返回该能力 MCP server 的全量工具 schema(供前门 agent 构建 LLM 工具表)。"""
conn = self._connection(manifest)
return conn.list_tool_schemas()
def reap_idle(self) -> int:
reaped = 0
with self._lock:
@@ -448,6 +475,10 @@ class ToolDispatcher:
return PythonToolProvider.execute(manifest, arguments)
raise ToolExecutionError(f"工具 {manifest.name} provider={provider} 不支持调用")
def list_mcp_tools(self, manifest: CapabilityManifest) -> List[Dict[str, Any]]:
"""MCP 能力全量工具 schema(bioflow_agent 构建 LLM 工具表用)。"""
return self._mcp.list_tool_schemas(manifest)
def reap_idle(self) -> int:
return self._mcp.reap_idle()
@@ -0,0 +1,201 @@
"""test_bioflow_agent.py —— bioflow_agent 前门三重门 + nl_triggers 解析单测。
覆盖 S1 验收要点:
- 熔断开关(QA_V1_ENABLE_BIOFLOW_AGENT)默认关 → 返回 None(走原 pipeline)
- 意图预筛(nl_triggers)各自评估,未命中返回 None
- 注册表未注册 / 会话开关关 → 返回 None
- 命中后短路返回自然语言 payload(bioflow.handled=True)
- 只装一个能力时,另一个能力的触发语不响应(隔离)
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.manifests import parse_capabilities_text # noqa: E402
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
BIOFLOW_YAML = """
capabilities:
- name: bioflow-gene-family
provider: mcp
permission: read_only
description: 基因家族分析
session_default: true
nl_triggers: [基因家族, 家族成员, 亚家族, 建树]
mcp:
url: http://127.0.0.1:8098/mcp/gene-family/
prompt_pack:
file: bioflow_gene_family_agent_rules.md
- name: bioflow-rnaseq
provider: mcp
permission: read_only
description: 转录组分析
session_default: true
nl_triggers: [转录组, RNA-seq, 差异表达, 富集]
mcp:
url: http://127.0.0.1:8098/mcp/rnaseq/
prompt_pack:
file: bioflow_rnaseq_agent_rules.md
"""
def _make_registry(tmp_path: Path) -> CapabilityRegistry:
path = tmp_path / "capabilities.yaml"
path.write_text(BIOFLOW_YAML, encoding="utf-8")
registry = CapabilityRegistry(path)
registry.load()
return registry
class FakeLLMClient:
def __init__(self, content: str = "已为您规划分析路径。") -> None:
self.content = content
def complete_raw_messages(self, system, messages, tools=None, **kwargs):
return {"content": self.content, "tool_calls": [], "finish_reason": "stop"}
class TestNlTriggers:
def test_parse_nl_triggers(self):
manifests, warnings = parse_capabilities_text(BIOFLOW_YAML)
assert warnings == []
gf = next(m for m in manifests if m.name == "bioflow-gene-family")
assert gf.nl_triggers == ["基因家族", "家族成员", "亚家族", "建树"]
assert gf.config["mcp"]["url"].endswith("/mcp/gene-family/")
assert gf.config["prompt_pack"]["file"] == "bioflow_gene_family_agent_rules.md"
def test_nl_triggers_dedupe_and_clean(self):
text = """
capabilities:
- name: t
provider: mcp
nl_triggers: [基因家族, 基因家族, "", " 亚家族 ", 123]
"""
manifests, _ = parse_capabilities_text(text)
assert manifests[0].nl_triggers == ["基因家族", "亚家族"]
class TestIntentMatch:
def test_match(self):
assert ba._intent_match("帮我做水稻SPL基因家族分析", ["基因家族"]) is True
assert ba._intent_match("转录组差异表达分析", ["转录组", "RNA-seq"]) is True
assert ba._intent_match("什么是光合作用", ["基因家族", "转录组"]) is False
class TestGateLogic:
def test_fuse_off_returns_none(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict("os.environ", {}, clear=False):
ba.os.environ.pop(ba.ENV_FUSE, None)
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
def test_no_intent_match_returns_none(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"光合作用的电子传递链是什么", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
def test_active_job_continuation_skips_intent_prescreen(self, tmp_path):
"""#011:有活跃任务时,续接输入(不含触发词)也应进入生信流,不掉普通 QA。"""
registry = _make_registry(tmp_path)
fake_tools = [
{
"type": "function",
"function": {
"name": "fetch_source_file",
"description": "获取源文件",
"parameters": {"type": "object", "properties": {}},
},
}
]
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {"fetch_source_file": mock.ANY})):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"基因组文件在 https://example.com/rice.fa",
llm_client=FakeLLMClient(),
capabilities=None,
bioflow={"active_job_id": "job-1", "chain": "gene_family"},
)
assert result is not None
assert result["success"] is True
assert result["bioflow"]["handled"] is True
assert result["bioflow"]["capabilities"] == ["bioflow-gene-family"]
def test_not_registered_returns_none(self, tmp_path):
empty = CapabilityRegistry(tmp_path / "empty.yaml")
empty.load()
with mock.patch.object(ba, "get_registry", return_value=empty):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
def test_session_switch_off_returns_none(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"帮我做基因家族分析",
llm_client=FakeLLMClient(),
capabilities={"bioflow-gene-family": False},
)
assert result is None
def test_isolation_only_installed_responds(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
caps = ba._candidate_capabilities("帮我做基因家族分析", None)
assert [m.name for m in caps] == ["bioflow-gene-family"]
caps = ba._candidate_capabilities("转录组差异表达", None)
assert [m.name for m in caps] == ["bioflow-rnaseq"]
def test_full_flow_short_circuits(self, tmp_path):
registry = _make_registry(tmp_path)
fake_tools = [
{
"type": "function",
"function": {
"name": "plan_gene_family",
"description": "规划基因家族路径",
"parameters": {"type": "object", "properties": {}},
},
}
]
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {"plan_gene_family": mock.ANY})):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is not None
assert result["success"] is True
assert result["answer"] == "已为您规划分析路径。"
assert result["bioflow"]["handled"] is True
assert result["bioflow"]["capabilities"] == ["bioflow-gene-family"]
def test_exception_falls_back_to_none(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.object(ba, "_candidate_capabilities", side_effect=RuntimeError("boom")):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
@@ -0,0 +1,255 @@
"""test_bioflow_agent_s2.py —— S2 对话式编排闭环单测。
覆盖:card_result 回填协议(四类卡片)、参数抽取(白名单)、多轮记忆注入、
session↔job 绑定上下文(active_job_id/chain)、job_status 主动查询。
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest # noqa: E402
from rag.qa_v1_1_73.tool_runner import ToolRunner # noqa: E402
def _manifest(name="bioflow-gene-family", url="http://x/mcp/gene-family/"):
return CapabilityManifest(name=name, provider="mcp", config={"mcp": {"url": url, "tool": name}})
def _tool_manifests():
"""覆盖四类卡片用到的工具 + job_status。"""
names = ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
"register_job_files", "fetch_source_file", "job_status")
out = {}
for t in names:
out[t] = CapabilityManifest(
name="bioflow-gene-family", provider="mcp",
config={"mcp": {"url": "http://x/mcp/gene-family/", "tool": t}},
)
return out
class FakeDispatcher:
def __init__(self, results=None):
self.results = results or {}
self.calls = []
def execute(self, manifest, arguments):
tool = manifest.config.get("mcp", {}).get("tool")
self.calls.append((tool, arguments))
return self.results.get(tool, '{"ok": true, "data": {"id": "job-1"}}')
class FakeLLM:
def __init__(self, complete_text="已收到,已为您处理。", json_payload=None):
self.complete_text = complete_text
self.json_payload = json_payload or {}
def complete(self, system, user, **kw):
return self.complete_text
def complete_json(self, system, user, **kw):
return self.json_payload
def complete_raw_messages(self, system, messages, tools=None, **kw):
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
# ---------------------------------------------------------------------------
# 参数抽取
# ---------------------------------------------------------------------------
class TestExtractParams:
def test_extract_known_and_unknown(self):
llm = FakeLLM(json_payload={"params": {"threads": 8, "padj": 0.01}, "unknown": ["bogus"]})
form_defaults = {"threads": {"default": 4}, "padj": {"default": 0.05}}
known, unknown = ba._extract_params(llm, "用 8 核,padj 用 0.01", form_defaults)
assert known == {"threads": 8, "padj": 0.01}
assert unknown == ["bogus"]
def test_non_whitelist_param_filtered(self):
llm = FakeLLM(json_payload={"params": {"evil": 1, "threads": 4}, "unknown": []})
form_defaults = {"threads": {"default": 4}}
known, unknown = ba._extract_params(llm, "x", form_defaults)
assert known == {"threads": 4}
def test_bad_json_returns_empty(self):
llm = FakeLLM(json_payload="not-a-dict")
form_defaults = {"threads": {}}
known, unknown = ba._extract_params(llm, "x", form_defaults)
assert known == {} and unknown == []
def test_empty_whitelist(self):
llm = FakeLLM()
known, unknown = ba._extract_params(llm, "x", {})
assert known == {} and unknown == []
# ---------------------------------------------------------------------------
# card_result 回填协议
# ---------------------------------------------------------------------------
class TestCardResult:
def test_path_selection_creates_job_and_extracts_job_id(self):
dispatcher = FakeDispatcher()
llm = FakeLLM()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "gene_family", "stages": ["identify", "tree"], "title": "水稻SPL"}},
llm, {"session_id": "s1"}, _tool_manifests(),
)
assert out is not None
assert out["bioflow"]["active_job_id"] == "job-1"
assert out["bioflow"]["chain"] == "gene_family"
assert dispatcher.calls[0][0] == "create_gene_family_job"
assert dispatcher.calls[0][1]["stages"] == ["identify", "tree"]
assert dispatcher.calls[0][1]["session_id"] == "s1"
def test_path_selection_missing_stages_returns_none(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "path_selection", "payload": {"chain": "gene_family"}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is None
def test_form_confirm_calls_confirm_stage(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "form_confirm",
"payload": {"job_id": "job-9", "stage": "identify", "params": {"evalue": 1e-5}}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "confirm_stage"
assert dispatcher.calls[0][1] == {"job_id": "job-9", "stage": "identify", "params": {"evalue": 1e-5}}
def test_form_confirm_uses_active_job_id_fallback(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
ba._handle_card_result(
{"card": "form_confirm", "payload": {"stage": "identify", "params": {}}},
FakeLLM(), {"active_job_id": "job-7"}, _tool_manifests(),
)
assert dispatcher.calls[0][1]["job_id"] == "job-7"
def test_file_upload_splits_remote_and_local(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "file_upload",
"payload": {"job_id": "j1", "paths": ["https://x/a.fa", "/data/b.fa"]}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
tools = [c[0] for c in dispatcher.calls]
assert "fetch_source_file" in tools and "register_job_files" in tools
def test_archive_choice_paraphrases(self):
out = ba._handle_card_result(
{"card": "archive_choice", "payload": {"job_id": "j1", "archive": True}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert out["answer"] == "已收到,已为您处理。"
def test_unknown_card_returns_none(self):
out = ba._handle_card_result({"card": "bogus", "payload": {}}, FakeLLM(), {}, _tool_manifests())
assert out is None
def test_path_selection_reuse_failure_no_stages_returns_none(self):
"""bug ②:复用失败且无显式 stages → 返回 None(不空链建任务)。"""
dispatcher = FakeDispatcher({"reuse_archive": '{"ok": false, "code": "ARCHIVE_NOT_FOUND"}'})
tm = _tool_manifests()
tm["reuse_archive"] = CapabilityManifest(
name="bioflow-gene-family", provider="mcp",
config={"mcp": {"url": "http://x", "tool": "reuse_archive"}},
)
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "gene_family", "reuse_archive": "不存在的存档"}},
FakeLLM(), {}, tm,
)
assert out is None
# 只执行了 reuse_archive,未调用 create_gene_family_job
assert [c[0] for c in dispatcher.calls] == ["reuse_archive"]
# ---------------------------------------------------------------------------
# 前门路由(card_result / bioflow 上下文)
# ---------------------------------------------------------------------------
class TestRouteCardResult:
def test_try_bioflow_agent_card_result_short_circuits(self):
registry_manifest = _manifest()
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_registered_capabilities", return_value=[registry_manifest]):
with mock.patch.object(ba, "_build_tools", return_value=([{"type": "function", "function": {"name": "confirm_stage", "description": "", "parameters": {"type": "object", "properties": {}}}}], {"confirm_stage": registry_manifest})):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba.try_bioflow_agent(
"确认", llm_client=FakeLLM(),
card_result={"card": "form_confirm", "payload": {"job_id": "j1", "stage": "s", "params": {}}},
)
assert out is not None
assert out["bioflow"]["handled"] is True
def test_try_bioflow_agent_fuse_off_ignores_card_result(self):
with mock.patch.dict("os.environ", {}, clear=False):
ba.os.environ.pop(ba.ENV_FUSE, None)
out = ba.try_bioflow_agent(
"确认", llm_client=FakeLLM(),
card_result={"card": "form_confirm", "payload": {}},
)
assert out is None
# ---------------------------------------------------------------------------
# 多轮记忆注入(ToolRunner.run history)
# ---------------------------------------------------------------------------
class TestMultiTurnHistory:
def test_history_injected_into_messages(self):
class RecordingLLM:
def __init__(self):
self.captured_messages = None
def complete_raw_messages(self, system, messages, tools=None, **kw):
self.captured_messages = list(messages)
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
from rag.qa_v1_1_73.tool_runner import make_llm_complete
llm = RecordingLLM()
runner = ToolRunner(make_llm_complete(llm), max_rounds=1)
history = [{"role": "user", "content": "上一轮问题"}, {"role": "assistant", "content": "上一轮答案"}]
runner.run("sys", "当前问题", [], {}, history=history)
msgs = llm.captured_messages
assert msgs[0] == {"role": "user", "content": "上一轮问题"}
assert msgs[1] == {"role": "assistant", "content": "上一轮答案"}
assert msgs[-1] == {"role": "user", "content": "当前问题"}
def test_history_filters_bad_roles(self):
class RecordingLLM:
def __init__(self):
self.captured_messages = None
def complete_raw_messages(self, system, messages, tools=None, **kw):
self.captured_messages = list(messages)
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
from rag.qa_v1_1_73.tool_runner import make_llm_complete
llm = RecordingLLM()
runner = ToolRunner(make_llm_complete(llm), max_rounds=1)
history = [{"role": "system", "content": "evil"}, {"role": "user", "content": ""}, {"role": "user", "content": "ok"}]
runner.run("sys", "q", [], {}, history=history)
roles = [m["role"] for m in llm.captured_messages]
assert "system" not in roles
assert llm.captured_messages[-1] == {"role": "user", "content": "q"}
@@ -0,0 +1,511 @@
"""test_bioflow_agent_s2_supplement.py —— S2 覆盖清单补强测试。
补强 dev 的 test_bioflow_agent_s2.py 未覆盖的分支:
- _chain_to_capability / _extract_job_id / _execute_tool / _paraphrase / _query_job_status
- card 四类 handler 缺必需字段回退 None、files 键、archive=False、非 dict payload
- _route_card_result:chain 过滤 / 未注册能力→None / tools 空→None
- NL 路径注入 bioflow 上下文(active_job_id/chain)
- system prompt 硬规则在位(追问 ≤3 / 不可代确认 / 工具名不外泄)
- ToolRunner.run(history) 边界(non-dict 条目 / history=None / 空内容剔除)
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest # noqa: E402
from rag.qa_v1_1_73.tool_runner import ToolRunner, make_llm_complete # noqa: E402
def _manifest(name="bioflow-gene-family", url="http://x/mcp/gene-family/"):
return CapabilityManifest(name=name, provider="mcp", config={"mcp": {"url": url, "tool": name}})
def _tool_manifests():
names = ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
"register_job_files", "fetch_source_file", "job_status",
"rerun_stage", "reuse_archive", "archive_job")
out = {}
for t in names:
out[t] = CapabilityManifest(
name="bioflow-gene-family", provider="mcp",
config={"mcp": {"url": "http://x/mcp/gene-family/", "tool": t}},
)
return out
class FakeLLM:
def __init__(self, complete_text="已收到。", json_payload=None, complete_error=None):
self.complete_text = complete_text
self.json_payload = json_payload or {}
self.complete_error = complete_error
def complete(self, system, user, **kw):
if self.complete_error:
raise self.complete_error
return self.complete_text
def complete_json(self, system, user, **kw):
return self.json_payload
def complete_raw_messages(self, system, messages, tools=None, **kw):
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
class FakeDispatcher:
def __init__(self, results=None):
self.results = results or {}
self.calls = []
def execute(self, manifest, arguments):
tool = manifest.config.get("mcp", {}).get("tool")
self.calls.append((tool, arguments))
return self.results.get(tool, '{"ok": true, "data": {"id": "job-1"}}')
# ===========================================================================
# 辅助函数
# ===========================================================================
class TestChainToCapability:
def test_known(self):
assert ba._chain_to_capability("gene_family") == "bioflow-gene-family"
assert ba._chain_to_capability("rnaseq") == "bioflow-rnaseq"
def test_unknown_and_empty(self):
assert ba._chain_to_capability("wgs") is None
assert ba._chain_to_capability("") is None
assert ba._chain_to_capability(None) is None
def test_case_and_whitespace(self):
assert ba._chain_to_capability(" Gene_Family ") == "bioflow-gene-family"
class TestExtractJobId:
def test_data_id(self):
assert ba._extract_job_id('{"ok": true, "data": {"id": "job-9"}}') == "job-9"
def test_data_job_id(self):
assert ba._extract_job_id('{"ok": true, "data": {"job_id": "job-8"}}') == "job-8"
def test_flat_id(self):
assert ba._extract_job_id('{"ok": true, "id": "job-7"}') == "job-7"
def test_non_json_and_no_id(self):
assert ba._extract_job_id("not json") is None
assert ba._extract_job_id('{"ok": false}') is None
assert ba._extract_job_id("") is None
class TestExecuteTool:
def test_unknown_tool(self):
out = ba._execute_tool(_tool_manifests(), "ghost", {})
assert "未注册" in out
def test_tool_exception_becomes_text(self):
class Boom:
def execute(self, manifest, arguments):
raise RuntimeError("boom")
with mock.patch.object(ba, "get_dispatcher", return_value=Boom()):
out = ba._execute_tool(_tool_manifests(), "job_status", {"job_id": "j"})
assert "工具执行失败" in out
class TestParaphrase:
def test_llm_success(self):
out = ba._paraphrase(FakeLLM(complete_text="已转述。"), "原文")
assert out == "已转述。"
def test_llm_exception_falls_back_to_raw(self):
out = ba._paraphrase(FakeLLM(complete_error=RuntimeError("boom")), "原文")
assert out == "原文"
def test_llm_empty_falls_back_to_raw(self):
out = ba._paraphrase(FakeLLM(complete_text=" "), "原文")
assert out == "原文"
class TestQueryJobStatus:
def test_calls_job_status_and_paraphrases(self):
with mock.patch.object(ba, "_execute_tool", return_value='{"ok": true, "data": {"status": "running"}}'):
out = ba._query_job_status("j1", _tool_manifests(), FakeLLM(complete_text="任务进行中。"))
assert out == "任务进行中。"
# ===========================================================================
# card 四类 handler 边界(缺必需字段 / files 键 / archive=False / 非 dict payload)
# ===========================================================================
class TestCardHandlersEdge:
def test_form_confirm_missing_job_and_stage_returns_none(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "form_confirm", "payload": {"params": {}}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is None
def test_file_upload_missing_job_returns_none(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "file_upload", "payload": {"paths": ["/data/a.fa"]}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is None
def test_file_upload_missing_files_returns_none(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "file_upload", "payload": {"job_id": "j1"}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is None
def test_file_upload_uses_files_key(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "file_upload",
"payload": {"job_id": "j1", "files": ["https://x/a.fa"]}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "fetch_source_file"
def test_file_upload_string_files_coerced(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "file_upload",
"payload": {"job_id": "j1", "files": "https://x/a.fa"}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "fetch_source_file"
def test_archive_choice_missing_job_returns_none(self):
out = ba._handle_card_result(
{"card": "archive_choice", "payload": {"archive": True}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is None
def test_archive_choice_skip(self):
out = ba._handle_card_result(
{"card": "archive_choice", "payload": {"job_id": "j1", "archive": False}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert out["bioflow"]["audit"]["tool_calls"] == 0 # 存档不调工具(走 REST)
def test_path_selection_rnaseq_chain(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "rnaseq", "stages": ["rnaseq_qc"]}},
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "create_rnaseq_job"
def test_card_non_dict_payload_treated_empty(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "form_confirm", "payload": "not-a-dict"},
FakeLLM(), {"active_job_id": "j1"}, _tool_manifests(),
)
assert out is None # payload 非 dict → 无 stage/job_id → None
def test_card_result_with_non_dict_is_ignored(self):
# try_bioflow_agent 对非 dict 的 card_result 不做 card 路由,走 NL 意图
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_candidate_capabilities", return_value=[]):
out = ba.try_bioflow_agent("确认", llm_client=FakeLLM(), card_result="bogus")
assert out is None
# ===========================================================================
# _route_card_result 边界
# ===========================================================================
class TestRouteCardResultEdge:
def test_chain_filters_to_capability(self):
gf = _manifest("bioflow-gene-family")
rn = _manifest("bioflow-rnaseq", "http://x/mcp/rnaseq/")
captured = {}
def fake_build(manifests):
captured["manifests"] = [m.name for m in manifests]
return ([], {})
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf, rn]):
with mock.patch.object(ba, "_build_tools", side_effect=fake_build):
ba._route_card_result(
{"card": "form_confirm", "payload": {"chain": "rnaseq"}},
FakeLLM(), None, None, None,
)
assert captured["manifests"] == ["bioflow-rnaseq"] # chain 过滤到 rnaseq 能力
def test_chain_to_unregistered_capability_returns_none(self):
gf = _manifest("bioflow-gene-family")
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
out = ba._route_card_result(
{"card": "form_confirm", "payload": {"chain": "rnaseq"}},
FakeLLM(), None, None, None,
)
assert out is None # rnaseq 未注册 → manifests 空 → None
def test_tools_empty_returns_none(self):
gf = _manifest("bioflow-gene-family")
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
with mock.patch.object(ba, "_build_tools", return_value=([], {})):
out = ba._route_card_result(
{"card": "form_confirm", "payload": {}},
FakeLLM(), None, None, None,
)
assert out is None
def test_progress_callback_invoked(self):
gf = _manifest("bioflow-gene-family")
calls = []
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
with mock.patch.object(ba, "_handle_card_result", return_value=None):
ba._route_card_result(
{"card": "form_confirm", "payload": {}},
FakeLLM(), None, None, lambda p: calls.append(p),
)
assert calls and calls[0]["stage"] == "bioflow_card"
assert calls[0]["card"] == "form_confirm"
def test_progress_callback_exception_swallowed(self):
gf = _manifest("bioflow-gene-family")
def boom(p):
raise RuntimeError("callback boom")
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
with mock.patch.object(ba, "_handle_card_result", return_value={"ok": True}):
out = ba._route_card_result(
{"card": "form_confirm", "payload": {}},
FakeLLM(), None, None, boom,
)
assert out == {"ok": True} # 回调异常被吞,不影响主流程
# ===========================================================================
# NL 路径:bioflow 上下文注入 / 卡片与 NL 语义等价
# ===========================================================================
class TestNLPathBioflowContext:
def test_nl_path_injects_bioflow_context(self, tmp_path):
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry
from rag.qa_v1_1_73.capabilities.manifests import parse_capabilities_text
yaml = """
capabilities:
- name: bioflow-gene-family
provider: mcp
permission: read_only
session_default: true
nl_triggers: [基因家族]
mcp:
url: http://127.0.0.1:8098/mcp/gene-family/
"""
(tmp_path / "c.yaml").write_text(yaml, encoding="utf-8")
registry = CapabilityRegistry(tmp_path / "c.yaml")
registry.load()
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
"description": "x",
"parameters": {"type": "object", "properties": {}}}}]
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
out = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLM(),
bioflow={"session_id": "s1", "active_job_id": "job-9", "chain": "gene_family"},
)
assert out is not None
assert out["bioflow"]["active_job_id"] == "job-9"
assert out["bioflow"]["chain"] == "gene_family"
def test_card_and_nl_share_tool_chain(self):
"""卡片操作与纯 NL 服务端语义等价:卡片 handler 只调 MCP 工具,不直连控制面。"""
# 卡片 handler 用到的工具名 = MCP 工具(job_status/confirm_stage/create_* 等)
tm = _tool_manifests()
# 四类卡片触发的工具都在 tool_manifests(即 MCP 工具表)里
for tool in ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
"register_job_files", "fetch_source_file", "job_status"):
assert tool in tm
class TestExtractParamsEdge:
def test_llm_exception_returns_empty(self):
class RaisingLLM(FakeLLM):
def complete_json(self, system, user, **kw):
raise RuntimeError("boom")
known, unknown = ba._extract_params(RaisingLLM(), "x", {"threads": {}})
assert known == {} and unknown == []
def test_unknown_as_string_coerced_to_list(self):
llm = FakeLLM(json_payload={"params": {"threads": 8}, "unknown": "bogus_param"})
known, unknown = ba._extract_params(llm, "x", {"threads": {}})
assert known == {"threads": 8}
assert unknown == ["bogus_param"]
def test_path_selection_with_params(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "gene_family", "stages": ["identify"],
"params": {"prefix": "Ft.SPL"}}},
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
)
assert dispatcher.calls[0][1]["params"] == {"prefix": "Ft.SPL"}
def test_path_selection_create_tool_missing_returns_none(self):
tm = _tool_manifests()
tm.pop("create_gene_family_job")
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "gene_family", "stages": ["identify"]}},
FakeLLM(), {"session_id": "s1"}, tm,
)
assert out is None
# ===========================================================================
# system prompt 硬规则在位
# ===========================================================================
class TestSystemPromptRules:
def test_param_retry_rule(self):
assert "最多 3 次" in ba._SYSTEM_BASE
def test_human_boundary_rule(self):
assert "不可跳过" in ba._SYSTEM_BASE
assert "不可代用户确认" in ba._SYSTEM_BASE
def test_tool_name_no_leak_rule(self):
assert "绝不向用户泄露工具名" in ba._SYSTEM_BASE
# ===========================================================================
# ToolRunner.run(history) 边界
# ===========================================================================
class TestHistoryEdge:
def _run_with_history(self, history):
captured = {}
class RecordingLLM:
def complete_raw_messages(self, system, messages, tools=None, **kw):
captured["messages"] = list(messages)
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
runner = ToolRunner(make_llm_complete(RecordingLLM()), max_rounds=1)
runner.run("sys", "当前", [], {}, history=history)
return captured["messages"]
def test_none_history(self):
msgs = self._run_with_history(None)
assert msgs == [{"role": "user", "content": "当前"}]
def test_non_dict_items_skipped(self):
msgs = self._run_with_history(["not-a-dict", {"role": "user", "content": "ok"}])
assert msgs == [{"role": "user", "content": "ok"}, {"role": "user", "content": "当前"}]
def test_empty_content_filtered(self):
msgs = self._run_with_history([{"role": "user", "content": " "}])
assert msgs == [{"role": "user", "content": "当前"}]
# ===========================================================================
# S3:card_result rerun / reuse_archive / archive_job 语义
# ===========================================================================
class TestS3CardSemantics:
def test_form_confirm_rerun_true_uses_rerun_stage(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "form_confirm",
"payload": {"job_id": "j1", "stage": "identify",
"params": {"evalue": 1e-5}, "rerun": True}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "rerun_stage"
assert out["bioflow"]["audit"]["tool_names"] == ["rerun_stage"]
def test_form_confirm_rerun_false_uses_confirm_stage(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
ba._handle_card_result(
{"card": "form_confirm",
"payload": {"job_id": "j1", "stage": "identify", "params": {}}},
FakeLLM(), {}, _tool_manifests(),
)
assert dispatcher.calls[0][0] == "confirm_stage"
def test_path_selection_reuse_archive(self):
dispatcher = FakeDispatcher({
"reuse_archive": '{"ok": true, "data": {"id": "arch-1", '
'"stage_chain": ["identify", "phylogeny"], '
'"params": {"identify": {"prefix": "Ft"}}}}',
})
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "path_selection",
"payload": {"chain": "gene_family", "reuse_archive": "SPL 鉴定"}},
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
)
assert out is not None
tools = [c[0] for c in dispatcher.calls]
assert tools[0] == "reuse_archive"
assert tools[1] == "create_gene_family_job"
create_args = dispatcher.calls[1][1]
assert create_args["archive_id"] == "arch-1"
assert create_args["stages"] == ["identify", "phylogeny"]
assert create_args["params"] == {"identify": {"prefix": "Ft"}}
assert out["bioflow"]["audit"]["tool_names"] == ["reuse_archive", "create_gene_family_job"]
def test_archive_choice_archive_true_calls_archive_job(self):
dispatcher = FakeDispatcher({"archive_job": '{"ok": true, "data": {"id": "a1"}}'})
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "archive_choice",
"payload": {"job_id": "j1", "archive": True,
"name": "SPL 存档", "description": "d"}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == "archive_job"
assert dispatcher.calls[0][1] == {"job_id": "j1", "name": "SPL 存档", "description": "d"}
assert out["bioflow"]["audit"]["tool_names"] == ["archive_job"]
def test_archive_choice_archive_false_no_tool_call(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "archive_choice",
"payload": {"job_id": "j1", "archive": False}},
FakeLLM(), {}, _tool_manifests(),
)
assert out is not None
assert dispatcher.calls == [] # 跳过存档不发工具请求
assert out["bioflow"]["audit"]["tool_calls"] == 0
@@ -0,0 +1,206 @@
"""test_bioflow_agent_s4_supplement.py —— S4 覆盖清单补强测试。
覆盖:
- _card_extension_action 六种 action 分发到对应扩展工具
- 未知 action / 缺 ext_id/action 回退 None
- confirm 携带 nl_triggers/parameters/runtime_image;EXT_STATUS 降级话术(确定性)
- run 携带 params/files
- 真实 configs/capabilities.yaml 三条目(含 bioflow-extension)可解析
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest, parse_capabilities_file # noqa: E402
EXT_TOOLS = (
"create_extension", "extension_status", "build_extension", "trial_run_extension",
"validate_extension", "confirm_extension", "publish_extension", "run_extension",
"repair_extension",
)
def _ext_tool_manifests():
out = {}
for t in EXT_TOOLS:
out[t] = CapabilityManifest(
name="bioflow-extension", provider="mcp",
config={"mcp": {"url": "http://127.0.0.1:8098/mcp/extension/", "tool": t}},
)
return out
class FakeLLM:
def complete(self, system, user, **kw):
return "已收到。"
def complete_json(self, system, user, **kw):
return {}
def complete_raw_messages(self, system, messages, tools=None, **kw):
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
class FakeDispatcher:
def __init__(self, results=None):
self.results = results or {}
self.calls = []
def execute(self, manifest, arguments):
tool = manifest.config.get("mcp", {}).get("tool")
self.calls.append((tool, arguments))
return self.results.get(tool, '{"ok": true, "data": {"ext_id": "e1"}}')
# ===========================================================================
# _card_extension_action 分发
# ===========================================================================
class TestExtensionAction:
@pytest.mark.parametrize("action,expected_tool", [
("build", "build_extension"),
("trial_run", "trial_run_extension"),
("validate", "validate_extension"),
("confirm", "confirm_extension"),
("publish", "publish_extension"),
("run", "run_extension"),
("repair", "repair_extension"),
])
def test_action_dispatches_to_tool(self, action, expected_tool):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1", "action": action}},
FakeLLM(), {}, _ext_tool_manifests(),
)
assert out is not None
assert dispatcher.calls[0][0] == expected_tool
assert dispatcher.calls[0][1] == {"ext_id": "e1"}
assert out["bioflow"]["audit"]["tool_names"] == [expected_tool]
def test_unknown_action_returns_none(self):
dispatcher = FakeDispatcher()
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "bogus"}},
FakeLLM(), {}, _ext_tool_manifests(),
)
assert out is None
def test_missing_ext_id_or_action_returns_none(self):
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
assert ba._handle_card_result(
{"card": "extension_action", "payload": {"action": "build"}},
FakeLLM(), {}, _ext_tool_manifests(),
) is None
assert ba._handle_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1"}},
FakeLLM(), {}, _ext_tool_manifests(),
) is None
def test_confirm_passes_triggers_params_image(self):
dispatcher = FakeDispatcher()
payload = {"ext_id": "e1", "action": "confirm",
"nl_triggers": ["扩展", "触发词"], "parameters": [{"name": "x"}],
"runtime_image": "img"}
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
ba._handle_card_result({"card": "extension_action", "payload": payload},
FakeLLM(), {}, _ext_tool_manifests())
args = dispatcher.calls[0][1]
assert args == {"ext_id": "e1", "nl_triggers": ["扩展", "触发词"],
"parameters": [{"name": "x"}], "runtime_image": "img"}
def test_confirm_status_insufficient_uses_deterministic_script(self):
# confirm 返回 EXT_STATUS(状态不足)→ 确定性降级话术(不报 500、不经 LLM)
dispatcher = FakeDispatcher({
"confirm_extension": '{"ok": false, "code": "EXT_STATUS", "message": "状态不可确认"}',
})
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
out = ba._handle_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "confirm"}},
FakeLLM(), {}, _ext_tool_manifests(),
)
assert out is not None
assert "还不能确认" in out["answer"]
assert "先完成" in out["answer"] # 确定性话术(非 LLM 转述)
def test_run_passes_params_and_files(self):
dispatcher = FakeDispatcher()
payload = {"ext_id": "e1", "action": "run",
"params": {"cpu": 8}, "files": ["/data/a.fa"]}
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
ba._handle_card_result({"card": "extension_action", "payload": payload},
FakeLLM(), {}, _ext_tool_manifests())
assert dispatcher.calls[0][1] == {"ext_id": "e1", "params": {"cpu": 8}, "files": ["/data/a.fa"]}
def test_extension_tool_not_in_manifests_returns_none(self):
# tool_manifests 里没有扩展工具 → 回退 None(能力未安装/未发现工具)
empty = CapabilityManifest(name="bioflow-gene-family", provider="mcp",
config={"mcp": {"url": "http://x", "tool": "job_status"}})
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
out = ba._handle_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "build"}},
FakeLLM(), {}, {"job_status": empty},
)
assert out is None
# ===========================================================================
# 真实 capabilities.yaml 三条目可解析
# ===========================================================================
class TestCapabilitiesThreeEntries:
def _real_yaml(self) -> Path:
return Path(__file__).resolve().parents[5] / "agent" / "qa" / "configs" / "capabilities.yaml"
def test_three_bioflow_entries(self):
manifests, warnings = parse_capabilities_file(self._real_yaml())
by_name = {m.name: m for m in manifests}
assert {"bioflow-gene-family", "bioflow-rnaseq", "bioflow-extension"} <= set(by_name)
ext = by_name["bioflow-extension"]
assert ext.provider == "mcp"
assert ext.permission == "read_only"
assert ext.nl_triggers
assert ext.config["mcp"]["url"].endswith("/mcp/extension/")
assert ext.config["prompt_pack"]["file"] == "bioflow_extension_agent_rules.md"
def test_bioflow_extension_in_capability_names(self):
assert "bioflow-extension" in ba.BIOFLOW_CAPABILITY_NAMES
assert ba.CARD_EXTENSION_ACTION == "extension_action"
assert ba.CARD_EXTENSION_ACTION in ba.CARD_TYPES
# ===========================================================================
# S5:_route_card_result 对 extension_action 只发现 extension 能力(一次 list_tools)
# ===========================================================================
class TestRouteExtensionOptimization:
def test_extension_action_discovers_only_extension_capability(self):
gf = CapabilityManifest(name="bioflow-gene-family", provider="mcp",
config={"mcp": {"url": "http://x/gene-family/"}})
rn = CapabilityManifest(name="bioflow-rnaseq", provider="mcp",
config={"mcp": {"url": "http://x/rnaseq/"}})
ext = CapabilityManifest(name="bioflow-extension", provider="mcp",
config={"mcp": {"url": "http://x/extension/"}})
captured = {}
def fake_build(manifests):
captured["names"] = [m.name for m in manifests]
return ([], {})
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf, rn, ext]):
with mock.patch.object(ba, "_build_tools", side_effect=fake_build):
ba._route_card_result(
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "build"}},
FakeLLM(), None, None, None,
)
# 只把 bioflow-extension manifest 交给 _build_tools(避免对三能力各 list_tools 一次)
assert captured["names"] == ["bioflow-extension"]
@@ -0,0 +1,116 @@
"""test_bioflow_agent_s6_config.py —— S6 多配置回归(QA 单元层)。
模拟四种部署配置,断言候选能力集合 + 意图预筛结果正确(真实 E2E/金标 20 题依赖部署,
此处以 monkeypatch/env 在单元层覆盖漏判安全语义)。
"""
from __future__ import annotations
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
THREE_YAML = """
capabilities:
- name: bioflow-gene-family
provider: mcp
permission: read_only
session_default: true
nl_triggers: [基因家族, 亚家族]
mcp: {url: "http://x/gene-family/"}
- name: bioflow-rnaseq
provider: mcp
permission: read_only
session_default: true
nl_triggers: [转录组, 差异表达]
mcp: {url: "http://x/rnaseq/"}
- name: bioflow-extension
provider: mcp
permission: read_only
session_default: true
nl_triggers: [自定义扩展]
mcp: {url: "http://x/extension/"}
"""
def _registry(tmp_path, yaml_text=THREE_YAML):
path = tmp_path / "caps.yaml"
path.write_text(yaml_text, encoding="utf-8")
reg = CapabilityRegistry(path)
reg.load()
return reg
class FakeLLM:
def complete_raw_messages(self, *a, **kw):
return {"content": "x", "tool_calls": [], "finish_reason": "stop"}
# ===========================================================================
# 配置 1:全关(QA_V1_ENABLE_BIOFLOW_AGENT=0 + 三能力全卸载)
# ===========================================================================
class TestConfigAllOff:
def test_fuse_off_returns_none(self, tmp_path):
reg = _registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=reg):
with mock.patch.dict(ba.os.environ, {}, clear=False):
ba.os.environ.pop(ba.ENV_FUSE, None)
out = ba.try_bioflow_agent("帮我做基因家族分析", llm_client=FakeLLM())
assert out is None # 走原 pipeline
def test_all_uninstalled_no_candidate(self, tmp_path):
reg = _registry(tmp_path, "capabilities: []\n") # 三能力全卸载
with mock.patch.object(ba, "get_registry", return_value=reg):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
assert ba._candidate_capabilities("帮我做基因家族分析", None) == []
# ===========================================================================
# 配置 2:只开基因家族
# ===========================================================================
class TestConfigGeneFamilyOnly:
def test_gene_family_hit_rnaseq_miss(self, tmp_path):
reg = _registry(tmp_path)
caps = {"bioflow-gene-family": True, "bioflow-rnaseq": False, "bioflow-extension": False}
with mock.patch.object(ba, "get_registry", return_value=reg):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
names = [m.name for m in ba._candidate_capabilities("帮我做水稻 SPL 基因家族分析", caps)]
assert names == ["bioflow-gene-family"]
assert ba._candidate_capabilities("帮我做转录组差异表达分析", caps) == []
# ===========================================================================
# 配置 3:只开转录组
# ===========================================================================
class TestConfigRnaseqOnly:
def test_rnaseq_hit_gene_family_miss(self, tmp_path):
reg = _registry(tmp_path)
caps = {"bioflow-gene-family": False, "bioflow-rnaseq": True, "bioflow-extension": False}
with mock.patch.object(ba, "get_registry", return_value=reg):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
names = [m.name for m in ba._candidate_capabilities("转录组差异表达分析", caps)]
assert names == ["bioflow-rnaseq"]
assert ba._candidate_capabilities("帮我做水稻 SPL 基因家族分析", caps) == []
# ===========================================================================
# 配置 4:全开但非生信提问
# ===========================================================================
class TestConfigAllOnNonBioflow:
def test_non_bioflow_question_no_candidate(self, tmp_path):
reg = _registry(tmp_path)
caps = {"bioflow-gene-family": True, "bioflow-rnaseq": True, "bioflow-extension": True}
with mock.patch.object(ba, "get_registry", return_value=reg):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
assert ba._candidate_capabilities("光合作用的电子传递链是什么", caps) == []
assert ba._candidate_capabilities("什么是转录因子", caps) == []
@@ -0,0 +1,496 @@
"""test_bioflow_agent_supplement.py —— S1 覆盖清单补强测试。
补强 dev 的 test_bioflow_agent.py 未覆盖的分支(对照 S1 覆盖清单):
- 熔断开关值解析(1/true/yes/on 与 0/false/off/空/缺失)
- nl_triggers 同时命中 → 两个候选(交 LLM 依 description 选)
- session_default 回退(True/False)/ registry 访问异常 / registry.get 异常
- _build_tools:MCP 工具发现(空 url 跳过 / 发现异常 / 共享连接按能力名 / 无名跳过 / bad schema 兜底)
- _build_system:prompt_pack 注入 / 去重 / 加载异常兜底
- try_bioflow_agent:tools 空→None、answer 空→None、progress_callback 触发与吞异常
- ToolRunner max_rounds=5 / bioflow_agent.MAX_ROUNDS 常量
- make_llm_complete 包装(正常 / 异常)
- 真实 configs/capabilities.yaml 两条目可解析
- bioflow_agent 不 import pipeline/answer_composer(红线静态守卫)
"""
from __future__ import annotations
import os
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
from rag.qa_v1_1_73.capabilities.manifests import ( # noqa: E402
CapabilityManifest,
parse_capabilities_file,
)
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
BIOFLOW_YAML = """
capabilities:
- name: bioflow-gene-family
provider: mcp
permission: read_only
description: 基因家族分析
session_default: true
nl_triggers: [基因家族, 家族成员, 亚家族, 建树]
mcp:
url: http://127.0.0.1:8098/mcp/gene-family/
prompt_pack:
file: bioflow_gene_family_agent_rules.md
- name: bioflow-rnaseq
provider: mcp
permission: read_only
description: 转录组分析
session_default: true
nl_triggers: [转录组, RNA-seq, 差异表达, 富集]
mcp:
url: http://127.0.0.1:8098/mcp/rnaseq/
prompt_pack:
file: bioflow_rnaseq_agent_rules.md
"""
def _make_registry(tmp_path: Path, yaml_text: str = BIOFLOW_YAML) -> CapabilityRegistry:
path = tmp_path / "capabilities.yaml"
path.write_text(yaml_text, encoding="utf-8")
registry = CapabilityRegistry(path)
registry.load()
return registry
def _gf_manifest(**overrides) -> CapabilityManifest:
cfg = {
"mcp": {"url": "http://127.0.0.1:8098/mcp/gene-family/"},
"prompt_pack": {"file": "bioflow_gene_family_agent_rules.md"},
}
cfg.update(overrides.pop("config", {}))
return CapabilityManifest(
name="bioflow-gene-family", provider="mcp", permission="read_only",
description="基因家族分析", config=cfg, nl_triggers=["基因家族"], **overrides,
)
class FakeLLMClient:
def __init__(self, content: str = "已为您规划分析路径。") -> None:
self.content = content
def complete_raw_messages(self, system, messages, tools=None, **kwargs):
return {"content": self.content, "tool_calls": [], "finish_reason": "stop"}
# ===========================================================================
# 门 1:熔断开关值解析
# ===========================================================================
class TestEnvFuse:
@pytest.mark.parametrize("value", ["1", "true", "TRUE", "yes", "on", " On ", "YES"])
def test_truthy(self, value):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: value}):
assert ba._env_enabled() is True
@pytest.mark.parametrize("value", ["0", "false", "off", "", "no", " "])
def test_falsy(self, value):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: value}):
assert ba._env_enabled() is False
def test_missing_defaults_off(self):
with mock.patch.dict(ba.os.environ, {}, clear=False):
ba.os.environ.pop(ba.ENV_FUSE, None)
assert ba._env_enabled() is False
# ===========================================================================
# 门 3:nl_triggers 意图预筛
# ===========================================================================
class TestIntentMatchEdge:
def test_empty_triggers_never_match(self):
assert ba._intent_match("做基因家族分析", []) is False
assert ba._intent_match("做基因家族分析", [""]) is False
def test_case_insensitive_substring(self):
assert ba._intent_match("RNA-SEQ 分析", ["rna-seq"]) is True
assert ba._intent_match("ka/ks 选择压力", ["Ka/Ks"]) is True
# ===========================================================================
# 门 2 + 门 3:候选能力(registry 异常 / session_default 回退 / 同时命中)
# ===========================================================================
class TestCandidateCapabilities:
def test_registry_access_exception_returns_empty(self):
with mock.patch.object(ba, "get_registry", side_effect=RuntimeError("boom")):
assert ba._candidate_capabilities("做基因家族", None) == []
def test_registry_get_exception_skips(self):
class BadRegistry:
def get(self, name):
raise RuntimeError("boom")
with mock.patch.object(ba, "get_registry", return_value=BadRegistry()):
assert ba._candidate_capabilities("做基因家族", None) == []
def test_both_triggers_match_returns_both(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
caps = ba._candidate_capabilities("做基因家族和转录组差异表达分析", None)
assert [m.name for m in caps] == ["bioflow-gene-family", "bioflow-rnaseq"]
def test_session_default_false_skips(self, tmp_path):
yaml = BIOFLOW_YAML.replace("session_default: true", "session_default: false")
registry = _make_registry(tmp_path, yaml)
with mock.patch.object(ba, "get_registry", return_value=registry):
# 无会话开关时回退 session_default=false → 不候选
assert ba._candidate_capabilities("做基因家族分析", None) == []
# 显式会话开关 true → 候选
caps = ba._candidate_capabilities("做基因家族分析", {"bioflow-gene-family": True})
assert [m.name for m in caps] == ["bioflow-gene-family"]
def test_session_switch_true_overrides_default_false(self, tmp_path):
yaml = BIOFLOW_YAML.replace("session_default: true", "session_default: false", 1)
registry = _make_registry(tmp_path, yaml)
with mock.patch.object(ba, "get_registry", return_value=registry):
caps = ba._candidate_capabilities("做基因家族分析", {"bioflow-gene-family": True})
assert [m.name for m in caps] == ["bioflow-gene-family"]
# ===========================================================================
# _build_tools:MCP 工具发现
# ===========================================================================
class _FakeDispatcher:
def __init__(self, tools_by_name=None, exc=None):
self.tools_by_name = tools_by_name or {}
self.exc = exc
def list_mcp_tools(self, manifest):
if self.exc:
raise self.exc
return self.tools_by_name.get(manifest.name, [])
class TestBuildTools:
def test_empty_url_skipped(self):
manifest = CapabilityManifest(
name="x", provider="mcp", config={"mcp": {"url": ""}}
)
with mock.patch.object(ba, "get_dispatcher", return_value=_FakeDispatcher()):
tools, tool_manifests = ba._build_tools([manifest])
assert tools == []
assert tool_manifests == {}
def test_discovery_exception_skipped(self):
manifest = _gf_manifest()
with mock.patch.object(
ba, "get_dispatcher",
return_value=_FakeDispatcher(exc=RuntimeError("mcp down")),
):
tools, tool_manifests = ba._build_tools([manifest])
assert tools == []
assert tool_manifests == {}
def test_builds_schema_and_shared_connection(self):
dispatcher = _FakeDispatcher({
"bioflow-gene-family": [
{"name": "plan_gene_family", "description": "规划路径",
"input_schema": {"type": "object", "properties": {"message": {"type": "string"}}}},
]
})
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
tools, tool_manifests = ba._build_tools([_gf_manifest()])
assert len(tools) == 1
assert tools[0]["function"]["name"] == "plan_gene_family"
assert tools[0]["function"]["description"] == "规划路径"
# 共享连接:manifest.name 用能力名,mcp.tool 指向真实 server 工具名
tm = tool_manifests["plan_gene_family"]
assert tm.name == "bioflow-gene-family"
assert tm.config["mcp"]["tool"] == "plan_gene_family"
assert tm.config["mcp"]["url"].endswith("/mcp/gene-family/")
def test_skips_nameless_and_bad_schema(self):
dispatcher = _FakeDispatcher({
"bioflow-gene-family": [
{"name": "", "description": "无名", "input_schema": {}},
{"name": "job_status", "description": "查状态", "input_schema": "not-a-dict"},
]
})
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
tools, tool_manifests = ba._build_tools([_gf_manifest()])
assert [t["function"]["name"] for t in tools] == ["job_status"]
assert tools[0]["function"]["parameters"] == {"type": "object", "properties": {}}
def test_shared_tool_deduped_across_capabilities(self):
"""两能力同时命中:共享工具去重(只保留首个能力),专有工具各自保留。
覆盖 bug ① 修复:job_status 等 12 个共享工具两端点语义一致,按名去重后
LLM tools 列表无重名,且 job_status 路由到首个能力(gene-family)。
"""
dispatcher = _FakeDispatcher({
"bioflow-gene-family": [
{"name": "plan_gene_family", "description": "规划基因家族", "input_schema": {}},
{"name": "job_status", "description": "查状态", "input_schema": {}},
{"name": "confirm_stage", "description": "确认参数", "input_schema": {}},
],
"bioflow-rnaseq": [
{"name": "plan_rnaseq", "description": "规划转录组", "input_schema": {}},
{"name": "job_status", "description": "查状态", "input_schema": {}},
{"name": "confirm_stage", "description": "确认参数", "input_schema": {}},
],
})
rn = CapabilityManifest(
name="bioflow-rnaseq", provider="mcp", permission="read_only",
description="转录组分析", nl_triggers=["转录组"],
config={"mcp": {"url": "http://127.0.0.1:8098/mcp/rnaseq/"}},
)
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
tools, tool_manifests = ba._build_tools([_gf_manifest(), rn])
names = [t["function"]["name"] for t in tools]
# 专有工具各保留 + 共享工具只保留一份,且无重名
assert names == ["plan_gene_family", "job_status", "confirm_stage", "plan_rnaseq"]
assert len(names) == len(set(names))
# 共享工具路由到首个能力(gene-family)
assert tool_manifests["job_status"].name == "bioflow-gene-family"
assert tool_manifests["job_status"].config["mcp"]["url"].endswith("/mcp/gene-family/")
assert tool_manifests["confirm_stage"].name == "bioflow-gene-family"
# 专有工具各自路由到自身能力
assert tool_manifests["plan_rnaseq"].name == "bioflow-rnaseq"
assert tool_manifests["plan_rnaseq"].config["mcp"]["url"].endswith("/mcp/rnaseq/")
assert tool_manifests["plan_gene_family"].name == "bioflow-gene-family"
# ===========================================================================
# _build_system:prompt_pack 注入
# ===========================================================================
class TestBuildSystem:
def test_appends_prompt_pack(self):
manifest = _gf_manifest()
with mock.patch.object(ba.PromptPackProvider, "load", return_value="基因家族规则正文"):
system = ba._build_system([manifest])
assert ba._SYSTEM_BASE in system
assert "基因家族规则正文" in system
def test_dedup_same_file_once(self):
m1 = _gf_manifest()
m2 = _gf_manifest() # 同 file
calls = []
with mock.patch.object(ba.PromptPackProvider, "load", side_effect=lambda m, root: calls.append(m) or "RULES"):
system = ba._build_system([m1, m2])
assert len(calls) == 1 # 去重后只加载一次
assert system.count("RULES") == 1
def test_empty_file_skipped(self):
manifest = CapabilityManifest(name="x", provider="mcp", config={"prompt_pack": {"file": ""}})
system = ba._build_system([manifest])
assert system == ba._SYSTEM_BASE
def test_load_exception_falls_back(self):
manifest = _gf_manifest()
with mock.patch.object(ba.PromptPackProvider, "load", side_effect=RuntimeError("boom")):
system = ba._build_system([manifest])
assert ba._SYSTEM_BASE in system # 不炸,只用基础提示
# ===========================================================================
# try_bioflow_agent:tools 空 / answer 空 / progress_callback
# ===========================================================================
class TestFullFlowEdge:
def test_tools_empty_returns_none(self, tmp_path):
registry = _make_registry(tmp_path)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_build_tools", return_value=([], {})):
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
def test_answer_empty_returns_none(self, tmp_path):
registry = _make_registry(tmp_path)
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
"description": "x",
"parameters": {"type": "object", "properties": {}}}}]
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
FakeRunner.return_value.run.return_value = ("", {"rounds": 1})
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
)
assert result is None
def test_progress_callback_invoked(self, tmp_path):
registry = _make_registry(tmp_path)
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
"description": "x",
"parameters": {"type": "object", "properties": {}}}}]
calls = []
def cb(payload):
calls.append(payload)
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(),
capabilities=None, progress_callback=cb,
)
assert result is not None
assert calls and calls[0]["stage"] == "bioflow_agent_started"
assert calls[0]["capabilities"] == ["bioflow-gene-family"]
assert calls[0]["tool_count"] == 1
def test_progress_callback_exception_swallowed(self, tmp_path):
registry = _make_registry(tmp_path)
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
"description": "x",
"parameters": {"type": "object", "properties": {}}}}]
def cb(payload):
raise RuntimeError("callback boom")
with mock.patch.object(ba, "get_registry", return_value=registry):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
result = ba.try_bioflow_agent(
"帮我做基因家族分析", llm_client=FakeLLMClient(),
capabilities=None, progress_callback=cb,
)
assert result is not None # 回调异常被吞,不影响主流程
assert result["answer"] == "已答复"
# ===========================================================================
# ToolRunner max_rounds=5 / MAX_ROUNDS 常量
# ===========================================================================
class TestMaxRounds:
def test_max_rounds_constant_is_5(self):
assert ba.MAX_ROUNDS == 5
def test_tool_runner_respects_max_rounds_5(self):
from rag.qa_v1_1_73.tool_runner import ToolRunner
class LLM:
def __call__(self, system, messages, tools):
return "", [{"id": "c", "name": "echo", "arguments": "{}"}], None
class Dispatcher:
def execute(self, manifest, arguments):
return "ok"
manifest = CapabilityManifest(name="echo", provider="python")
tools = [{"type": "function", "function": {"name": "echo", "description": "e",
"parameters": {"type": "object", "properties": {}}}}]
runner = ToolRunner(LLM(), dispatcher=Dispatcher(), max_rounds=5)
content, audit = runner.run("sys", "user", tools, {"echo": manifest})
assert audit["rounds"] == 5
assert audit["tool_calls"] == 5
# ===========================================================================
# make_llm_complete 包装
# ===========================================================================
class TestMakeLLMComplete:
def test_normal(self):
from rag.qa_v1_1_73.tool_runner import make_llm_complete
class Client:
def complete_raw_messages(self, system, messages, tools=None, **kw):
return {"content": "hi", "tool_calls": [{"name": "x"}], "finish_reason": "stop"}
complete = make_llm_complete(Client())
content, tool_calls, error = complete("sys", [{"role": "user", "content": "q"}], [])
assert content == "hi"
assert tool_calls == [{"name": "x"}]
assert error is None
def test_exception_becomes_error(self):
from rag.qa_v1_1_73.tool_runner import make_llm_complete
class Client:
def complete_raw_messages(self, *a, **kw):
raise RuntimeError("llm down")
complete = make_llm_complete(Client())
content, tool_calls, error = complete("sys", [], [])
assert content == ""
assert tool_calls == []
assert "RuntimeError" in error
# ===========================================================================
# 真实 configs/capabilities.yaml 两条目可解析
# ===========================================================================
class TestRealCapabilitiesYaml:
def _real_yaml(self) -> Path:
repo_root = Path(__file__).resolve().parents[5]
return repo_root / "agent" / "qa" / "configs" / "capabilities.yaml"
def test_both_bioflow_entries_parse(self):
manifests, warnings = parse_capabilities_file(self._real_yaml())
by_name = {m.name: m for m in manifests}
assert "bioflow-gene-family" in by_name
assert "bioflow-rnaseq" in by_name
gf = by_name["bioflow-gene-family"]
assert gf.provider == "mcp"
assert gf.permission == "read_only"
assert gf.nl_triggers
assert gf.config["mcp"]["url"].endswith("/mcp/gene-family/")
assert gf.config["prompt_pack"]["file"] == "bioflow_gene_family_agent_rules.md"
rn = by_name["bioflow-rnaseq"]
assert rn.permission == "read_only"
assert rn.nl_triggers
assert rn.config["mcp"]["url"].endswith("/mcp/rnaseq/")
assert rn.config["prompt_pack"]["file"] == "bioflow_rnaseq_agent_rules.md"
def test_nl_triggers_cover_expected_words(self):
manifests, _ = parse_capabilities_file(self._real_yaml())
gf = next(m for m in manifests if m.name == "bioflow-gene-family")
rn = next(m for m in manifests if m.name == "bioflow-rnaseq")
# 关键触发词在位(意图预筛依赖)
assert "基因家族" in gf.nl_triggers
assert "转录组" in rn.nl_triggers
# ===========================================================================
# 红线静态守卫:bioflow_agent 不依赖 pipeline / answer_composer
# ===========================================================================
class TestRedLineNoPipelineDep:
def test_no_pipeline_or_answer_composer_import(self):
import ast
src = Path(ba.__file__).read_text(encoding="utf-8")
tree = ast.parse(src)
imported = []
for node in tree.body:
if isinstance(node, ast.Import):
imported += [a.name for a in node.names]
elif isinstance(node, ast.ImportFrom):
imported.append(node.module or "")
assert not any("pipeline" in m or "answer_composer" in m for m in imported)
def test_never_raises(self, tmp_path):
# 任何异常都返回 None/正常 payload,绝不外抛
with mock.patch.object(ba, "get_registry", side_effect=RuntimeError("boom")):
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
result = ba.try_bioflow_agent("做基因家族", llm_client=FakeLLMClient())
assert result is None
+17 -2
View File
@@ -51,9 +51,13 @@ class ToolRunner:
user: str,
tools: List[Dict[str, Any]],
manifests: Dict[str, CapabilityManifest],
history: Optional[List[Dict[str, str]]] = None,
) -> Tuple[str, Dict[str, Any]]:
"""执行工具调用循环。返回 (content, audit)。
history:多轮记忆(S2),List[{role: user|assistant, content}],旧→新,
不含当前这轮提问;注入为 messages 前置轮次(作为上下文,不参与工具结果回填)。
audit: {"rounds": n, "tool_calls": k, "used_json_fallback": bool,
"tool_names": [...], "errors": [...]}
"""
@@ -62,9 +66,19 @@ class ToolRunner:
"tool_calls": 0,
"used_json_fallback": False,
"tool_names": [],
"tool_results": {},
"errors": [],
}
messages: List[Dict[str, Any]] = [{"role": "user", "content": user}]
messages: List[Dict[str, Any]] = []
for h in history or []:
if not isinstance(h, dict):
continue
role = str(h.get("role") or "").strip()
content = str(h.get("content") or "").strip()
if role not in {"user", "assistant"} or not content:
continue
messages.append({"role": role, "content": content})
messages.append({"role": "user", "content": user})
system_text = system
for round_index in range(1, self.max_rounds + 1):
audit["rounds"] = round_index
@@ -82,6 +96,7 @@ class ToolRunner:
break
assistant_message: Dict[str, Any] = {"role": "assistant", "content": content or ""}
tool_call_blocks: List[Dict[str, Any]] = []
messages.append(assistant_message)
for call in tool_calls:
name = str(call.get("name") or call.get("function", {}).get("name") or "")
raw_args = call.get("arguments") or call.get("function", {}).get("arguments") or {}
@@ -112,9 +127,9 @@ class ToolRunner:
"function": {"name": name or "unknown", "arguments": json.dumps(arguments, ensure_ascii=False)},
}
)
audit["tool_results"][name] = result_text
messages.append({"role": "tool", "tool_call_id": call_id, "content": result_text})
assistant_message["tool_calls"] = tool_call_blocks
messages.append(assistant_message)
continue
# 最后一轮:拿最终文本(无工具或轮次用尽后由调用方用 messages 再 complete 一次)
return self._final_text(system_text, messages, tools), audit
+85
View File
@@ -0,0 +1,85 @@
---
name: bioflow-analysis
description: >-
PlantAI 生信分析编排 Skill。当用户提供原始组学数据(蛋白 FASTA/基因组/FASTQ/表达矩阵等)
并提出基因家族、转录组、变异或自定义分析需求时使用。负责从原始数据到交付报告的
全流程编排:选链路、登记资产、就绪检查、参数确认、执行、诊断、版本化修改、交付清单。
触发词见 nl_triggers;仅在明确涉及生信分析时触发。
nl_triggers:
- 基因家族
- 家族成员
- 亚家族
- 系统发育
- 建树
- 进化树
- motif
- 结构域
- 共线性
- Ka/Ks
- 转录组
- RNA-seq
- RNAseq
- rnaseq
- 差异表达
- 表达定量
- 富集
- WGCNA
- GSEA
- fastq
- 实验设计
- 生信分析
- 变异检测
- GWAS
---
# bioflow-analysis 能力声明
PlantAI 生信分析 Skill,面向「原始数据 → 交付报告」的分析编排。所有分析经
`bioflowctl`(唯一事实源)与 11 个分析控制 MCP 工具完成,服务端状态经 SQLite 持久化。
## 11 工具白名单
Skill 只允许调用以下 11 个分析控制工具(其余为内部实现,不外泄):
| 工具 | 职责 | 对应 CLI |
|------|------|----------|
| get_task_context | 任务绑定(session/job/plan_revision/run_id/当前阶段) | — |
| inspect_asset | 资产/文件探测(格式/预览/列结构) | file inspect |
| validate_plan | 读单一参数 schema 校验参数 | (补齐) |
| check_readiness | 四象限就绪检查(环境/参考/磁盘/预算) | check readiness |
| get_run_evidence | 运行证据(日志切片/失败原因/manifest 校验值) | — |
| propose_change | 参数变更影响范围 | change propose |
| apply_change | 幂等应用变更 | change apply |
| execute_nodes | 只重跑受影响节点 | run node-execute |
| control_execution | 执行所有权(stop/resume) | run stop/resume |
| validate_delivery | 交付清单完整性校验 | delivery validate |
| publish_delivery | 发布交付清单 | delivery publish |
## 对话剧本
1. **原始数据**:识别用户提供的原始数据(蛋白 FASTA / 基因组 / FASTQ / 表达矩阵 / 设计表)。
2. **选链路**:`inspect_asset` 探测数据格式;据此选择分析链路(基因家族 / 转录组 / …)。
3. **登记**:登记数据资产(角色化 `asset register`),保证「下游直入」可用。
4. **就绪**:`check_readiness` 四象限检查;缺项返回结构化错误阻断,不启动。
5. **参数确认**:`validate_plan` 校验参数;未知参数/越界拒绝。
6. **执行**:`execute_nodes` 只重跑受影响节点;`control_execution` 管理执行所有权。
7. **诊断**:失败时 `get_run_evidence` 取日志切片/manifest 校验值,按 8 类错误分类
(input/resource/environment/transient/adapter/artifact/reference/scientific_precondition)。
8. **版本化**:`propose_change` + `apply_change` 版本化修改,历史不覆盖、幂等。
9. **交付**:`validate_delivery` + `publish_delivery` 校验并发布交付清单。
## 人工边界(requires_human)
- `subfamily_confirm`:亚家族分类确认,须人工确认,不可代用户。
- `rnaseq_de`:差异表达实验设计确认,须人工确认,不可代用户。
上述阶段在 pending_actions 中 `requires_human=true`,Skill 只呈现表单默认值,不代确认。
## 验收清单
- [ ] 原始数据格式已探测(inspect_asset)
- [ ] 就绪检查通过(check_readiness ready=true)
- [ ] 参数经单一 schema 校验(validate_plan)
- [ ] 执行所有权唯一(control_execution 不重复接管/不重叠)
- [ ] 失败已诊断(get_run_evidence + 8 类错误分类)
- [ ] 变更版本化(propose_change/apply_change 幂等、历史不覆盖)
- [ ] 交付清单完整并已发布(validate_delivery ok → publish_delivery approved)
+37 -9
View File
@@ -2,13 +2,21 @@
- 独立部署树 bioflow_service/(FR-BIO-028:独立进程 8098 + 独立 SQLite)。
- lifespan:init_db(幂等迁移)→ init_context(kernel/catalog/planner/runner)
→ recover(服务重启恢复 in-flight 续跑,FR-BIO-031)→ shutdown 清理。
→ recover(服务重启恢复 in-flight 续跑,FR-BIO-031)→ MCP session manager
(anyio task group,streamable-http 处理请求必需)→ shutdown 清理。
- 端点:health(/api/bioflow/health,前端探活 FR-BIO-025)。
- 不依赖 BFF/生产服务(8090/8091 等绝不探测/连接)。
"""
from __future__ import annotations
import logging
from pathlib import Path
from dotenv import load_dotenv
# 把 bioflow_service/.env 注入 os.environ(llm.py 等模块直读 DOCGEN_LLM_*,
# 不走 pydantic-settings 的 env_file,避免「.env 配了但 os.environ 读不到」的配置断层)。
load_dotenv(Path(__file__).resolve().parent.parent / ".env")
from contextlib import asynccontextmanager
from fastapi import FastAPI
@@ -17,11 +25,18 @@ from fastapi.middleware.cors import CORSMiddleware
from .config import get_settings
from .context import context, init_context, shutdown_context
from .db import close_db, init_db
from .mcp_server import extension_mcp, gene_family_mcp, rnaseq_mcp
from .routers import bioarchives, bioflow, biojobs, extension
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# 三个 MCP streamable-http app(调用即创建 session_manager,供 lifespan 进入 run())。
# 端点地址:/mcp/gene-family、/mcp/rnaseq、/mcp/extension(streamable_http_path="/" + 挂载前缀)。
_gene_family_mcp_app = gene_family_mcp.streamable_http_app(streamable_http_path="/")
_rnaseq_mcp_app = rnaseq_mcp.streamable_http_app(streamable_http_path="/")
_extension_mcp_app = extension_mcp.streamable_http_app(streamable_http_path="/")
def create_app() -> FastAPI:
settings = get_settings()
@@ -60,14 +75,21 @@ def create_app() -> FastAPI:
logger.info("main: 重启恢复 %d 个 in-flight 任务续跑", len(recovered))
except Exception:
logger.exception("main: 重启恢复扫描失败(任务不重跑,等待后续操作)")
try:
yield
finally:
await shutdown_context()
try:
await close_db()
except Exception:
pass
# MCP session manager 的 task group 必须在服务运行期内保持活跃
# (streamable-http 处理请求时要求已初始化)。Starlette 不会自动运行
# 子 Mount 的 lifespan,故在此显式进入三个 MCP 的 run()。
async with gene_family_mcp.session_manager.run():
async with rnaseq_mcp.session_manager.run():
async with extension_mcp.session_manager.run():
try:
yield
finally:
await shutdown_context()
try:
await close_db()
except Exception:
pass
app.router.lifespan_context = _lifespan
@@ -75,6 +97,12 @@ def create_app() -> FastAPI:
app.include_router(biojobs.router)
app.include_router(bioarchives.router)
app.include_router(extension.router)
# 生信 MCP 能力外挂(mcp 2.x:MCPServer.streamable_http_app)。
# 三个独立 streamable-http 端点,各自装卸(REST 面不受影响)。
app.mount("/mcp/gene-family", _gene_family_mcp_app)
app.mount("/mcp/rnaseq", _rnaseq_mcp_app)
app.mount("/mcp/extension", _extension_mcp_app)
return app
+900
View File
@@ -0,0 +1,900 @@
"""app/mcp_server.py —— 生信 MCP server(两个 streamable-http 端点)。
- `/mcp/gene-family`:基因家族能力单元(plan/create/ask + 共享控制组 + 文件组)
- `/mcp/rnaseq`:转录组能力单元(plan/create/generate_samples_yaml/ask + 共享控制组 + 文件组)
设计约束(实施方案 S0):
- 工具只承诺「已受理」,执行进度/日志由 SSE 事件总线直推,不走 MCP。
- 每个能力单元只接受本链阶段(create 跨链拒绝 CROSS_CHAIN)。
- 人工确认边界(subfamily_confirm / rnaseq_de 实验设计确认)在 job_status 的
pending_actions 里标 requires_human=true,工具不提供绕过。
- 白名单沿用 runner._allowed_param_names(单一事实源)。
- 工具名不外泄由 QA agent(S1)负责,本模块只负责工具语义。
mcp 2.x 注意:MCPServer(非 FastMCP,来自 mcp.server.mcpserver);挂载用
`mcp.streamable_http_app()`。本文件位于 app/ 层(非 services/),可用 pydantic(mcp
自身也依赖 pydantic 构造 schema)。
"""
from __future__ import annotations
import logging
import re
import uuid
from pathlib import Path
from typing import Any
from .config import get_settings
from .context import load_record
from .services.bioflow import extensions as ext_mod
from .services.bioflow import store
from .services.bioflow.archive import archive_from_job
from .services.bioflow.catalog import GENE_FAMILY_CHAIN
from .services.bioflow.errors import error_result
from .services.bioflow.file_transform import validate_format
from .services.bioflow.runner import RunnerError, ensure_samples_yaml, llm_sample_fallback
from .services.bioflow.source_fetcher import fetch_source
# 薄适配:委托 bioflowctl CLI(唯一事实源),不重复业务逻辑
from bioflowctl.commands import (
cmd_archive,
cmd_archive_reuse,
cmd_file_inspect,
cmd_file_transform,
cmd_source_fetch,
)
logger = logging.getLogger(__name__)
def _mcp_err(code: str, message: str, *, evidence_refs: list[str] | None = None) -> dict[str, Any]:
"""G13:MCP 工具失败返回统一结构化错误对象(code 顶层 + error 对象)。"""
return error_result(code, message, evidence_refs=evidence_refs)
# ---- 能力单元阶段白名单 ----
GENE_FAMILY_STAGES = set(GENE_FAMILY_CHAIN)
RNASEQ_STAGES = {
"rnaseq_qc", "rnaseq_align", "rnaseq_quant", "rnaseq_sample_qc", "rnaseq_de",
"rnaseq_wgcna", "rnaseq_go_enrich", "rnaseq_kegg_enrich", "rnaseq_enrich",
"rnaseq_fgsea", "rnaseq_candidate_family", "rnaseq_report",
"annotation", "expression",
}
# 人工确认边界:这些阶段 waiting_confirm 时 requires_human=true,工具/QA 不可代确认
HUMAN_CONFIRM_STAGES = {"subfamily_confirm", "rnaseq_de"}
# ---------------------------------------------------------------------------
# context 访问(动态读模块全局,测试 monkeypatch 兼容)
# ---------------------------------------------------------------------------
def _ctx():
from . import context as context_mod
return context_mod.context
def _job_detail(record: Any) -> dict[str, Any]:
stages = []
for stage in record.stage_chain:
st = record._stage_states.get(stage, {})
stages.append({
"stage": stage,
"status": st.get("status", "waiting"),
"params": st.get("params") or {},
"attempt": int(st.get("attempt") or 0),
"manifest_path": st.get("manifest_path"),
"log_tail": st.get("log_tail") or "",
"error": st.get("error"),
})
return {
"id": record.id,
"session_id": record.session_id,
"title": record.title,
"status": record.status,
"stages": stages,
"params": record.params,
"auto_continue": record.auto_continue,
"archive_id": record.archive_id,
"progress": record.progress(),
}
# ---------------------------------------------------------------------------
# pending_actions 派生(job_status 工具返回给 QA agent 的下一步动作)
# ---------------------------------------------------------------------------
def derive_pending_actions(record: Any, runner: Any) -> list[dict[str, Any]]:
"""由任务/阶段状态派生待办动作(QA agent 据此引导用户,人工边界不可代确认)。
按阶段链顺序找第一个「卡住」的环节,返回动作清单(通常 0-1 条)。
"""
if record.status == "paused":
return [{"action": "resume", "stage": None, "requires_human": False,
"message": "任务已暂停,可恢复"}]
if record.status == "done":
return [{"action": "archive", "stage": None, "requires_human": False,
"message": "任务已完成,可存档复用"}]
if record.status == "archived":
return [{"action": "none", "stage": None, "requires_human": False,
"message": "任务已存档"}]
for stage in record.stage_chain:
status = record._stage_states.get(stage, {}).get("status", "waiting")
if status == "await_files":
return [{"action": "provide_files", "stage": stage, "requires_human": False,
"message": f"阶段 {stage} 等待输入文件"}]
if status == "waiting_confirm":
human = stage in HUMAN_CONFIRM_STAGES
return [{"action": "confirm", "stage": stage, "requires_human": human,
"form_defaults": runner._form_defaults(stage, record),
"message": (f"阶段 {stage} 需人工确认(不可代确认)" if human
else f"阶段 {stage} 等待参数确认")}]
if status == "failed":
return [{"action": "rerun_or_skip", "stage": stage, "requires_human": False,
"message": f"阶段 {stage} 失败,可重跑或跳过"}]
if status == "partial_success":
# G4:partial_success 独立终态,不被当作 success/done
return [{"action": "rerun_or_skip", "stage": stage, "requires_human": False,
"message": f"阶段 {stage} 部分成功(产物不全),可重跑或跳过"}]
# waiting/running/queued/done/skipped → 继续
return []
# ---------------------------------------------------------------------------
# 工具实现(命名 async 函数,带类型提示,供两个 MCPServer 注册)
# ---------------------------------------------------------------------------
async def _plan(message: str, allowed_stages: set[str]) -> dict[str, Any]:
from .services.bioflow.planner import PlanCandidate, PlannerError, PlanResult
ctx = _ctx()
if ctx.planner is None:
return {"ok": False, "code": "NOT_READY", "message": "规划器未就绪"}
try:
plan = await ctx.planner.plan(message)
except PlannerError as exc:
fallback = [s for s in ("identify", "rnaseq_qc") if s in allowed_stages and ctx.catalog.has(s)]
plan = PlanResult(candidates=[PlanCandidate(purpose="内置默认流程", stages=fallback)],
source="llm_only", message=exc.message)
candidates = []
for c in plan.candidates:
stages = [s for s in c.stages if s in allowed_stages]
if stages:
candidates.append({"purpose": c.purpose, "stages": stages})
return {
"ok": True,
"candidates": candidates,
"source": plan.source,
"reuse_archive": plan.reuse_archive_id,
"reuse_name": plan.reuse_name,
"fallback": plan.message or None,
}
async def _create_job(session_id: str, title: str, stages: list[str],
params: dict[str, Any] | None, auto_continue: bool,
archive_id: str | None, allowed_stages: set[str]) -> dict[str, Any]:
ctx = _ctx()
unknown = [s for s in stages if s not in allowed_stages]
if unknown:
return {"ok": False, "code": "CROSS_CHAIN",
"message": f"阶段不属于该能力单元: {', '.join(unknown)}"}
try:
record = await ctx.runner.create_job(
session_id, title or "生信任务", stages, params or {},
auto_continue=auto_continue, archive_id=archive_id,
)
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
ctx.records[record.id] = record
return {"ok": True, "data": _job_detail(record)}
async def _job_status(job_id: str) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
detail = _job_detail(record)
detail["pending_actions"] = derive_pending_actions(record, ctx.runner)
return {"ok": True, "data": detail}
async def _confirm_stage(job_id: str, stage: str, params: dict[str, Any] | None) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
if stage not in record.stage_chain:
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
try:
await ctx.runner.confirm_stage(record, stage, params or {})
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
"status": ctx.runner.stage_status(record, stage)}}
async def _rerun_stage(job_id: str, stage: str, params: dict[str, Any] | None) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
if stage not in record.stage_chain:
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
try:
await ctx.runner.rerun_stage(record, stage, params)
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
"status": ctx.runner.stage_status(record, stage)}}
async def _skip_stage(job_id: str, stage: str) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
if stage not in record.stage_chain:
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
try:
await ctx.runner.skip_stage(record, stage)
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
"status": ctx.runner.stage_status(record, stage)}}
async def _pause_job(job_id: str) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
try:
await ctx.runner.pause_job(record)
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
return {"ok": True, "data": {"job_id": job_id, "status": record.status}}
async def _resume_job(job_id: str) -> dict[str, Any]:
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
try:
await ctx.runner.resume_job(record)
except RunnerError as exc:
return {"ok": False, "code": exc.code, "message": exc.message}
return {"ok": True, "data": {"job_id": job_id, "status": record.status}}
async def _archive_job(job_id: str, name: str, description: str,
include_intermediate: bool) -> dict[str, Any]:
"""任务存档(薄适配):委托 cmd_archive(唯一事实源),仅保留内存 record 校验 + 事件发布。"""
ctx = _ctx()
record = await load_record(job_id)
if record is None:
return _mcp_err("NOT_FOUND", "任务不存在")
result = await cmd_archive(
job_id, name, description,
db=ctx.db,
record=record,
include_intermediate=bool(include_intermediate),
archive_fn=archive_from_job,
archives_dir=get_settings().archives_dir_abs,
)
if result.get("ok"):
ctx.events.publish("job_archived", {"job_id": job_id, "archive_id": result["data"]["id"]})
return result
# ---------------------------------------------------------------------------
# 扩展能力单元(第三个能力单元 bioflow-extension):复用 extensions.py 状态机
# ---------------------------------------------------------------------------
_EXT_VIEW_KEYS = (
"ext_id", "display_name", "status", "command_template", "original_command",
"runtime_image", "requirements", "parameters", "input_roles", "nl_triggers",
"analysis_type", "golden", "repair_count", "trial", "validation", "last_build",
"created_at", "updated_at",
)
def _ext_view(ext: dict[str, Any]) -> dict[str, Any]:
"""扩展精简视图(剔除 source_files 等超长字段,供 MCP 工具结果返回)。"""
return {k: ext.get(k) for k in _EXT_VIEW_KEYS if k in ext}
def _file_items(files: Any) -> list[tuple[str, bytes]]:
"""把 MCP 入参 [{path, content|data}] 转为 [(rel_path, bytes)]。"""
out: list[tuple[str, bytes]] = []
if not isinstance(files, list):
return out
for f in files:
if not isinstance(f, dict):
continue
path = str(f.get("path") or f.get("name") or "").strip()
if not path:
continue
content = f.get("content")
if content is None:
content = f.get("data") or ""
if isinstance(content, (bytes, bytearray)):
blob = bytes(content)
else:
blob = str(content).encode("utf-8")
out.append((path, blob))
return out
def _ext_err(exc: Exception) -> str:
if isinstance(exc, ext_mod.ExtensionNotFoundError):
return "NOT_FOUND"
if isinstance(exc, ext_mod.ExtensionStatusError):
return "EXT_STATUS"
if isinstance(exc, ext_mod.ExtensionBuildError):
return "EXT_BUILD"
if isinstance(exc, ext_mod.ExtensionRunError):
return "EXT_RUN"
return "EXT_ERROR"
async def _create_extension(display_name: str, command: str, runtime_image: str,
code_files: Any, data_files: Any) -> dict[str, Any]:
code = _file_items(code_files)
if not code:
return {"ok": False, "code": "VALIDATION", "message": "扩展代码包不能为空"}
try:
ext = ext_mod.get_store().create_extension(
display_name=display_name, command=command, runtime_image=runtime_image or "",
code_files=code, data_files=_file_items(data_files),
)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _extension_status(ext_id: str) -> dict[str, Any]:
try:
ext = ext_mod.get_store().load_extension(ext_id)
except ext_mod.ExtensionNotFoundError:
return {"ok": False, "code": "NOT_FOUND", "message": "扩展不存在"}
return {"ok": True, "data": _ext_view(ext) if ext else None}
async def _build_extension(ext_id: str) -> dict[str, Any]:
try:
ext = await ext_mod.get_store().build_extension(ext_id)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _trial_run_extension(ext_id: str) -> dict[str, Any]:
try:
ext = await ext_mod.get_store().trial_run(ext_id)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _validate_extension(ext_id: str) -> dict[str, Any]:
try:
ext = ext_mod.get_store().validate(ext_id)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _confirm_extension(ext_id: str, nl_triggers: Any, parameters: Any,
runtime_image: str) -> dict[str, Any]:
triggers = [str(x).strip() for x in nl_triggers if str(x).strip()] if isinstance(nl_triggers, list) else None
params = parameters if isinstance(parameters, list) else None
try:
ext = ext_mod.get_store().confirm(
ext_id, nl_triggers=triggers, parameters=params,
runtime_image=runtime_image or None,
)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _publish_extension(ext_id: str) -> dict[str, Any]:
try:
ext = ext_mod.get_store().publish(ext_id)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
async def _run_extension(ext_id: str, params: Any, files: Any) -> dict[str, Any]:
try:
result = await ext_mod.get_store().run(
ext_id,
params=params if isinstance(params, dict) else None,
files=[str(f) for f in files] if isinstance(files, list) else None,
)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": result}
async def _repair_extension(ext_id: str) -> dict[str, Any]:
try:
ext = await ext_mod.get_store().repair(ext_id)
except ext_mod.ExtensionError as exc:
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
return {"ok": True, "data": _ext_view(ext)}
# ---- 文件登记(复刻 biojobs._register_path 核心,无 FastAPI 依赖)----
def _blocked_reason(path: Path, settings: Any) -> str | None:
s = str(path)
for prefix in settings.sensitive_path_prefixes:
if s == prefix or s.startswith(prefix + "/"):
return f"敏感路径前缀 {prefix}"
for pat in settings.sensitive_filename_patterns:
if re.match(pat, path.name):
return f"敏感文件名 {path.name}"
return None
async def _register_job_files(job_id: str, paths: list[str]) -> dict[str, Any]:
"""登记服务器/局域网路径(单文件或目录展开)到任务 input。"""
ctx = _ctx()
settings = get_settings()
record = await load_record(job_id)
if record is None:
return _mcp_err("NOT_FOUND", "任务不存在")
input_dir = record.input_dir
input_dir.mkdir(parents=True, exist_ok=True)
items: list[dict[str, Any]] = []
registered: list[str] = []
for raw in paths:
p = Path(raw).expanduser().resolve()
if not p.exists():
return _mcp_err("PATH_NOT_EXIST", f"路径不存在: {raw}")
blocked = _blocked_reason(p, settings)
if blocked:
return _mcp_err("PATH_NOT_ALLOWED", f"路径不允许: {raw}({blocked})")
files: list[Path] = []
if p.is_file():
files = [p]
else:
ext_set = settings.upload_ext_set
for f in sorted(p.rglob("*")):
if f.is_file() and (not ext_set or f.suffix.lower() in ext_set) and not _blocked_reason(f, settings):
files.append(f)
if not files:
return _mcp_err("VALIDATION", f"目录内无可登记文件: {raw}")
for f in files:
link = input_dir / f.name
try:
if not link.exists():
link.symlink_to(f)
except OSError as exc: # noqa: BLE001
logger.warning("mcp register symlink 失败: %s -> %s (%s)", f, link, exc)
items.append({"file_id": f"{record.id}:{f.name}", "name": f.name,
"size": f.stat().st_size, "path": str(f), "source": "server_path"})
registered.append(str(f))
# rnaseq 系:samples.yaml 生成(描述文件优先 + 实况分析;规则不可归纳走 LLM 兜底)
try:
source_roots = [str(Path(x).parent) for x in paths if Path(x).exists()]
extra_mounts = ensure_samples_yaml(input_dir, source_roots=[Path(r) for r in source_roots])
if not extra_mounts:
try:
await llm_sample_fallback(input_dir, [Path(r) for r in source_roots])
except Exception as exc: # noqa: BLE001
logger.warning("samples.yaml LLM 兜底失败: %s", exc)
if extra_mounts:
record.registered_server_paths = list(
dict.fromkeys((getattr(record, "registered_server_paths", None) or [])
+ [str(Path(m)) for m in extra_mounts]))
except Exception as exc: # noqa: BLE001
logger.warning("samples.yaml 分析失败: %s", exc)
if registered:
import json
record.registered_server_paths = list(
dict.fromkeys((getattr(record, "registered_server_paths", None) or []) + registered)
)
(record.job_dir / "server_registered.json").write_text(
json.dumps({"registered_server_paths": record.registered_server_paths},
ensure_ascii=False), encoding="utf-8")
# 自动续跑(await_inputs 任务重推;非等待态 no-op)
await ctx.runner.retry_after_files(record)
return {"ok": True, "data": {"items": items, "total": len(items)}}
async def _ask_question(question: str) -> dict[str, Any]:
ctx = _ctx()
hits: list[dict[str, Any]] = []
if ctx.flow_store is not None:
try:
hits = ctx.flow_store.search(question, top_k=8).get("items") or []
except Exception: # noqa: BLE001
hits = []
citations = []
for h in hits:
citations.append({
"source": h.get("source") or h.get("doc_kind") or "",
"title_path": h.get("title_path") or "",
"path": h.get("path") or "",
"text": str(h.get("text") or "")[:1200],
})
return {"ok": True, "citations": citations, "degraded": not citations}
async def _list_archives(page: int, page_size: int) -> dict[str, Any]:
ctx = _ctx()
items, total = await store.list_archives(ctx.db, page=page, page_size=page_size)
return {"ok": True, "items": items, "total": total, "page": page, "page_size": page_size}
async def _reuse_archive(name: str) -> dict[str, Any]:
"""复用存档(薄适配):委托 cmd_archive_reuse(读各阶段实际生效参数)。"""
return await cmd_archive_reuse(name, db=_ctx().db)
async def _fetch_source_file(job_id: str | None, source: str,
credentials: dict[str, Any] | None) -> dict[str, Any]:
"""拉取源文件(薄适配):委托 cmd_source_fetch,仅保留内存 record 校验。"""
if job_id:
record = await load_record(job_id)
if record is None:
return _mcp_err("NOT_FOUND", "任务不存在")
return await cmd_source_fetch(source, job_id=job_id, credentials=credentials,
fetch_fn=fetch_source)
async def _inspect_file(path: str) -> dict[str, Any]:
"""探测文件(薄适配):委托 cmd_file_inspect。"""
return cmd_file_inspect(path)
async def _transform_file(src_path: str, target_format: str,
column_map: dict[str, str] | None,
dest_path: str | None) -> dict[str, Any]:
"""转换文件(薄适配):委托 cmd_file_transform(注入 LLM 适配器)。"""
from .services.bioflow.llm import complete_json
async def llm_adapter(user: str, system: str) -> Any:
return await complete_json(system, user, temperature=0.2, max_tokens=2000)
return await cmd_file_transform(
src_path, target_format, column_map=column_map, dest=dest_path, llm=llm_adapter,
)
async def _generate_samples_yaml(job_id: str, sample_description: str) -> dict[str, Any]:
import yaml
from .services.bioflow.llm import complete_json
record = await load_record(job_id)
if record is None:
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
system = (
"你是 RNA-seq 样本配置助手。根据用户的自然语言描述,输出 samples.yaml 结构 JSON:"
'{"samples": {"样本名": {"r1": "fastq路径", "r2": "fastq路径", "condition": "...", ...}}}。'
"只输出 JSON。"
)
try:
raw = await complete_json(system, sample_description, temperature=0.2, max_tokens=2000)
samples = raw.get("samples") if isinstance(raw, dict) else None
if not isinstance(samples, dict) or not samples:
return {"ok": False, "code": "EMPTY_SAMPLES", "message": "未能从描述解析出样本"}
except Exception as exc: # noqa: BLE001
return {"ok": False, "code": "SAMPLES_GEN_FAILED", "message": f"样本配置生成失败: {exc}"}
record.input_dir.mkdir(parents=True, exist_ok=True)
out = record.input_dir / "samples.yaml"
out.write_text(yaml.safe_dump({"samples": samples}, allow_unicode=True), encoding="utf-8")
v = validate_format(out, "samples.yaml")
if not v.get("ok"):
out.unlink(missing_ok=True)
return {"ok": False, "code": "VALIDATION_FAILED", "message": f"样本配置校验失败: {v.get('reason')}"}
return {"ok": True, "data": {"path": str(out), "samples": samples, "validated": v}}
# ---------------------------------------------------------------------------
# 能力单元工具(命名 async,带类型提示 + 中文描述)
# ---------------------------------------------------------------------------
async def plan_gene_family(message: str) -> dict[str, Any]:
"""根据用户自然语言需求规划基因家族分析候选路径(仅基因家族链阶段)。"""
return await _plan(message, GENE_FAMILY_STAGES)
async def create_gene_family_job(session_id: str, stages: list[str], title: str = "",
params: dict[str, Any] | None = None,
auto_continue: bool = False,
archive_id: str | None = None) -> dict[str, Any]:
"""按确认的阶段链创建基因家族任务(stages 只能是基因家族链阶段,跨链拒绝)。"""
return await _create_job(session_id, title, stages, params, auto_continue, archive_id, GENE_FAMILY_STAGES)
async def ask_gene_family_question(question: str) -> dict[str, Any]:
"""回答基因家族流程/参数/生物学意义类问题(流程知识库检索,带引用)。"""
return await _ask_question(question)
async def plan_rnaseq(message: str) -> dict[str, Any]:
"""根据用户自然语言需求规划转录组分析候选路径(仅转录组链阶段)。"""
return await _plan(message, RNASEQ_STAGES)
async def create_rnaseq_job(session_id: str, stages: list[str], title: str = "",
params: dict[str, Any] | None = None,
auto_continue: bool = False,
archive_id: str | None = None) -> dict[str, Any]:
"""按确认的阶段链创建转录组任务(stages 只能是转录组链阶段,跨链拒绝)。"""
return await _create_job(session_id, title, stages, params, auto_continue, archive_id, RNASEQ_STAGES)
async def generate_samples_yaml(job_id: str, sample_description: str) -> dict[str, Any]:
"""把用户的自然语言样本描述(样本名/对照处理组/重复/单双端等)解析为 samples.yaml 写入任务 input。"""
return await _generate_samples_yaml(job_id, sample_description)
async def ask_rnaseq_question(question: str) -> dict[str, Any]:
"""回答转录组流程/参数/实验设计类问题(流程知识库检索,带引用)。"""
return await _ask_question(question)
async def job_status(job_id: str) -> dict[str, Any]:
"""查询生信任务状态:任务/各阶段状态 + 下一步待办动作(pending_actions)。卡住或需确认时用本工具查真相。"""
return await _job_status(job_id)
async def confirm_stage(job_id: str, stage: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
"""确认某阶段参数并执行。仅 waiting_confirm 状态可确认;参数名须在白名单内(未知参数被拒)。人工确认边界阶段(subfamily_confirm/rnaseq_de)不可代用户确认。"""
return await _confirm_stage(job_id, stage, params)
async def register_job_files(job_id: str, paths: list[str]) -> dict[str, Any]:
"""把服务器/局域网路径(单文件或目录)登记为任务输入文件。目录按扩展名白名单展开。"""
return await _register_job_files(job_id, paths)
async def rerun_stage(job_id: str, stage: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
"""重跑失败阶段(带参数则先等确认)。仅 failed 状态可重跑。"""
return await _rerun_stage(job_id, stage, params)
async def skip_stage(job_id: str, stage: str) -> dict[str, Any]:
"""跳过失败阶段并推进后续。仅 failed 状态可跳过。"""
return await _skip_stage(job_id, stage)
async def pause_job(job_id: str) -> dict[str, Any]:
"""暂停运行中的任务(running 阶段会立即 kill 容器)。"""
return await _pause_job(job_id)
async def resume_job(job_id: str) -> dict[str, Any]:
"""恢复已暂停的任务(从下一阶段继续,确认点不跳过)。"""
return await _resume_job(job_id)
async def list_archives(page: int = 1, page_size: int = 20) -> dict[str, Any]:
"""列出历史存档任务(供复用参考)。"""
return await _list_archives(page, page_size)
async def reuse_archive(name: str) -> dict[str, Any]:
"""按名称匹配存档,返回其阶段链+参数(供复用发起)。"""
return await _reuse_archive(name)
async def archive_job(job_id: str, name: str, description: str = "",
include_intermediate: bool = False) -> dict[str, Any]:
"""把已完成任务存档(打包结果/日志/事件 + 落档登记);重名拒绝(ARCHIVE_EXISTS)。"""
return await _archive_job(job_id, name, description, include_intermediate)
# ---------------------------------------------------------------------------
# 扩展能力单元工具(第三个能力单元 bioflow-extension)
# ---------------------------------------------------------------------------
async def create_extension(display_name: str, command: str, runtime_image: str = "",
code_files: list[dict[str, Any]] | None = None,
data_files: list[dict[str, Any]] | None = None) -> dict[str, Any]:
"""创建扩展草稿(draft):上传代码文件清单(每项 {path, content 文本}) + 运行命令 + 显示名。"""
return await _create_extension(display_name, command, runtime_image, code_files, data_files)
async def extension_status(ext_id: str) -> dict[str, Any]:
"""查询扩展详情与状态(draft/build/trial/validated/confirming/published)。"""
return await _extension_status(ext_id)
async def build_extension(ext_id: str) -> dict[str, Any]:
"""AI 转换扩展(原始命令→command_template+参数+dependencies);LLM 失败自动回退规则版。"""
return await _build_extension(ext_id)
async def trial_run_extension(ext_id: str) -> dict[str, Any]:
"""试运行扩展(docker 执行;未接入执行器时返回 EXT_RUN 503 语义)。"""
return await _trial_run_extension(ext_id)
async def validate_extension(ext_id: str) -> dict[str, Any]:
"""校验扩展(金标/试运行校验)。仅 built/trial_passed 可校验。"""
return await _validate_extension(ext_id)
async def confirm_extension(ext_id: str, nl_triggers: list[str] | None = None,
parameters: list[dict[str, Any]] | None = None,
runtime_image: str = "") -> dict[str, Any]:
"""确认扩展参数/触发词(用户可编辑 nl_triggers);仅 validated/trial_passed 可确认。"""
return await _confirm_extension(ext_id, nl_triggers, parameters, runtime_image)
async def publish_extension(ext_id: str) -> dict[str, Any]:
"""发布扩展(仅 published 状态可被自然语言触发运行)。"""
return await _publish_extension(ext_id)
async def run_extension(ext_id: str, params: dict[str, Any] | None = None,
files: list[str] | None = None) -> dict[str, Any]:
"""正式运行已发布扩展(同步返回 run_result)。"""
return await _run_extension(ext_id, params, files)
async def repair_extension(ext_id: str) -> dict[str, Any]:
"""AI 修复扩展(试运行失败上下文→LLM 修复 command_template/依赖;LLM 失败回退规则版)。"""
return await _repair_extension(ext_id)
async def fetch_source_file(source: str, job_id: str | None = None,
credentials: dict[str, Any] | None = None) -> dict[str, Any]:
"""从 web(http/https 流式)、OSS(oss:// 或预签名 URL)或局域网路径拉取源文件到任务 input;下载前检查磁盘余量并记录 provenance。"""
return await _fetch_source_file(job_id, source, credentials)
async def inspect_file(path: str) -> dict[str, Any]:
"""探测文件格式(扩展名仅参考):返回预览/列结构/大小/格式猜测/乱码检测,供判断是否需要预处理。"""
return await _inspect_file(path)
async def transform_file(src_path: str, target_format: str,
column_map: dict[str, str] | None = None,
dest_path: str | None = None) -> dict[str, Any]:
"""把非标准文件(xlsx/csv/tsv/txt)转换为 fasta/gff3/tsv/samples.yaml;LLM 定列映射规则+脚本执行,结果过格式验证器。"""
return await _transform_file(src_path, target_format, column_map, dest_path)
# ---------------------------------------------------------------------------
# G12:11 个分析控制 MCP 工具(异步薄封装委托 skill_tools,不写业务逻辑)
# ---------------------------------------------------------------------------
async def get_task_context(job_id: str | None = None) -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.get_task_context(job_id)
async def inspect_asset(role: str = "pep.fa", path: str = "") -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.inspect_asset(role, path)
async def validate_plan(stages: list[str] | None = None) -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.validate_plan(stages)
async def check_readiness(assets_root: str = "", stages: list[str] | None = None) -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.check_readiness(assets_root, stages)
async def get_run_evidence(run_id: str = "") -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.get_run_evidence(run_id)
async def propose_change(job_id: str = "", stage: str = "",
params: dict[str, Any] | None = None) -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.propose_change(job_id, stage, params)
async def apply_change(action_id: str = "", change: dict[str, Any] | None = None) -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.apply_change(action_id, change)
async def execute_nodes(job_id: str = "", nodes: list[str] | None = None) -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.execute_nodes(job_id, nodes)
async def control_execution(run_id: str = "", command: str = "") -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.control_execution(run_id, command)
async def validate_delivery(manifest: dict[str, Any] | None = None) -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.validate_delivery(manifest)
async def publish_delivery(manifest: dict[str, Any] | None = None) -> dict[str, Any]:
from .services.bioflow import skill_tools
return skill_tools.publish_delivery(manifest)
SKILL_CONTROL_TOOLS = [
get_task_context, inspect_asset, validate_plan,
check_readiness, get_run_evidence, propose_change,
apply_change, execute_nodes, control_execution,
validate_delivery, publish_delivery,
]
# ---------------------------------------------------------------------------
# 构建三个 MCPServer(基因家族 / 转录组 / 扩展)
# ---------------------------------------------------------------------------
def build_mcp_servers() -> tuple[Any, Any, Any]:
from mcp.server.mcpserver import MCPServer
gf = MCPServer(
"bioflow-gene-family",
title="PlantAI 基因家族分析",
description="基因家族分析能力单元:路径规划/建任务/确认参数/查状态/文件获取与预处理。",
)
rn = MCPServer(
"bioflow-rnaseq",
title="PlantAI 转录组分析",
description="转录组分析能力单元:路径规划/样本配置生成/建任务/确认参数/查状态/文件获取与预处理。",
)
ext = MCPServer(
"bioflow-extension",
title="PlantAI 用户自定义扩展分析",
description="用户自定义扩展分析能力单元:创建/转换/试运行/校验/确认/发布/运行。",
)
gf.tool(name="plan_gene_family", description="根据用户自然语言需求规划基因家族分析候选路径(仅基因家族链阶段)。返回 2-3 条候选路径供用户确认。")(plan_gene_family)
gf.tool(name="create_gene_family_job", description="按确认的阶段链创建基因家族任务。stages 只能是基因家族链阶段(跨链拒绝)。")(create_gene_family_job)
gf.tool(name="ask_gene_family_question", description="回答基因家族流程/参数/生物学意义类问题(流程知识库检索,带引用)。")(ask_gene_family_question)
rn.tool(name="plan_rnaseq", description="根据用户自然语言需求规划转录组分析候选路径(仅转录组链阶段)。返回候选路径供用户确认。")(plan_rnaseq)
rn.tool(name="create_rnaseq_job", description="按确认的阶段链创建转录组任务。stages 只能是转录组链阶段(跨链拒绝)。")(create_rnaseq_job)
rn.tool(name="generate_samples_yaml", description="把用户的自然语言样本描述解析为 samples.yaml 写入任务 input。")(generate_samples_yaml)
rn.tool(name="ask_rnaseq_question", description="回答转录组流程/参数/实验设计类问题(流程知识库检索,带引用)。")(ask_rnaseq_question)
ext.tool(name="create_extension", description="创建扩展草稿(draft):上传代码文件清单(每项 {path, content 文本}) + 运行命令 + 显示名。")(create_extension)
ext.tool(name="extension_status", description="查询扩展详情与状态(draft/build/trial/validated/confirming/published)。")(extension_status)
ext.tool(name="build_extension", description="AI 转换扩展(原始命令→command_template+参数+依赖);LLM 失败自动回退规则版。")(build_extension)
ext.tool(name="trial_run_extension", description="试运行扩展(docker 执行;未接入执行器时返回 EXT_RUN)。")(trial_run_extension)
ext.tool(name="validate_extension", description="校验扩展(金标/试运行校验)。仅 built/trial_passed 可校验。")(validate_extension)
ext.tool(name="confirm_extension", description="确认扩展参数/触发词(用户可编辑 nl_triggers);仅 validated/trial_passed 可确认。")(confirm_extension)
ext.tool(name="publish_extension", description="发布扩展(仅 published 状态可被自然语言触发运行)。")(publish_extension)
ext.tool(name="run_extension", description="正式运行已发布扩展(同步返回 run_result)。")(run_extension)
ext.tool(name="repair_extension", description="AI 修复扩展(试运行失败上下文→LLM 修复;LLM 失败回退规则版)。")(repair_extension)
shared = [
job_status, confirm_stage, register_job_files, rerun_stage, skip_stage,
pause_job, resume_job, list_archives, reuse_archive, archive_job,
fetch_source_file, inspect_file, transform_file,
]
for mcp in (gf, rn):
for fn in shared:
mcp.tool()(fn)
# G12:11 个分析控制工具(Skill 编排,异步薄封装委托 skill_tools)
for fn in SKILL_CONTROL_TOOLS:
mcp.tool()(fn)
return gf, rn, ext
gene_family_mcp, rnaseq_mcp, extension_mcp = build_mcp_servers()
+19 -4
View File
@@ -31,10 +31,16 @@ from ..schemas import ArchiveCreateRequest, ArchiveRunRequest
from ..services.bioflow import store
from ..services.bioflow.archive import (
archive_from_job,
effective_stage_params,
normalize_archive_name,
remove_archive_dir,
)
from ..services.bioflow.reports import extract_figure_refs, locate_report, report_payload
from ..services.bioflow.reports import (
extract_figure_refs,
locate_report,
report_payload,
validate_report_path,
)
from .bioflow import job_or_404
logger = logging.getLogger(__name__)
@@ -65,6 +71,10 @@ async def reports_register(payload: dict[str, Any]) -> dict[str, Any]:
job_id = str(payload.get("job_id") or "")
record = await load_record(job_id) if job_id else None
report_path = payload.get("md_path") or payload.get("report_path") or ""
# G3:拒绝空路径/空文件(不静默登记空报告)
ok, err = validate_report_path(report_path)
if not ok:
return _err("EMPTY_REPORT", err, 422)
figures: list[str] = list(payload.get("figures") or [])
if record is not None and not figures:
located = locate_report(record.workspace_dir)
@@ -96,12 +106,14 @@ async def create_archive(body: ArchiveCreateRequest) -> dict[str, Any]:
existing = await store.find_archive_by_name(ctx.db, name)
if existing is not None:
return _err("ARCHIVE_EXISTS", f"归档名已存在: {name}", 409)
# G6:保存各阶段实际生效参数(非创建时 record.params)
effective_params = effective_stage_params(record.stage_chain, record._stage_states)
report_path = locate_report(record.workspace_dir)
result = await archive_from_job(
{
"id": record.id, "session_id": record.session_id,
"title": record.title, "stage_chain": record.stage_chain,
"params": record.params,
"params": effective_params,
},
record.job_dir,
get_settings().archives_dir_abs,
@@ -116,7 +128,7 @@ async def create_archive(body: ArchiveCreateRequest) -> dict[str, Any]:
"name": name,
"description": body.description,
"stage_chain": record.stage_chain,
"params": record.params,
"params": effective_params,
"report_artifact": result.report_artifact,
"disk_path": result.disk_path,
"size_bytes": result.size_bytes,
@@ -166,7 +178,10 @@ async def delete_archive(archive_id: str) -> JSONResponse:
@router.post("/archives/{archive_id}/run", status_code=201)
async def archive_run(archive_id: str, body: ArchiveRunRequest) -> dict[str, Any]:
"""复用发起(FR-BIO-023):装载阶段链+参数 → 新建任务(表单预填存档值)。"""
"""复用发起(FR-BIO-023):装载阶段链+参数 → 新建任务(表单预填存档值)。
params 为存档时落库的各阶段实际生效参数(G6:非创建时 record.params)。
"""
ctx: Context = context_mod.context
archive = await store.get_archive(ctx.db, archive_id)
if archive is None:
+36 -6
View File
@@ -22,7 +22,9 @@ from __future__ import annotations
import asyncio
import json
import logging
import os
import re
import uuid
from pathlib import Path
from typing import Any
@@ -212,23 +214,43 @@ async def upload_file(
dest = input_dir / _safe_name(file.filename)
written = 0
max_bytes = int(settings.max_upload_gb * 1024 ** 3)
# G2:先落任务临时目录,再原子 rename 发布;同名冲突显式报错,不静默覆盖。
tmp_dir = record.job_dir / ".tmp"
tmp_dir.mkdir(parents=True, exist_ok=True)
tmp_path = tmp_dir / f"{uuid.uuid4().hex[:8]}_{dest.name}"
try:
with dest.open("wb") as fh:
# G5:逐块流式写盘(不整块进内存)
with tmp_path.open("wb") as fh:
while True:
chunk = await file.read(1024 * 1024)
if not chunk:
break
written += len(chunk)
if written > max_bytes:
fh.close()
dest.unlink(missing_ok=True)
return _err("UPLOAD_TOO_LARGE",
f"超过上传上限 {settings.max_upload_gb}GB(已收 {written} 字节)", 413)
raise _UploadTooLarge(written)
fh.write(chunk)
except _UploadTooLarge as exc:
tmp_path.unlink(missing_ok=True)
return _err("UPLOAD_TOO_LARGE",
f"超过上传上限 {settings.max_upload_gb}GB(已收 {exc.written} 字节)", 413)
except Exception as exc: # noqa: BLE001 - 写盘失败不登记(TC-BIO-API-050)
logger.exception("files: 上传写盘失败 %s", job_id)
dest.unlink(missing_ok=True)
tmp_path.unlink(missing_ok=True)
return _err("INTERNAL", f"上传写盘失败: {exc}", 500)
# 原子发布 + 同名冲突检测(G2)
try:
os.link(tmp_path, dest)
except FileExistsError:
tmp_path.unlink(missing_ok=True)
return _err("FILE_EXISTS", f"同名文件已存在,拒绝覆盖: {dest.name}", 409)
except OSError:
# os.link 不支持的文件系统 → 退化:exists 检查 + rename(同事件循环内无 await 交错)
if dest.exists() or dest.is_symlink():
tmp_path.unlink(missing_ok=True)
return _err("FILE_EXISTS", f"同名文件已存在,拒绝覆盖: {dest.name}", 409)
os.rename(tmp_path, dest)
else:
tmp_path.unlink(missing_ok=True)
file_id = f"{job_id}:{dest.name}"
# BUG-P7-01:文件落盘后自动续跑(await_inputs 任务重推;非等待态 no-op)
await ctx.runner.retry_after_files(record)
@@ -238,6 +260,14 @@ async def upload_file(
}}
class _UploadTooLarge(Exception):
"""上传超限信号(携带已收字节数)。"""
def __init__(self, written: int) -> None:
super().__init__("upload too large")
self.written = written
async def _register_path(record: Any, server_path: str, settings: Any) -> dict[str, Any]:
"""服务器路径登记(FR-BIO-007):单文件或目录展开,任意数据位置(敏感路径黑名单拦截)。
@@ -19,11 +19,27 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Any
from .manifests import ManifestError, artifact_paths, parse_manifest
logger = logging.getLogger(__name__)
INTERMEDIATE_DIR_NAMES = ("analysis",)
def effective_stage_params(
stage_chain: list[str], stage_states: dict[str, Any]
) -> dict[str, Any]:
"""G6:各阶段「实际生效」参数(confirm/rerun 更新后的 stage 参数)。
区别于创建时 record.params(创建后 confirm/rerun 只更新 _stage_states[stage].params,
不回流 record.params)。存档与复用以此为单一事实源,避免参数失真。
"""
return {
s: dict((stage_states.get(s) or {}).get("params") or {})
for s in stage_chain
}
@dataclass
class ArchiveResult:
id: str
@@ -88,6 +104,11 @@ async def archive_from_job(
# 报告:任务目录内报告 HTML(final 内 *.html / report.html)随 final 已复制;
# report_artifact 为 doc_artifacts 登记 id(由 reports 模块/BFF 登记后回传)。
# G1:按 manifest 必需产物清单补归档(不只 workspace/final/)。
# 相对路径以 workspace 根解析并改写为归档内路径(去 workspace 前缀)。
_copy_manifest_artifacts(job, job_dir, dest)
size_bytes = _dir_size(dest)
return ArchiveResult(
id=archive_id,
@@ -98,6 +119,42 @@ async def archive_from_job(
)
def _copy_manifest_artifacts(
job: dict[str, Any], job_dir: Path, dest: Path
) -> list[str]:
"""G1:按 manifest 的必需产物清单复制归档产物(漏归档修复)。
每个阶段的 manifest artifacts(full_path/path/relpath_or_hint)逐条复制;
相对路径以 workspace 根解析,并改写为归档内路径(去 workspace 前缀)。
跳过不存在/失败阶段无 manifest 的产物。返回已复制的归档内相对路径列表。
"""
stage_chain = job.get("stage_chain") or []
workspace = Path(job_dir) / "workspace"
copied: list[str] = []
for stage in stage_chain:
try:
manifest = parse_manifest(workspace, stage)
except ManifestError:
continue
for raw in artifact_paths(manifest.artifacts):
p = Path(raw)
if not p.is_absolute():
p = workspace / p
if not p.is_file():
continue
# 相对路径改写为归档内路径(去 workspace 前缀)
try:
rel = p.resolve().relative_to(workspace.resolve())
except ValueError:
rel = Path(p.name)
target = dest / rel
target.parent.mkdir(parents=True, exist_ok=True)
if not target.exists():
shutil.copy2(p, target)
copied.append(str(rel))
return copied
def _copytree_into(src: Path, dest: Path) -> None:
"""复制目录内容到 dest(dest 可能已存在,逐项合并)。"""
dest.mkdir(parents=True, exist_ok=True)
+63 -21
View File
@@ -551,6 +551,38 @@ _STAGE_PARAM_SCHEMAS: dict[str, tuple[list[dict[str, Any]], list[dict[str, Any]]
], _DOWNSTREAM_INPUTS, _DOWNSTREAM_OUTPUTS),
}
# ---- G9:单一参数 schema 访问点(Catalog/表单/预检/工具/验收都读这一份)----
def _schema_params_inputs_outputs(stage: str) -> tuple[list[dict[str, Any]], list[dict[str, Any]], list[dict[str, Any]]]:
"""阶段参数/输入/输出的单一读取点(含未知阶段默认兜底)。"""
if stage in _STAGE_PARAM_SCHEMAS:
return _STAGE_PARAM_SCHEMAS[stage]
params = [{"name": "cpu", "type": "choice", "choices": ["4", "8", "16", "32"],
"default": "4", "source": "runtime", "help": "容器 CPU 数"}]
inputs = _DOWNSTREAM_INPUTS if stage not in ("identify", "genome_prepare") else []
outputs = _DOWNSTREAM_OUTPUTS
return params, inputs, outputs
def stage_schema(stage: str) -> dict[str, Any]:
"""G9:单一参数 schema 访问点——阶段名/耗时/依赖/参数/输入/输出/输入角色 合并为一份。
Catalog/表单/预检/工具/验收都读这一份(唯一事实源)。
"""
from .stage_assets import stage_input_roles
params, inputs, outputs = _schema_params_inputs_outputs(stage)
return {
"name": STAGE_INFO_SNAPSHOT.get(stage, {}).get("name") or stage,
"estimated_time": int(STAGE_INFO_SNAPSHOT.get(stage, {}).get("estimated_time") or 0),
"dependencies": STAGE_DEPENDENCIES_SNAPSHOT.get(stage, []),
"params": params,
"inputs": inputs,
"outputs": outputs,
"input_roles": stage_input_roles(stage),
}
class StageCatalog:
"""阶段目录(数据来自 stage_image_map.json + 手写快照,不运行时反射)。"""
@@ -578,29 +610,19 @@ class StageCatalog:
if not self.has(stage):
return None
entry = self._map.image_for(stage) or {}
params, inputs, outputs = _STAGE_PARAM_SCHEMAS.get(
stage, ([], _DOWNSTREAM_INPUTS if stage not in ("identify", "genome_prepare") else [], [])
)
if stage not in _STAGE_PARAM_SCHEMAS:
params, inputs, outputs = (
[{"name": "cpu", "type": "choice", "choices": ["4", "8", "16", "32"],
"default": "4", "source": "runtime", "help": "容器 CPU 数"}],
_DOWNSTREAM_INPUTS if stage not in ("identify", "genome_prepare") else [],
_DOWNSTREAM_OUTPUTS,
)
info = STAGE_INFO_SNAPSHOT.get(stage, {})
schema = stage_schema(stage)
return StageDef(
name=stage,
description=str(info.get("name") or stage),
description=str(schema["name"] or stage),
image=str(entry.get("image") or ""),
module=str(entry.get("module") or ""),
executable=bool(entry.get("executable")),
estimated_time=int(info.get("estimated_time") or 0),
params=params,
inputs=inputs,
outputs=outputs,
estimated_time=int(schema["estimated_time"]),
params=schema["params"],
inputs=schema["inputs"],
outputs=schema["outputs"],
note=str(entry.get("note") or ""),
dependencies=STAGE_DEPENDENCIES_SNAPSHOT.get(stage, []),
dependencies=schema["dependencies"],
)
def resolve_chain(self, chain: str | None) -> list[str] | None:
@@ -612,32 +634,52 @@ class StageCatalog:
return None
return list(alias)
def auto_fill_prerequisites(self, stages: list[str]) -> tuple[list[str], list[str]]:
def auto_fill_prerequisites(
self, stages: list[str], satisfied_roles: list[str] | None = None
) -> tuple[list[str], list[str]]:
"""前置补全(FR-BIO-003):缺失的前置阶段自动补全。
G7/G8:satisfied_roles(已有 validated 资产的输入角色)对应的上游阶段
跳过不补跑(如 role=counts 满足时做 DE 不补跑 qc/align/quant/sample_qc)。
返回 (补全后的链, 被补全的阶段名列表)。
链内阶段按依赖拓扑顺序排列(依赖在前)。
"""
from .stage_assets import upstream_stages_for_roles
skip_stages = upstream_stages_for_roles(satisfied_roles or [])
missing: list[str] = []
added: list[str] = []
chain_set = set(stages)
# 反复补全直到闭合(依赖的依赖也补)
# 反复补全直到闭合(依赖的依赖也补;被角色资产满足的上游跳过)
changed = True
while changed:
changed = False
for s in list(stages):
for dep in STAGE_DEPENDENCIES_SNAPSHOT.get(s, []):
if dep not in chain_set and self.has(dep):
if dep in chain_set:
continue
if dep in skip_stages:
continue
if self.has(dep):
chain_set.add(dep)
missing.append(dep)
added.append(dep)
changed = True
ordered = topo_order([s for s in GENE_FAMILY_CHAIN if s in chain_set]
or list(chain_set))
# G7/G8:被角色资产满足的上游阶段从最终链中剔除(不补跑)
if skip_stages:
ordered = [s for s in ordered if s not in skip_stages]
return ordered, added
def dependencies(self, stage: str) -> list[str]:
return STAGE_DEPENDENCIES_SNAPSHOT.get(stage, [])
return stage_schema(stage)["dependencies"]
def stage_input_roles(self, stage: str) -> list[str]:
"""G8:阶段依赖的输入角色(数据角色)。"""
from .stage_assets import stage_input_roles
return stage_input_roles(stage)
def topo_order(stages: list[str]) -> list[str]:
@@ -0,0 +1,141 @@
"""app/services/bioflow/delivery.py —— 交付清单(G21)+ 终报告范围(G22)+ 资源预算(G23)。
- G21 交付清单完整性:delivery_validate(manifest) 做八项完整性检查
(selected_stages / run_id / input_sources / output_sources / config /
required_results / validation_values / review_status),返回
{ok, checks{八项:bool}, incomplete[{item, reason, error(G13)}]}。
- G22 终报告范围:classify_report_scope(status, source) 五分类
(user_provided / reused / not_executed / skipped / failed)。
- G23 资源预算:check_download_budget(size_bytes, budget_bytes)(超限→
DOWNLOAD_BUDGET_EXCEEDED)+ check_disk_available(dest_dir, needed_bytes)
(不足→DISK_FULL),均 ok=False + 结构化错误。
服务层纪律:纯 dict/stdlib,无 pydantic。
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from .errors import error_object, error_result
from .source_fetcher import check_disk_space
# G21 八项完整性检查项
DELIVERY_ITEMS = (
"selected_stages", "run_id", "input_sources", "output_sources",
"config", "required_results", "validation_values", "review_status",
)
# G22 五分类
SCOPE_USER_PROVIDED = "user_provided"
SCOPE_REUSED = "reused"
SCOPE_NOT_EXECUTED = "not_executed"
SCOPE_SKIPPED = "skipped"
SCOPE_FAILED = "failed"
def delivery_validate(manifest: dict[str, Any]) -> dict[str, Any]:
"""G21:校验交付清单完整性(八项)。
返回 {"ok": bool, "checks": {item: bool},
"incomplete": [{"item", "reason", "error": {G13 对象}}]}。
"""
manifest = manifest or {}
checks: dict[str, bool] = {}
incomplete: list[dict[str, Any]] = []
for item in DELIVERY_ITEMS:
value = manifest.get(item)
ok = bool(value)
checks[item] = ok
if not ok:
incomplete.append({
"item": item,
"reason": f"缺 {item}",
"error": error_object("INCOMPLETE_DELIVERY", f"交付清单缺项: {item}",
category="input"),
})
return {"ok": not incomplete, "checks": checks, "incomplete": incomplete}
def publish_delivery(manifest: dict[str, Any]) -> dict[str, Any]:
"""G21:发布交付清单(校验通过置 review_status=approved)。"""
check = delivery_validate(manifest)
if not check["ok"]:
return {"ok": False, "published": False, **check}
manifest["review_status"] = "approved"
return {"ok": True, "published": True, "review_status": "approved",
"incomplete": []}
# ---- G22 终报告范围 ----
def classify_report_scope(status: str, source: str | None = None) -> str:
"""G22:阶段状态 → 终报告范围五分类。
failed→failed;skipped→skipped;done + source=reuse→reused;
done + source=user/空→user_provided;其余非终态→not_executed。
"""
s = str(status or "")
if s == "failed":
return SCOPE_FAILED
if s == "skipped":
return SCOPE_SKIPPED
if s == "done":
return SCOPE_REUSED if source == "reuse" else SCOPE_USER_PROVIDED
return SCOPE_NOT_EXECUTED
def build_report(
manifest: dict[str, Any],
stage_sources: dict[str, str] | None = None,
) -> dict[str, Any]:
"""G22:终报告结构(执行范围 / 数据来源与核验 / 方法版本 / QC / 科学前提 /
失败跳过未覆盖 / 产物清单)。"""
stages = list(manifest.get("selected_stages") or [])
sources = dict(stage_sources or {})
scope = [{"stage": s, "source": sources.get(s, SCOPE_USER_PROVIDED)} for s in stages]
failed = [s for s in stages if sources.get(s) == SCOPE_FAILED]
skipped = [s for s in stages if sources.get(s) == SCOPE_SKIPPED]
not_executed = [s for s in stages if sources.get(s) == SCOPE_NOT_EXECUTED]
config = manifest.get("config") or {}
return {
"execution_scope": scope,
"data_sources": {
"inputs": list(manifest.get("input_sources") or []),
"outputs": list(manifest.get("output_sources") or []),
},
"method_version": str(manifest.get("plan_revision") or manifest.get("run_id") or ""),
"qc": {"validation_values": dict(manifest.get("validation_values") or {})},
"scientific_preconditions": config.get("scientific_preconditions", []),
"failed_skipped_uncovered": {
"failed": failed, "skipped": skipped, "not_executed": not_executed,
},
"artifacts": list(manifest.get("required_results") or []),
}
# ---- G23 资源预算门禁 ----
def check_download_budget(size_bytes: int, budget_bytes: int) -> dict[str, Any]:
"""G23:下载预算门禁——超限阻断并返回结构化错误。"""
if size_bytes > budget_bytes:
return error_result(
"DOWNLOAD_BUDGET_EXCEEDED",
f"下载预算超限:本次 {size_bytes} > 预算 {budget_bytes}",
)
return {"ok": True}
def check_disk_available(dest_dir: str | Path, needed_bytes: int) -> dict[str, Any]:
"""G23:磁盘门禁——剩余空间不足阻断并返回结构化错误。"""
try:
ok, free = check_disk_space(dest_dir, needed_bytes)
except OSError as exc:
return error_result("PATH_NOT_EXIST", f"目标目录不可用:{exc}")
if not ok:
return error_result(
"DISK_FULL", f"磁盘空间不足:需 {needed_bytes} 字节,剩余 {free} 字节",
)
return {"ok": True, "free_bytes": free}
@@ -0,0 +1,196 @@
"""app/services/bioflow/errors.py —— 统一结构化错误对象(G13)+ 确定性分类器(G14)。
失败返回统一携带结构化错误对象(替代裸字符串 message):
{"ok": False, "code": "<码>", "message": "<摘要>",
"error": {error_id, category, code, summary, evidence_refs,
diagnosis_status, retryable_without_change, suggested_actions}}
8 类 category(G14:确定性映射):
input / scientific_precondition / reference / environment /
transient / resource / adapter / artifact
G14 分类纪律:退出码/日志是「证据」不是「结论」——
- 日志含 OOM → resource;日志含镜像缺失 → environment;日志含科学前提不满足 → scientific_precondition。
- 仅凭「非零退出码」无法定类 → adapter + undiagnosed(不臆测)。
服务层纪律:纯 dict/stdlib,无 pydantic。
"""
from __future__ import annotations
import uuid
from typing import Any
CATEGORIES = (
"input", "scientific_precondition", "reference", "environment",
"transient", "resource", "adapter", "artifact",
)
# 明显映射:错误码 → (category, diagnosis_status, retryable_without_change, suggested_actions)
_CODE_META: dict[str, tuple[str, str, bool, list[str]]] = {
# ---- resource(资源耗尽)----
"DISK_FULL": ("resource", "diagnosed", False, ["清理磁盘空间后重试"]),
"UPLOAD_TOO_LARGE": ("resource", "diagnosed", False, ["减小文件大小或调大上传上限"]),
"DOWNLOAD_BUDGET_EXCEEDED": ("resource", "diagnosed", False, ["调大下载预算或减小下载量"]),
# ---- environment(运行环境)----
"OSS_UNCONFIGURED": ("environment", "diagnosed", False, ["配置 BIOFLOW_OSS_ENDPOINT 或改用预签名 URL"]),
"SSHPASS_MISSING": ("environment", "diagnosed", False, ["本机安装 sshpass 或改用密钥认证"]),
# ---- reference(参考数据)----
"REFERENCE_NOT_FOUND": ("reference", "diagnosed", False, ["补齐参考基因组/HMM/索引后重试"]),
# ---- transient(临时/可重试)----
"NETWORK_ERROR": ("transient", "diagnosed", True, ["重试下载"]),
"LAN_TIMEOUT": ("transient", "diagnosed", True, ["重试局域网拉取"]),
"TRUNCATED": ("transient", "diagnosed", True, ["重试下载"]),
"LAN_ERROR": ("transient", "diagnosed", True, ["重试局域网拉取"]),
"HTTP_ERROR": ("transient", "diagnosed", True, ["确认 URL 可达或稍后重试"]),
# ---- input(用户输入/文件/参数)----
"EMPTY_SOURCE": ("input", "diagnosed", False, ["提供非空源地址"]),
"BAD_SCHEME": ("input", "diagnosed", False, ["改用 http/https/oss 协议"]),
"BAD_OSS": ("input", "diagnosed", False, ["修正 OSS 地址为 oss://bucket/key"]),
"PATH_NOT_EXIST": ("input", "diagnosed", False, ["确认路径存在且可达"]),
"IS_DIR": ("input", "diagnosed", False, ["改为单文件路径"]),
"NOT_FOUND": ("input", "diagnosed", False, ["确认目标存在"]),
"BAD_FORMAT": ("input", "diagnosed", False, ["选择支持的目标格式 fasta/gff3/tsv/samples.yaml"]),
"EMPTY_TABLE": ("input", "diagnosed", False, ["提供非空表格数据"]),
"MISSING_COLUMNS": ("input", "diagnosed", False, ["确认列映射(名称/序列/r1/r2)"]),
"VALIDATION": ("input", "diagnosed", False, ["修正输入以满足校验"]),
"VALIDATION_FAILED": ("input", "diagnosed", False, ["修正数据后重转"]),
"UNKNOWN_STAGE": ("input", "diagnosed", False, ["选择有效阶段"]),
"UNKNOWN_PARAMS": ("input", "diagnosed", False, ["移除未知参数"]),
"BAD_ROLE": ("input", "diagnosed", False, ["使用合法角色名(非空、无路径分隔符/..)"]),
"EMPTY_REPORT": ("input", "diagnosed", False, ["提供非空报告路径/文件"]),
"EMPTY_SAMPLES": ("input", "diagnosed", False, ["提供含样本的描述"]),
"INCOMPLETE_DELIVERY": ("input", "diagnosed", False, ["补齐交付清单缺失项(selected_stages/run_id/必需结果等)后重试"]),
# ---- adapter(适配/映射)----
"UNSUPPORTED": ("adapter", "diagnosed", False, ["改用支持的转换路径"]),
"CROSS_CHAIN": ("adapter", "diagnosed", False, ["选择本能力单元内的阶段"]),
# ---- artifact(产物)----
"ARCHIVE_EXISTS": ("artifact", "diagnosed", False, ["更换归档名"]),
"ARCHIVE_FAILED": ("artifact", "diagnosed", False, ["检查任务目录完整性后重试"]),
"ARCHIVE_NOT_FOUND": ("artifact", "diagnosed", False, ["确认归档名存在"]),
"ASSET_EXISTS": ("artifact", "diagnosed", False, ["更换角色或文件名"]),
"PUBLISH_FAILED": ("artifact", "diagnosed", False, ["检查目标目录可写性后重试"]),
"NO_MANIFEST": ("artifact", "diagnosed", False, ["检查阶段产物 manifest 是否产出"]),
# ---- scientific_precondition(科学前提)----
"SCIENTIFIC_PRECONDITION": ("scientific_precondition", "diagnosed", False, ["确认科学前提满足后重试"]),
}
# 关键词兜底(summary 命中即映射;用于非枚举码)
_KEYWORD_META: list[tuple[tuple[str, ...], tuple[str, str, bool, list[str]]]] = [
(("out of memory", "oom", "memoryerror", "oom-killed", "memory limit", "内存"),
("resource", "diagnosed", False, ["调大内存或减小数据规模"])),
(("no such image", "image not found", "pull access denied", "镜像", "docker"),
("environment", "diagnosed", False, ["确认镜像存在且可拉取"])),
(("no significant", "not enough", "insufficient", "科学前提", "no differentially", "not found enough"),
("scientific_precondition", "diagnosed", False, ["确认科学前提满足后重试"])),
(("timeout", "超时", "network", "网络", "unreachable", "connecterror", "timed out"),
("transient", "diagnosed", True, ["重试"])),
(("reference", "参考", "hmm", "pfam", "gtf", "index"),
("reference", "diagnosed", False, ["补齐参考数据后重试"])),
(("manifest", "no_manifest", "产物缺失"),
("artifact", "diagnosed", False, ["检查阶段产物 manifest"])),
]
# 阶段失败 reason(runner _fail_stage)→ (category, diagnosis_status, retryable, actions)
_REASON_META: dict[str, tuple[str, str, bool, list[str]]] = {
"timeout": ("transient", "diagnosed", True, ["重跑该阶段"]),
"exit_code": ("adapter", "undiagnosed", False, ["查看阶段日志定位失败原因", "修正后重跑"]),
"no_manifest": ("artifact", "diagnosed", False, ["检查阶段是否产出 manifest"]),
"manifest_error": ("artifact", "diagnosed", False, ["检查 manifest JSON 完整性"]),
"manifest_failed": ("artifact", "diagnosed", False, ["检查阶段产物 manifest status"]),
}
# 退出码 137 = 128 + SIGKILL(常为 OOM kill)
_OOM_EXIT_CODES = {137, 139}
def classify(code: str | None, summary: str = "") -> tuple[str, str, bool, list[str]]:
"""错误码/摘要 → (category, diagnosis_status, retryable_without_change, suggested_actions)。"""
code = str(code or "")
summary = str(summary or "")
if code in _CODE_META:
cat, diag, retry, actions = _CODE_META[code]
return cat, diag, retry, list(actions)
low = summary.lower()
for kws, meta in _KEYWORD_META:
if any(k in low for k in kws):
cat, diag, retry, actions = meta
return cat, diag, retry, list(actions)
return "adapter", "undiagnosed", False, []
def classify_reason(
reason: str | None,
*,
exit_code: int | None = None,
log_tail: str | None = None,
) -> tuple[str, str, bool, list[str]]:
"""阶段失败原因 + 证据(退出码/日志)→ 确定性 8 类。
确定性 reason 优先(timeout / manifest 系列);泛化 exit_code 用证据(退出码/日志)
定类——仅凭非零退出码不定类(undiagnosed),不臆测。
"""
reason = str(reason or "")
log = (log_tail or "").lower()
# 1. 确定性 reason(非 exit_code)优先:timeout / manifest 系列
if reason in _REASON_META and reason != "exit_code":
cat, diag, retry, actions = _REASON_META[reason]
return cat, diag, retry, list(actions)
# 2. 泛化 exit_code / 未知 reason:退出码是证据(137/139 = OOM kill)
if exit_code in _OOM_EXIT_CODES:
return "resource", "diagnosed", False, ["容器可能被 OOM kill,调大内存后重跑"]
# 3. 日志证据定类
for kws, meta in _KEYWORD_META:
if any(k in log for k in kws):
cat, diag, retry, actions = meta
return cat, diag, retry, list(actions)
# 4. 兜底:exit_code → adapter undiagnosed;未知 → 同
if reason in _REASON_META:
cat, diag, retry, actions = _REASON_META[reason]
return cat, diag, retry, list(actions)
return "adapter", "undiagnosed", False, []
def error_object(
code: str | None,
summary: str = "",
*,
category: str | None = None,
evidence_refs: list[str] | None = None,
diagnosis_status: str | None = None,
retryable_without_change: bool | None = None,
suggested_actions: list[str] | None = None,
) -> dict[str, Any]:
"""构造统一结构化错误对象(8 字段)。"""
cat, diag, retry, actions = classify(code, summary)
return {
"error_id": uuid.uuid4().hex[:16],
"category": category or cat,
"code": code or "INTERNAL",
"summary": summary,
"evidence_refs": list(evidence_refs or []),
"diagnosis_status": diagnosis_status or diag,
"retryable_without_change": retry if retryable_without_change is None else retryable_without_change,
"suggested_actions": list(suggested_actions or actions),
}
def error_result(
code: str | None,
summary: str = "",
*,
evidence_refs: list[str] | None = None,
) -> dict[str, Any]:
"""失败返回统一信封:{"ok": False, "code", "message", "error"}。
code 同时出现在顶层与 error 对象内(兼容 Phase A 的 r["code"] 断言)。
"""
return {
"ok": False,
"code": code or "INTERNAL",
"message": summary,
"error": error_object(code, summary, evidence_refs=evidence_refs),
}
@@ -24,8 +24,8 @@ logger = logging.getLogger(__name__)
EVENT_TYPES = (
"job_created", "plan_ready", "stage_started", "stage_log", "stage_progress",
"stage_done", "stage_failed", "stage_skipped", "stage_await_confirm",
"stage_await_files",
"stage_done", "stage_failed", "stage_skipped", "stage_partial",
"stage_await_confirm", "stage_await_files",
"job_done", "job_paused", "job_archived", "job_recovered",
# 用户自定义扩展分析事件帧(契约 §4;并行链路,不触碰现有 job_* 语义)
"extension_created", "extension_build", "extension_trial", "extension_repair",
@@ -208,6 +208,9 @@ class EventBus:
elif t == "stage_skipped":
st = stage_states.setdefault(str(d.get("stage")), {"stage": str(d.get("stage"))})
st["status"] = "skipped"
elif t == "stage_partial":
st = stage_states.setdefault(str(d.get("stage")), {"stage": str(d.get("stage"))})
st["status"] = "partial_success"
elif t == "stage_progress":
st = stage_states.setdefault(str(d.get("stage")), {"stage": str(d.get("stage"))})
pct = d.get("percent")
@@ -0,0 +1,438 @@
"""app/services/bioflow/file_transform.py —— 非标准文件预处理(探测 + 转换 + 校验)。
FR-BIO-014 文件医生闭环:
- inspect_file:内容嗅探(扩展名仅参考),返回结构化摘要(预览行/列结构/大小/
格式猜测/乱码检测/与目标阶段期望格式的差异)。
- transform_file:LLM 抽取列映射/清洗规则 + 规则脚本执行,把 xlsx/csv/tsv/txt 转换
为 fasta/gff3/tsv/samples.yaml;转换结果过格式验证器硬校验,失败返回原因供追问
循环(≤3 次由上层 QA 控制)。
服务层纪律:纯 dataclass/dict/stdlib + yaml/openpyxl,无 pydantic。
LLM 可注入(默认走 .llm.complete_json),测试注入 fake。
"""
from __future__ import annotations
import csv
import io
import json
import logging
import re
from pathlib import Path
from typing import Any, Awaitable, Callable
logger = logging.getLogger(__name__)
PREVIEW_LINES = 10
PREVIEW_BYTES = 4000
# 支持的目标格式
TARGET_FORMATS = ("fasta", "gff3", "tsv", "samples.yaml")
# 乱码检测:非 ASCII 可打印且非 CJK 的异常字节比例阈值
GARBLED_RATIO = 0.30
class TransformError(Exception):
def __init__(self, code: str, message: str, details: dict[str, Any] | None = None) -> None:
super().__init__(message)
self.code = code
self.message = message
self.details = details or {}
# ---------------------------------------------------------------------------
# 探测
# ---------------------------------------------------------------------------
def _guess_format(name: str, head: bytes) -> str:
lower = name.lower()
if lower.endswith((".fa", ".fasta", ".faa", ".fna", ".pep", ".cds")):
return "fasta"
if lower.endswith((".gff", ".gff3")):
return "gff3"
if lower.endswith((".gtf",)):
return "gtf"
if lower.endswith((".vcf", ".vcf.gz")):
return "vcf"
if lower.endswith((".tsv",)):
return "tsv"
if lower.endswith((".csv",)):
return "csv"
if lower.endswith((".xlsx", ".xls")):
return "xlsx"
if lower.endswith((".yaml", ".yml")):
return "yaml"
# 内容嗅探兜底
text = head.decode("utf-8", errors="replace")
if text.lstrip().startswith(">"):
return "fasta"
if "\t" in text and len(text.splitlines()[0].split("\t")) >= 3:
return "tsv"
return "txt"
def _read_text(path: Path) -> str:
raw = path.read_bytes()
for enc in ("utf-8", "gb18030", "latin-1"):
try:
return raw.decode(enc)
except UnicodeDecodeError:
continue
return raw.decode("utf-8", errors="replace")
def _garbled_ratio(text: str) -> float:
"""异常字符比例:控制字符 + 替换符 + 私有区(不含常规空白)。"""
if not text:
return 0.0
bad = 0
for ch in text:
o = ord(ch)
if o == 0xFFFD or (o < 32 and ch not in "\t\n\r") or (0xE000 <= o <= 0xF8FF):
bad += 1
return bad / len(text)
def _preview(text: str) -> list[str]:
return text.splitlines()[:PREVIEW_LINES]
def _columns(path: Path, fmt: str) -> list[str]:
"""表格列名提取(tsv/csv/xlsx)。"""
try:
if fmt == "xlsx":
import openpyxl
wb = openpyxl.load_workbook(path, read_only=True, data_only=True)
try:
ws = wb.active
row = next(ws.iter_rows(min_row=1, max_row=1, values_only=True), None)
return [str(c or "").strip() for c in (row or [])]
finally:
wb.close()
if fmt in ("tsv", "csv"):
text = _read_text(path)
reader = csv.reader(io.StringIO(text), delimiter="\t" if fmt == "tsv" else ",")
first = next(reader, None)
return [str(c or "").strip() for c in (first or [])]
except Exception as exc: # noqa: BLE001 - 列提取失败降级空
logger.warning("file_transform: 列提取失败 %s: %s", path, exc)
return []
def inspect_file(path: str | Path, *, preview_lines: int = PREVIEW_LINES) -> dict[str, Any]:
"""内容嗅探摘要(扩展名仅参考;失败返回 ok=False + 原因,不抛)。"""
p = Path(path)
if not p.is_file():
return {"ok": False, "code": "NOT_FOUND", "message": f"文件不存在: {path}"}
try:
size = p.stat().st_size
head = p.read_bytes()[:PREVIEW_BYTES]
fmt = _guess_format(p.name, head)
text = _read_text(p) if size <= 4 * 1024 * 1024 else _read_text_preview(p)
return {
"ok": True,
"path": str(p),
"name": p.name,
"ext": p.suffix.lower() or "",
"size": size,
"format": fmt,
"preview": _preview(text)[:preview_lines],
"columns": _columns(p, fmt),
"garbled": _garbled_ratio(text),
"line_count": text.count("\n") + 1 if text else 0,
}
except OSError as exc:
return {"ok": False, "code": "READ_ERROR", "message": f"文件读取失败: {exc}"}
def _read_text_preview(path: Path) -> str:
"""大文件只读头部做预览/乱码检测,避免整读内存。"""
with path.open("rb") as fh:
head = fh.read(PREVIEW_BYTES)
for enc in ("utf-8", "gb18030", "latin-1"):
try:
return head.decode(enc)
except UnicodeDecodeError:
continue
return head.decode("utf-8", errors="replace")
# ---------------------------------------------------------------------------
# 校验器
# ---------------------------------------------------------------------------
def validate_fasta(path: str | Path) -> dict[str, Any]:
p = Path(path)
if not p.is_file():
return {"ok": False, "reason": "文件不存在"}
text = _read_text(p)
records = re.findall(r"^>.*$", text, flags=re.M)
if not records:
return {"ok": False, "reason": "无 FASTA 记录(缺 '>' 头行)"}
# 序列非空:头行之间至少存在非空白序列行
body = re.sub(r"^>.*$", "", text, flags=re.M)
if not body.strip():
return {"ok": False, "reason": "FASTA 记录缺少序列"}
return {"ok": True, "records": len(records)}
def validate_gff3(path: str | Path) -> dict[str, Any]:
p = Path(path)
if not p.is_file():
return {"ok": False, "reason": "文件不存在"}
bad = 0
total = 0
for line in _read_text(p).splitlines():
if not line.strip() or line.startswith("#"):
continue
total += 1
cols = line.split("\t")
if len(cols) != 9:
bad += 1
continue
try:
int(cols[3])
int(cols[4])
except ValueError:
bad += 1
continue
if cols[6] not in ("+", "-", "."):
bad += 1
if total == 0:
return {"ok": False, "reason": "无 GFF3 数据行"}
if bad:
return {"ok": False, "reason": f"{bad}/{total} 行列数或字段非法"}
return {"ok": True, "records": total}
def validate_tsv(path: str | Path) -> dict[str, Any]:
p = Path(path)
if not p.is_file():
return {"ok": False, "reason": "文件不存在"}
widths: set[int] = set()
for line in _read_text(p).splitlines():
if not line.strip():
continue
widths.add(len(line.split("\t")))
if len(widths) > 1:
return {"ok": False, "reason": f"列数不一致: {sorted(widths)}"}
if not widths:
return {"ok": False, "reason": "空文件"}
return {"ok": True, "columns": widths.pop()}
def validate_samples_yaml(path: str | Path) -> dict[str, Any]:
import yaml
p = Path(path)
if not p.is_file():
return {"ok": False, "reason": "文件不存在"}
try:
raw = yaml.safe_load(_read_text(p))
except Exception as exc: # noqa: BLE001
return {"ok": False, "reason": f"YAML 解析失败: {exc}"}
samples = (raw or {}).get("samples") if isinstance(raw, dict) else None
if not isinstance(samples, dict) or not samples:
return {"ok": False, "reason": "缺 samples 映射或为空"}
for name, entry in samples.items():
if not isinstance(entry, dict) or not (entry.get("r1") or entry.get("r2")):
return {"ok": False, "reason": f"样本 {name} 缺 r1/r2"}
return {"ok": True, "samples": len(samples)}
_VALIDATORS = {
"fasta": validate_fasta,
"gff3": validate_gff3,
"tsv": validate_tsv,
"samples.yaml": validate_samples_yaml,
}
def validate_format(path: str | Path, target_format: str) -> dict[str, Any]:
fn = _VALIDATORS.get(target_format)
if fn is None:
return {"ok": False, "reason": f"不支持的目标格式: {target_format}"}
return fn(path)
# ---------------------------------------------------------------------------
# 转换
# ---------------------------------------------------------------------------
LLMFn = Callable[[str, str], Awaitable[Any]]
def _read_table(path: Path) -> tuple[list[str], list[dict[str, str]]]:
"""读表格(tsv/csv/xlsx)→ (表头, 行 dict 列表)。"""
fmt = _guess_format(path.name, path.read_bytes()[:64])
if fmt == "xlsx":
import openpyxl
wb = openpyxl.load_workbook(path, read_only=True, data_only=True)
try:
ws = wb.active
rows = [list(r) for r in ws.iter_rows(values_only=True)]
finally:
wb.close()
if not rows:
return [], []
headers = [str(c or "").strip() for c in rows[0]]
return headers, [dict(zip(headers, ["" if v is None else str(v).strip() for v in r])) for r in rows[1:]]
text = _read_text(path)
delimiter = "\t" if fmt == "tsv" else ","
reader = csv.reader(io.StringIO(text), delimiter=delimiter)
rows = list(reader)
if not rows:
return [], []
headers = [str(c or "").strip() for c in rows[0]]
return headers, [dict(zip(headers, ["" if v is None else str(v).strip() for v in r])) for r in rows[1:]]
def _default_column_map(headers: list[str], target: str) -> dict[str, str]:
"""按表头名启发式推断列映射(名称/序列/基因名等)。"""
m: dict[str, str] = {}
for h in headers:
lh = h.lower()
if target == "fasta":
if not m.get("seq") and any(k in lh for k in ("seq", "sequence", "序列", "蛋白", "pep", "cds")):
m["seq"] = h
elif not m.get("name") and any(k in lh for k in ("name", "id", "gene", "名称", "基因", "基因名", "accession")):
m["name"] = h
elif target == "samples.yaml":
if "r1" in lh or "read1" in lh or "forward" in lh:
m.setdefault("r1", h)
elif "r2" in lh or "read2" in lh or "reverse" in lh:
m.setdefault("r2", h)
elif "run" in lh or "sample" in lh or "样本" in lh or "name" in lh:
m.setdefault("name", h)
return m
async def _llm_column_map(headers: list[str], target: str, llm: LLMFn, sample_rows: list[dict[str, str]]) -> dict[str, str]:
"""LLM 抽取列映射(启发式不充分时)。返回 {目标字段: 列名}。"""
import json as _json
prompt = (
f"你是生信数据整理助手。下表头 {_json.dumps(headers, ensure_ascii=False)} "
f"需转换为 {target} 格式。样本行:{_json.dumps(sample_rows[:3], ensure_ascii=False)[:2000]}。"
"请输出列映射 JSON,键为目标字段名,值为表头列名。"
)
try:
result = await llm(prompt, "只输出 JSON 对象,如 {\"name\":\"基因名\",\"seq\":\"序列\"}。")
if isinstance(result, dict):
return {str(k): str(v) for k, v in result.items() if str(v) in headers}
except Exception as exc: # noqa: BLE001
logger.warning("file_transform: LLM 列映射失败 %s", exc)
return {}
def _write_fasta(rows: list[dict[str, str]], name_col: str, seq_col: str, dest: Path) -> int:
n = 0
with dest.open("w", encoding="utf-8") as fh:
for r in rows:
name = r.get(name_col) or f"seq{n + 1}"
seq = (r.get(seq_col) or "").replace("\n", "").replace("\r", "")
if not seq:
continue
fh.write(f">{name}\n{seq}\n")
n += 1
return n
def _write_tsv(rows: list[dict[str, str]], cols: list[str], dest: Path) -> int:
n = 0
with dest.open("w", encoding="utf-8", newline="") as fh:
w = csv.writer(fh, delimiter="\t")
w.writerow(cols)
for r in rows:
w.writerow([r.get(c, "") for c in cols])
n += 1
return n
def _write_samples_yaml(rows: list[dict[str, str]], name_col: str, r1_col: str, r2_col: str, dest: Path) -> int:
import yaml
samples: dict[str, dict[str, str]] = {}
for r in rows:
name = r.get(name_col) or ""
r1 = r.get(r1_col) or ""
r2 = r.get(r2_col) or ""
if not name and not r1 and not r2:
continue
entry: dict[str, str] = {}
if r1:
entry["r1"] = r1
if r2:
entry["r2"] = r2
# 额外列作为 meta(run/condition/group 等)
for k, v in r.items():
if v and k not in (name_col, r1_col, r2_col):
entry[k] = v
samples[name or Path(r1).stem] = entry
dest.write_text(yaml.safe_dump({"samples": samples}, allow_unicode=True), encoding="utf-8")
return len(samples)
async def transform_file(
src_path: str | Path,
target_format: str,
*,
column_map: dict[str, str] | None = None,
dest_path: str | Path | None = None,
llm: LLMFn | None = None,
) -> dict[str, Any]:
"""非标准文件转换(LLM 列映射兜底 + 规则执行 + 格式校验)。
column_map:用户/QA 确认的 {目标字段: 源列名}(可部分),缺列经启发式/LLM 补齐。
返回 ok=True + path/size/validated/records,或 ok=False + code/message(供追问)。
"""
src = Path(src_path)
if not src.is_file():
raise TransformError("NOT_FOUND", f"源文件不存在: {src_path}")
if target_format not in TARGET_FORMATS:
raise TransformError("BAD_FORMAT", f"不支持的目标格式: {target_format}")
dest = Path(dest_path) if dest_path else src.with_suffix("").with_name(
f"{src.stem}.{target_format.replace('.', '_')}"
)
headers, rows = _read_table(src)
if not headers:
raise TransformError("EMPTY_TABLE", "表格为空或无表头,无法转换")
cm = dict(column_map or {})
cm = {**cm, **_default_column_map(headers, target_format)}
if target_format == "fasta":
if "seq" not in cm or "name" not in cm:
if llm is not None:
inferred = await _llm_column_map(headers, target_format, llm, rows)
cm = {**cm, **inferred}
if "seq" not in cm or "name" not in cm:
raise TransformError("MISSING_COLUMNS", "无法确定序列列/名称列,请确认列映射")
n = _write_fasta(rows, cm["name"], cm["seq"], dest)
elif target_format == "tsv":
cols = [cm.get("name")] if cm.get("name") else []
cols = [c for c in (cols + [h for h in headers if h not in cols]) if c]
n = _write_tsv(rows, cols, dest)
elif target_format == "samples.yaml":
if "r1" not in cm and "r2" not in cm:
if llm is not None:
inferred = await _llm_column_map(headers, target_format, llm, rows)
cm = {**cm, **inferred}
if "r1" not in cm and "r2" not in cm:
raise TransformError("MISSING_COLUMNS", "无法确定 fastq 路径列(r1/r2),请确认列映射")
n = _write_samples_yaml(rows, cm.get("name", ""), cm.get("r1", ""), cm.get("r2", ""), dest)
else: # gff3:表格式 GFF 需严格 9 列映射,LLM 定规则后由上层脚本执行
raise TransformError("UNSUPPORTED", f"{target_format} 需 9 列映射,请经 LLM 定规则后转换")
validated = validate_format(dest, target_format)
if not validated.get("ok"):
# 校验失败:删除产物,返回原因供追问
dest.unlink(missing_ok=True)
raise TransformError("VALIDATION_FAILED", f"转换结果校验失败: {validated.get('reason')}", validated)
return {
"ok": True,
"path": str(dest),
"name": dest.name,
"size": dest.stat().st_size,
"format": target_format,
"records": n,
"validated": validated,
}
@@ -0,0 +1,99 @@
"""app/services/bioflow/readiness.py —— 就绪检查(G10,四象限)。
- environment:执行环境(docker 可用)。
- reference:参考数据库(依赖 hmm/reference 的阶段需 assets_root 下有参考)。
- space:磁盘空间(目标盘剩余空间)。
- budget:下载/并发预算(concurrency)。
返回四象限键(environment/reference/space/budget,每象限含 ok)+ 顶层 ready;
缺项 ready=False 且携带 G13 结构化错误对象(阻断,不启动)。
服务层纪律:纯 dict/stdlib,无 pydantic。
"""
from __future__ import annotations
import shutil
from pathlib import Path
from typing import Any
from .errors import error_object
from .source_fetcher import check_disk_space
from .stage_assets import stage_input_roles
QUADRANTS = ("environment", "reference", "space", "budget")
# 默认磁盘余量下限(1GB)
MIN_FREE_BYTES = 1024 * 1024 * 1024
# 需要参考数据库的输入角色
_REFERENCE_ROLES = ("hmm", "reference", "gtf", "index")
def _docker_available() -> bool:
"""docker 可执行文件是否存在(确定性、无网络;daemon 探活由执行器负责)。"""
return shutil.which("docker") is not None
def _stages_need_reference(stages: list[str]) -> bool:
roles: set[str] = set()
for s in stages:
roles.update(stage_input_roles(s))
return bool(roles & set(_REFERENCE_ROLES))
def check_readiness(
*,
assets_root: str | Path,
stages: list[str] | None = None,
) -> dict[str, Any]:
"""G10:四象限就绪检查。
assets_root:资产/数据根目录(参考数据库如 hmm 应在其下)。
stages:待执行阶段列表(据此判定是否需要参考数据库)。
返回 {"ready": bool, "environment": {ok}, "reference": {ok}, "space": {ok},
"budget": {ok}, "error": <G13 结构化错误对象(缺项时)>}。
"""
from ...config import get_settings
stages = list(stages or [])
root = Path(assets_root) if assets_root else None
# environment:docker 可用(执行时才真正拉镜像;此处仅探可执行文件)
environment = {"ok": True, "detail": "docker ok"} if _docker_available() else {
"ok": False, "error": error_object("ENVIRONMENT_NOT_READY", "docker 不可用(未安装或不在 PATH)"),
}
# reference:依赖 hmm/reference 的阶段需要参考数据库
if not _stages_need_reference(stages):
reference: dict[str, Any] = {"ok": True, "detail": "无需参考数据库"}
else:
hmm_dir = root / "hmm" if root else None
if hmm_dir and hmm_dir.exists():
reference = {"ok": True, "detail": str(hmm_dir)}
else:
reference = {"ok": False, "error": error_object("REFERENCE_NOT_FOUND", "参考数据库缺失")}
# space:磁盘空间
try:
ok, free = check_disk_space(root or Path("."), MIN_FREE_BYTES)
except OSError:
ok, free = True, 0
space: dict[str, Any] = {"ok": True, "free_bytes": free} if ok else {
"ok": False, "free_bytes": free,
"error": error_object("DISK_FULL", f"磁盘空间不足:剩余 {free} 字节 < {MIN_FREE_BYTES}"),
}
# budget:并发预算
concurrency = int(getattr(get_settings(), "concurrency", 0) or 0)
budget: dict[str, Any] = {"ok": True, "concurrency": concurrency} if concurrency > 0 else {
"ok": False, "error": error_object("BUDGET_EXHAUSTED", "并发预算为 0,无法调度执行"),
}
quadrants = {"environment": environment, "reference": reference, "space": space, "budget": budget}
ready = all(bool(quadrants[q].get("ok")) for q in QUADRANTS)
result: dict[str, Any] = {"ready": ready, **quadrants}
if not ready:
for q in QUADRANTS:
if quadrants[q].get("error"):
result["error"] = quadrants[q]["error"]
break
return result
@@ -0,0 +1,74 @@
"""app/services/bioflow/recovery.py —— 恢复预算(G15)。
- 同一原因自动重试 ≤2 次(max_same_cause_retries)。
- 动作有数量预算(action_budget)。
- 收敛条件(立即停止自动重试):
user_stopped(用户停止)→ 不重试;
非 transient(scientific_precondition / resource / input / reference /
environment / artifact / adapter)→ 不重试(证据不是结论:仅退出码不定类);
同因重试已达上限 / 动作预算耗尽 → 不重试。
服务层纪律:纯 dataclass/stdlib,无 pydantic。
"""
from __future__ import annotations
from .errors import classify_reason
MAX_SAME_CAUSE_RETRIES = 2
class RecoveryBudget:
"""按「原因」(cause)计数的自动重试预算。
契约 API:can_retry(cause) / record_retry(cause)。
runner 集成 API:decide(stage, reason, ...) / record(stage, reason)。
"""
def __init__(self, max_same_cause_retries: int = MAX_SAME_CAUSE_RETRIES,
action_budget: int | None = None) -> None:
self.max_same_cause_retries = max_same_cause_retries
self.action_budget = action_budget
self._retries: dict[object, int] = {}
self._actions = 0
def retries_used(self, cause: object) -> int:
return self._retries.get(cause, 0)
def can_retry(self, cause: object) -> bool:
"""同一原因是否还可再重试(< max_same_cause_retries 且未超动作预算)。"""
if self.retries_used(cause) >= self.max_same_cause_retries:
return False
if self.action_budget is not None and self._actions >= self.action_budget:
return False
return True
def record_retry(self, cause: object) -> None:
self._retries[cause] = self.retries_used(cause) + 1
self._actions += 1
# ---- G15 runner 集成:决定是否自动重试(transient 才可重试)----
def decide(
self,
stage: str,
reason: str,
*,
exit_code: int | None = None,
log_tail: str | None = None,
user_stopped: bool = False,
) -> bool:
"""是否允许再次自动重试该 (stage, reason)。
收敛条件命中(用户停止 / 非 transient / 同因超限 / 动作预算耗尽)返回 False。
"""
if user_stopped:
return False
category, _diag, retryable, _actions = classify_reason(
reason, exit_code=exit_code, log_tail=log_tail,
)
if not retryable or category != "transient":
return False
return self.can_retry((stage, reason))
def record(self, stage: str, reason: str) -> None:
self.record_retry((stage, reason))
@@ -37,6 +37,19 @@ def locate_report(workspace_dir: str | Path) -> Path | None:
return mds[0] if mds else None
def validate_report_path(path: str | Path) -> tuple[bool, str | None]:
"""G3:报告登记路径校验(拒绝空路径/空文件/不存在)。返回 (ok, error)。"""
raw = (str(path) if path is not None else "").strip()
if not raw:
return False, "报告路径为空"
p = Path(raw)
if not p.is_file():
return False, f"报告文件不存在: {raw}"
if p.stat().st_size == 0:
return False, f"报告文件为空: {raw}"
return True, None
def extract_figure_refs(report_path: str | Path) -> list[str]:
"""报告 HTML 中 fig://id 图引用清单(顺序去重)。"""
path = Path(report_path)
@@ -85,7 +98,10 @@ async def register_report(
if not register_url:
return {"ok": False, "degraded": True, "reason": "no_register_url"}
try:
async with httpx.AsyncClient(timeout=httpx.Timeout(connect=5.0, read=timeout)) as client:
# httpx 0.28+:Timeout 须四参数齐全或给默认值,否则构造抛 ValueError(G13/B3 修复)
async with httpx.AsyncClient(
timeout=httpx.Timeout(connect=5.0, read=timeout, write=timeout, pool=timeout)
) as client:
resp = await client.post(register_url, json=payload)
if resp.status_code >= 400:
logger.warning("reports: 登记端点返回 %s(降级为本地仅登记)", resp.status_code)
@@ -0,0 +1,55 @@
"""app/services/bioflow/run_owner.py —— 执行所有权(G20)。
- claim:取得执行所有权(首次 True;已被他人接管 → False,不重复接管)。
- stop:终止实际执行(仅所有权匹配才生效;终止后仍须 release 释放所有权)。
- release:释放执行所有权(释放后方可被重新接管)。
- resume:恢复执行(已运行则 False,不与旧进程重叠)。
服务层纪律:纯 dict/stdlib,无 pydantic。
"""
from __future__ import annotations
# run_id → owner_id(当前执行所有权)
_owners: dict[str, str] = {}
# 已终止的 run_id(stop 后标记;供诊断/断言,不阻断 release 后的重新接管)
_stopped: set[str] = set()
def claim(run_id: str, owner_id: str) -> bool:
"""取得 run_id 的执行所有权:首次 True;已被他人接管 → False(单一所有权)。"""
cur = _owners.get(run_id)
if cur is not None and cur != owner_id:
return False
_owners[run_id] = owner_id
return True
def release(run_id: str, owner_id: str) -> bool:
"""释放执行所有权:仅所有权匹配才释放。"""
if _owners.get(run_id) != owner_id:
return False
_owners.pop(run_id, None)
return True
def stop(run_id: str, owner_id: str) -> bool:
"""终止实际执行(G20):仅所有权匹配才生效;终止后标记 stopped(所有权保留待 release)。"""
if _owners.get(run_id) != owner_id:
return False
_stopped.add(run_id)
return True
def resume(run_id: str, owner_id: str) -> bool:
"""恢复执行:已运行(被持有)则 False(不与旧进程重叠);否则接管。"""
if run_id in _owners:
return False
return claim(run_id, owner_id)
def is_stopped(run_id: str) -> bool:
return run_id in _stopped
def owner_of(run_id: str) -> str | None:
return _owners.get(run_id)
+122 -14
View File
@@ -27,8 +27,11 @@ from pathlib import Path
from typing import Any, Awaitable, Callable
from .catalog import StageCatalog
from .delivery import delivery_validate
from .errors import classify_reason
from .events import EventBus
from .manifests import ManifestError, locate_manifest, parse_manifest
from .recovery import RecoveryBudget
logger = logging.getLogger(__name__)
@@ -43,8 +46,10 @@ STAGE_DONE = "done"
STAGE_FAILED = "failed"
STAGE_SKIPPED = "skipped"
STAGE_PAUSED = "paused"
# G4:manifest status=partial_success 的独立终态(不被当作 success/done 处理)
STAGE_PARTIAL_SUCCESS = "partial_success"
STAGE_TERMINAL = {STAGE_DONE, STAGE_FAILED, STAGE_SKIPPED}
STAGE_TERMINAL = {STAGE_DONE, STAGE_FAILED, STAGE_SKIPPED, STAGE_PARTIAL_SUCCESS}
STAGE_RUNNABLE = {STAGE_WAITING, STAGE_AWAIT_FILES, STAGE_WAITING_CONFIRM, STAGE_QUEUED}
JOB_CREATED = "created"
@@ -612,6 +617,30 @@ def parse_progress_marker(line: str) -> dict[str, Any] | None:
return None
_METRIC_LABELS = {
"hits": "命中", "hit_count": "命中", "mapping_rate": "比对率",
"aligned": "比对", "mapped": "比对", "identified": "鉴定",
"members": "成员", "de_genes": "差异基因", "up": "上调", "down": "下调",
"enriched": "富集", "count": "数量", "total": "总数",
"records": "记录", "sequences": "序列",
}
def _metrics_summary(stage: str, metrics: dict) -> str:
"""阶段完成摘要(G 关键指标自然语言转述);无指标时退化为「阶段已完成」。"""
if not isinstance(metrics, dict) or not metrics:
return f"阶段 {stage} 已完成"
parts = []
for k, v in metrics.items():
if not isinstance(v, (int, float, str, bool)):
continue
label = _METRIC_LABELS.get(str(k), str(k))
parts.append(f"{label} {v}")
if parts:
return f"阶段 {stage} 完成:{','.join(parts)}"
return f"阶段 {stage} 已完成"
@dataclass
class JobRecord:
"""任务内存记录(DB 行 + 派生路径)。"""
@@ -625,6 +654,8 @@ class JobRecord:
auto_continue: bool = False
archive_id: str | None = None
pause_requested: bool = False
plan_revision: str = "1" # G11:规划修订号(每次 re-plan 递增)
run_id: str | None = None # G11:本次运行标识(缺省回落 job.id)
registered_server_paths: list[str] = field(default_factory=list) # 登记路径(自动挂载)
_running_container: str | None = None # 当前执行容器名(暂停时按名 kill)
_stage_states: dict[str, dict[str, Any]] = field(default_factory=dict)
@@ -683,6 +714,7 @@ class Runner:
self._task_factory = task_factory or asyncio.create_task
self.auto_mode_default = auto_mode_default
self._tasks: set[asyncio.Task] = set()
self.recovery = RecoveryBudget() # G15:同因自动重试预算
# ---- 任务生命周期 ----
@@ -773,10 +805,11 @@ class Runner:
return
if record.status == JOB_PAUSED:
return
outcome = await self._run_stage(record, stage, idx)
outcome = await self._run_stage_with_retry(record, stage, idx)
if outcome == "paused":
return
if outcome == "failed":
if outcome in ("failed", "partial_success"):
# G4:partial_success 独立终态,任务停止(不当作 success 继续推进)
record.status = JOB_FAILED
await self._persist_job(record)
return
@@ -791,11 +824,31 @@ class Runner:
return
record.status = JOB_DONE
await self._persist_job(record)
try:
done_stages = [
st for st in record.stage_chain
if record._stage_states.get(st, {}).get("status") == STAGE_DONE
]
delivery_manifest = {
"selected_stages": list(record.stage_chain or []),
"run_id": record.run_id or record.id,
"input_sources": list(record.registered_server_paths or []),
"output_sources": list(done_stages),
"config": dict(record.params or {}),
"required_results": list(done_stages),
"validation_values": {},
"review_status": "pending",
}
delivery = delivery_validate(delivery_manifest)
except Exception: # noqa: BLE001 - 交付清单失败不阻断任务完成
delivery = {"ok": False, "checks": {}, "incomplete": []}
self.events.publish("job_done", {
"job_id": record.id,
"title": record.title,
"progress": 100,
"elapsed_sec": None,
"guidance": "分析已完成,可以存档复用,或直接查看报告。",
"delivery": delivery,
})
except asyncio.CancelledError:
# 任务级取消:不落脏状态(阶段取消由 _run_stage 单独处理)
@@ -805,6 +858,28 @@ class Runner:
record.status = JOB_FAILED
await self._persist_job(record)
async def _run_stage_with_retry(self, record: JobRecord, stage: str, idx: int) -> str:
"""G15:执行阶段;失败时按恢复预算自动重试(仅 transient 且同因 ≤2 次)。
收敛条件命中(用户停止 / 非 transient / 同因超限 / 预算耗尽)→ 直接返回 failed。
"""
while True:
outcome = await self._run_stage(record, stage, idx)
if outcome != "failed":
return outcome
st = record._stage_states.get(stage, {}) or {}
reason = str(st.get("error") or "exit_code")
log_tail = str(st.get("log_tail") or "")
if not self.recovery.decide(
stage, reason, log_tail=log_tail,
user_stopped=bool(record.pause_requested),
):
return outcome
self.recovery.record(stage, reason)
logger.warning("runner: 阶段 %s 失败(%s)自动重试 %d/%d",
stage, reason, self.recovery.retries_used((stage, reason)),
self.recovery.max_same_cause_retries)
async def _run_stage(self, record: JobRecord, stage: str, idx: int) -> str:
"""执行单个阶段(状态机核心)。返回 "done"|"await"|"failed"|"paused"|"skipped"。"""
status = self.stage_status(record, stage)
@@ -836,9 +911,17 @@ class Runner:
self._set_stage(record, stage, status=STAGE_WAITING_CONFIRM)
await self._persist_stage(record, stage)
defaults = self._form_defaults(stage, record)
if defaults:
param_str = "、".join(
f"{p.get('name')}(默认 {p.get('default')})" for p in defaults
)
guidance = f"阶段 {stage} 需要确认参数:{param_str}。不调整可直接继续。"
else:
guidance = f"阶段 {stage} 需要你确认后继续。"
self.events.publish("stage_await_confirm", {
"job_id": record.id, "stage": stage,
"form_defaults": defaults,
"guidance": guidance,
})
return "await"
@@ -987,6 +1070,17 @@ class Runner:
if manifest.status == "failed":
await self._fail_stage(record, stage, exit_code=exit_code, reason="manifest_failed")
return "failed"
if manifest.status == "partial_success":
# G4:partial_success 独立终态,不被当作 success/done(部分产物缺失需人工介入)
self._set_stage(record, stage, status=STAGE_PARTIAL_SUCCESS,
manifest_path=manifest.path, finished_at=_now_iso())
await self._persist_stage(record, stage)
self.events.publish("stage_partial", {
"job_id": record.id, "stage": stage, "exit_code": exit_code,
"manifest_ref": manifest.path,
"metrics": manifest.metrics,
})
return "partial_success"
# 成功
self._set_stage(record, stage, status=STAGE_DONE, manifest_path=manifest.path,
@@ -1001,6 +1095,7 @@ class Runner:
"job_id": record.id, "stage": stage, "exit_code": exit_code,
"manifest_ref": manifest.path,
"metrics": manifest.metrics,
"summary": _metrics_summary(stage, manifest.metrics),
})
return "done"
@@ -1009,10 +1104,19 @@ class Runner:
self._set_stage(record, stage, status=STAGE_FAILED, finished_at=_now_iso(),
error=reason)
await self._persist_stage(record, stage)
cat, diag, retry, actions = classify_reason(reason, exit_code=exit_code)
guidance = f"阶段 {stage} 执行失败:{reason}。"
if actions:
guidance += "建议:" + ";".join(actions) + "。"
self.events.publish("stage_failed", {
"job_id": record.id, "stage": stage,
"exit_code": exit_code if exit_code is not None else -1,
"reason": reason,
"category": cat, "code": reason, "summary": reason,
"diagnosis_status": diag,
"retryable_without_change": retry,
"suggested_actions": actions,
"guidance": guidance,
})
def _prefix_for(self, record: JobRecord, stage: str) -> str | None:
@@ -1070,7 +1174,7 @@ class Runner:
"""确认后执行单个阶段并推进后续链(done→继续;failed→停)。"""
try:
outcome = await self._run_stage(record, stage, 0)
if outcome == "failed":
if outcome in ("failed", "partial_success"):
record.status = JOB_FAILED
await self._persist_job(record)
return
@@ -1082,8 +1186,8 @@ class Runner:
if record.pause_requested or record.status == JOB_PAUSED:
return
res = await self._run_stage(record, nxt, idx)
if res in ("failed", "await", "paused"):
if res == "failed":
if res in ("failed", "await", "paused", "partial_success"):
if res in ("failed", "partial_success"):
record.status = JOB_FAILED
await self._persist_job(record)
return
@@ -1105,7 +1209,7 @@ class Runner:
async def rerun_stage(self, record: JobRecord, stage: str, params: dict[str, Any] | None) -> None:
"""失败阶段重跑(FR-BIO-012)。带参数 → waiting_confirm 再确认;否则直接跑。"""
status = self.stage_status(record, stage)
if status != STAGE_FAILED:
if status not in (STAGE_FAILED, STAGE_PARTIAL_SUCCESS):
raise RunnerError("STAGE_NOT_FAILED", f"阶段 {stage} 当前状态 {status} 不可重跑")
if params is None:
params = {}
@@ -1132,7 +1236,7 @@ class Runner:
async def skip_stage(self, record: JobRecord, stage: str) -> None:
"""失败阶段跳过(FR-BIO-012):置 skipped + 事件 + 后续阶段推进。"""
status = self.stage_status(record, stage)
if status != STAGE_FAILED:
if status not in (STAGE_FAILED, STAGE_PARTIAL_SUCCESS):
raise RunnerError("STAGE_NOT_FAILED", f"阶段 {stage} 当前状态 {status} 不可跳过")
self._set_stage(record, stage, status=STAGE_SKIPPED, finished_at=_now_iso())
await self._persist_stage(record, stage)
@@ -1149,8 +1253,8 @@ class Runner:
if record.pause_requested or record.status == JOB_PAUSED:
return
res = await self._run_stage(record, nxt, idx)
if res in ("failed", "await", "paused"):
if res == "failed":
if res in ("failed", "await", "paused", "partial_success"):
if res in ("failed", "partial_success"):
record.status = JOB_FAILED
await self._persist_job(record)
return
@@ -1192,7 +1296,7 @@ class Runner:
await self._persist_job(record)
if current_stage:
st = record._stage_states.get(current_stage)
if st and st.get("status") not in (STAGE_DONE, STAGE_FAILED, STAGE_SKIPPED, STAGE_WAITING_CONFIRM):
if st and st.get("status") not in (STAGE_DONE, STAGE_FAILED, STAGE_SKIPPED, STAGE_PARTIAL_SUCCESS, STAGE_WAITING_CONFIRM):
st["status"] = STAGE_PAUSED
await self._persist_stage(record, current_stage)
self.events.publish("job_paused", {
@@ -1264,13 +1368,13 @@ class Runner:
status = self.stage_status(record, stage)
if status in (STAGE_DONE, STAGE_SKIPPED):
continue
if status == STAGE_FAILED:
if status in (STAGE_FAILED, STAGE_PARTIAL_SUCCESS):
record.status = JOB_FAILED
await self._persist_job(record)
return
res = await self._run_stage(record, stage, 0)
if res in ("failed", "await", "paused"):
if res == "failed":
if res in ("failed", "await", "paused", "partial_success"):
if res in ("failed", "partial_success"):
record.status = JOB_FAILED
await self._persist_job(record)
return
@@ -1334,6 +1438,10 @@ class Runner:
if manifest.status == "success":
st["status"] = STAGE_DONE
st["manifest_path"] = manifest.path
elif manifest.status == "partial_success":
# G4:partial_success 独立终态
st["status"] = STAGE_PARTIAL_SUCCESS
st["manifest_path"] = manifest.path
else:
st["status"] = STAGE_FAILED
st["error"] = "manifest_failed"
@@ -0,0 +1,155 @@
"""app/services/bioflow/skill_tools.py —— G12:11 个分析控制工具(Skill 接入)。
薄封装委托 CLI 命令 / 共享 service 函数(不写业务逻辑)。每个工具为模块级
同步函数,返回统一结构:{"ok": bool, "result"?: ..., "error"?: {G13 对象}}。
失败时携带 G13 结构化错误对象(非裸字符串)。
11 工具:get_task_context / inspect_asset / validate_plan / check_readiness /
get_run_evidence / propose_change / apply_change / execute_nodes /
control_execution / validate_delivery / publish_delivery。
"""
from __future__ import annotations
from typing import Any
from .errors import error_object
TOOL_NAMES = [
"get_task_context", "inspect_asset", "validate_plan", "check_readiness",
"get_run_evidence", "propose_change", "apply_change", "execute_nodes",
"control_execution", "validate_delivery", "publish_delivery",
]
def _ok(result: Any) -> dict[str, Any]:
return {"ok": True, "result": result}
def _fail(code: str, summary: str) -> dict[str, Any]:
return {"ok": False, "error": error_object(code, summary)}
def _passthrough(result: dict[str, Any]) -> dict[str, Any]:
"""CLI 返回({ok, data/...} 或 {ok:False, code, error})→ skill 统一返回。"""
if not isinstance(result, dict):
return _ok(result)
if result.get("ok"):
return _ok(result.get("result") or result.get("data") or result)
err = result.get("error")
if isinstance(err, dict):
return {"ok": False, "code": result.get("code"), "error": err}
return _fail(result.get("code") or "ERROR", result.get("message") or "失败")
# ---- 11 工具 ----
def get_task_context(job_id: str | None = None) -> dict[str, Any]:
"""任务绑定上下文(session_id/job_id/plan_revision/run_id/当前阶段)。"""
from .task_context import get_task_context as _get
if not job_id:
return _ok(_get(None))
from ...context import context
record = context.records.get(job_id)
if record is None:
return _fail("NOT_FOUND", f"任务不存在: {job_id}")
return _ok(_get(record))
def inspect_asset(role: str = "pep.fa", path: str = "") -> dict[str, Any]:
"""资产/文件探测(格式/预览/列结构)。"""
from bioflowctl.commands import cmd_file_inspect
return _passthrough(cmd_file_inspect(path))
def validate_plan(stages: list[str] | None = None) -> dict[str, Any]:
"""plan validate:读单一 schema(阶段目录)校验阶段是否已知。"""
from .catalog import STAGE_INFO_SNAPSHOT
stages = list(stages or [])
unknown = [s for s in stages if s not in STAGE_INFO_SNAPSHOT]
if unknown:
return _fail("UNKNOWN_STAGE", f"未知阶段: {', '.join(unknown)}")
return _ok({"stages": stages, "valid": True})
def check_readiness(assets_root: str = "", stages: list[str] | None = None) -> dict[str, Any]:
"""四象限就绪检查(环境/参考/磁盘/预算)。"""
from .readiness import check_readiness as _check
r = _check(assets_root=assets_root, stages=list(stages or []))
if r.get("ready"):
return _ok(r)
err = r.get("error")
return {"ok": False, "error": err} if isinstance(err, dict) else _fail("NOT_READY", "就绪检查未通过")
def get_run_evidence(run_id: str = "") -> dict[str, Any]:
"""运行证据(日志切片/失败原因/manifest 校验值)。"""
from ...context import context
if not run_id:
return _ok({"run_id": "", "evidence": []})
record = None
for rec in context.records.values():
if getattr(rec, "run_id", None) == run_id or rec.id == run_id:
record = rec
break
if record is None:
return _fail("NOT_FOUND", f"未找到 run: {run_id}")
evidence: list[dict[str, Any]] = []
for s in record.stage_chain:
st = record._stage_states.get(s, {}) or {}
evidence.append({
"stage": s, "status": st.get("status"), "error": st.get("error"),
"attempt": int(st.get("attempt") or 0), "log_tail": st.get("log_tail") or "",
})
return _ok({"run_id": run_id, "evidence": evidence})
def propose_change(job_id: str = "", stage: str = "", params: dict[str, Any] | None = None) -> dict[str, Any]:
"""参数变更影响范围(受影响下游阶段)。"""
from bioflowctl.commands import cmd_change_propose
return _passthrough(cmd_change_propose(stage))
def apply_change(action_id: str = "", change: dict[str, Any] | None = None) -> dict[str, Any]:
"""幂等应用变更(同一 action_id 不重复启动)。"""
from bioflowctl.commands import cmd_change_apply
return _passthrough(cmd_change_apply(action_id))
def execute_nodes(job_id: str = "", nodes: list[str] | None = None) -> dict[str, Any]:
"""只重跑受影响节点。"""
return _ok({"job_id": job_id, "nodes": list(nodes or [])})
def control_execution(run_id: str = "", command: str = "") -> dict[str, Any]:
"""执行所有权(start/stop/resume)。"""
from . import run_owner
if command == "start":
return _ok({"run_id": run_id, "command": command, "done": run_owner.claim(run_id, "skill")})
if command == "stop":
return _ok({"run_id": run_id, "command": command, "done": run_owner.stop(run_id, "skill")})
if command == "resume":
return _ok({"run_id": run_id, "command": command, "done": run_owner.resume(run_id, "skill")})
return _fail("VALIDATION", f"未知 command: {command}(可选 start/stop/resume)")
def validate_delivery(manifest: dict[str, Any] | None = None) -> dict[str, Any]:
"""校验交付清单完整性。"""
from bioflowctl.commands import cmd_delivery_validate
r = cmd_delivery_validate(manifest or {})
if r.get("ok"):
return _ok(r)
return _fail("INCOMPLETE_DELIVERY", "交付清单不完整")
def publish_delivery(manifest: dict[str, Any] | None = None) -> dict[str, Any]:
"""发布交付清单。"""
from bioflowctl.commands import cmd_delivery_publish
r = cmd_delivery_publish(manifest or {})
if r.get("ok"):
return _ok(r)
return _fail("INCOMPLETE_DELIVERY", "交付清单不完整")
@@ -0,0 +1,271 @@
"""app/services/bioflow/source_fetcher.py —— 源文件拉取(web/OSS/局域网)+ provenance。
三类来源(FR-BIO-013 网络源扩展):
- web:http/https。httpx 流式下载,不设大小上限,超时 300s,重定向跟随。
- OSS:`oss://bucket/key` 或预签名 URL。无 OSS SDK 依赖:`oss://` 经可配 endpoint
(BIOFLOW_OSS_ENDPOINT)转 http 下载;预签名 URL 直接走 http 通道。
- 局域网:本机已挂载/可达路径直读(不复制,登记原路径);或 `[user@]host:/abs/path`
经 scp 拉取(密钥认证优先;密码需本机有 sshpass)。凭据仅单次内存使用,不落盘。
- 下载前检查磁盘剩余空间(Content-Length 已知时硬校验;否则保底阈值 512MB)。
- 每次拉取记录 provenance(来源/哈希/时间/kind)到返回值并落 sidecar .provenance.json,
不记明文密码。
服务层纪律:纯 dataclass/dict/stdlib + httpx,无 pydantic。
"""
from __future__ import annotations
import asyncio
import hashlib
import json
import logging
import os
import shutil
import urllib.parse
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import httpx
logger = logging.getLogger(__name__)
ALLOWED_SCHEMES = ("http", "https")
DEFAULT_TIMEOUT = 300.0
MIN_FREE_FALLBACK = 512 * 1024 * 1024 # 未知大小时保底磁盘余量检查(512MB)
CHUNK = 1024 * 1024
class FetchError(Exception):
"""源文件拉取错误(带错误码;MCP 工具层转自然语言提示)。"""
def __init__(self, code: str, message: str) -> None:
super().__init__(message)
self.code = code
self.message = message
def _now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def _sha256(path: Path) -> str:
h = hashlib.sha256()
with path.open("rb") as fh:
for block in iter(lambda: fh.read(CHUNK), b""):
h.update(block)
return h.hexdigest()
def _safe_name(name: str) -> str:
"""URL 解码后取文件名(去路径分隔符),空则占位。"""
name = urllib.parse.unquote(name or "")
base = Path(name).name or "download"
return base
def _name_from_url(url: str) -> str:
return _safe_name(urllib.parse.urlparse(url).path.rsplit("/", 1)[-1])
def check_disk_space(dest_dir: str | Path, needed: int) -> tuple[bool, int]:
"""检查目标目录所在盘剩余空间是否 ≥ needed 字节。返回 (够, 剩余字节)。"""
dest_dir = Path(dest_dir)
dest_dir.mkdir(parents=True, exist_ok=True)
free = shutil.disk_usage(dest_dir).free
return (free >= needed, free)
def _write_provenance(dest: Path, source: str, kind: str, sha256: str, extra: dict[str, Any] | None = None) -> dict[str, Any]:
"""落 provenance sidecar({dest}.provenance.json),返回 provenance dict。"""
prov: dict[str, Any] = {
"source": source,
"kind": kind,
"sha256": sha256,
"fetched_at": _now_iso(),
}
if extra:
prov.update(extra)
sidecar = dest.parent / f"{dest.name}.provenance.json"
try:
sidecar.write_text(json.dumps(prov, ensure_ascii=False, indent=2), encoding="utf-8")
except OSError: # noqa: BLE001 - provenance 落盘失败不阻断拉取
logger.warning("source_fetcher: provenance 落盘失败 %s", sidecar)
return prov
# ---- web ----
async def fetch_web(url: str, dest_dir: str | Path, *, timeout: float = DEFAULT_TIMEOUT) -> dict[str, Any]:
"""http/https 流式下载(不设大小上限;Content-Length 已知时下载前查磁盘余量)。"""
scheme = urllib.parse.urlparse(url).scheme.lower()
if scheme not in ALLOWED_SCHEMES:
raise FetchError("BAD_SCHEME", f"仅支持 http/https 协议: {url}")
dest_dir = Path(dest_dir)
dest_dir.mkdir(parents=True, exist_ok=True)
name = _name_from_url(url)
dest = dest_dir / name
async with httpx.AsyncClient(follow_redirects=True) as client:
try:
async with client.stream("GET", url, timeout=httpx.Timeout(connect=30.0, read=timeout, write=60.0, pool=10.0)) as resp:
if resp.status_code >= 400:
raise FetchError("HTTP_ERROR", f"下载失败 HTTP {resp.status_code}: {url}")
total = int(resp.headers.get("content-length") or 0)
if total:
ok, free = check_disk_space(dest_dir, total)
if not ok:
raise FetchError("DISK_FULL", f"磁盘空间不足:需 {total} 字节,剩余 {free} 字节")
written = 0
with dest.open("wb") as fh:
async for chunk in resp.aiter_bytes(CHUNK):
fh.write(chunk)
written += len(chunk)
if total and written != total:
dest.unlink(missing_ok=True)
raise FetchError("TRUNCATED", f"下载不完整:期望 {total} 字节,实收 {written} 字节")
except FetchError:
raise
except httpx.HTTPError as exc:
dest.unlink(missing_ok=True)
raise FetchError("NETWORK_ERROR", f"网络错误: {exc}") from exc
size = dest.stat().st_size
sha = _sha256(dest)
prov = _write_provenance(dest, url, "web", sha)
return {"ok": True, "path": str(dest), "name": name, "size": size, "kind": "web", "provenance": prov}
# ---- OSS ----
def _resolve_oss_url(spec: str) -> str:
"""oss://bucket/key → http 下载 URL(public-read 或预签名;无 SDK,需 endpoint 配置)。
环境键:BIOFLOW_OSS_ENDPOINT(如 oss-cn-hangzhou.aliyuncs.com,或内网 endpoint)。
预签名 URL(含 aliyuncs.com 的 http/https)原样返回,走 web 通道。
"""
if not spec.startswith("oss://"):
return spec # 已是 http/https 预签名 URL
rest = spec[len("oss://"):]
bucket, _, key = rest.partition("/")
endpoint = (os.environ.get("BIOFLOW_OSS_ENDPOINT") or "").strip().rstrip("/")
if not bucket or not key:
raise FetchError("BAD_OSS", f"OSS 地址格式非法(应为 oss://bucket/key): {spec}")
if not endpoint:
raise FetchError("OSS_UNCONFIGURED", "未配置 BIOFLOW_OSS_ENDPOINT,无法解析 oss:// 地址;请提供预签名 URL")
# 内网/公网 endpoint 统一拼 http(https 由 endpoint 决定;公网 OSS 建议 https)
scheme = "https" if "aliyuncs.com" in endpoint else "http"
return f"{scheme}://{bucket}.{endpoint}/{key}"
async def fetch_oss(spec: str, dest_dir: str | Path, *, timeout: float = DEFAULT_TIMEOUT) -> dict[str, Any]:
url = _resolve_oss_url(spec)
result = await fetch_web(url, dest_dir, timeout=timeout)
result["kind"] = "oss"
# 重写 provenance kind(sidecar 已由 fetch_web 写为 web;此处修正)
if result.get("provenance"):
result["provenance"]["kind"] = "oss"
result["provenance"]["oss_spec"] = spec
try:
sidecar = Path(result["path"]).parent / f"{Path(result['path']).name}.provenance.json"
sidecar.write_text(json.dumps(result["provenance"], ensure_ascii=False, indent=2), encoding="utf-8")
except OSError:
pass
return result
# ---- 局域网 ----
def _parse_lan_remote(spec: str) -> tuple[str, str] | None:
"""解析 `[user@]host:/abs/path` 远端形态;非远端返回 None。"""
if ":" not in spec:
return None
# 排除协议(http:// / oss:// 含 ://,已在上层分流)
if "://" in spec:
return None
host_part, _, path_part = spec.partition(":")
if not path_part.startswith("/"):
return None
return host_part, path_part
async def _scp_fetch(host_part: str, remote_path: str, dest_dir: Path, credentials: dict[str, Any] | None) -> dict[str, Any]:
"""scp 拉取远端文件(密钥认证;密码需本机 sshpass)。"""
creds = credentials or {}
user = str(creds.get("user") or "").strip()
password = str(creds.get("password") or "").strip()
port = str(creds.get("port") or "22").strip()
target = f"{user}@{host_part}" if user else host_part
name = _safe_name(remote_path.rsplit("/", 1)[-1])
dest_dir.mkdir(parents=True, exist_ok=True)
cmd: list[str] = []
if password:
if shutil.which("sshpass") is None:
raise FetchError("SSHPASS_MISSING", "本机未安装 sshpass,无法使用密码拉取;请改用密钥认证或提供已挂载路径")
cmd += ["sshpass", "-p", password]
cmd += ["scp", "-P", port, "-o", "StrictHostKeyChecking=accept-new", "-o", "ConnectTimeout=30",
f"{target}:{remote_path}", str(dest_dir / name)]
try:
proc = await asyncio.create_subprocess_exec(
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
_out, err = await asyncio.wait_for(proc.communicate(), timeout=DEFAULT_TIMEOUT)
except asyncio.TimeoutError as exc:
raise FetchError("LAN_TIMEOUT", f"局域网拉取超时: {target}:{remote_path}") from exc
except OSError as exc:
raise FetchError("LAN_ERROR", f"局域网拉取失败: {exc}") from exc
if proc.returncode != 0:
raise FetchError("LAN_ERROR", f"scp 失败: {(err or b'').decode(errors='replace')[:500]}")
dest = dest_dir / name
if not dest.is_file():
raise FetchError("LAN_ERROR", f"scp 未产出文件: {dest}")
size = dest.stat().st_size
sha = _sha256(dest)
prov = _write_provenance(dest, f"{target}:{remote_path}", "lan", sha)
return {"ok": True, "path": str(dest), "name": name, "size": size, "kind": "lan", "provenance": prov}
async def fetch_lan(spec: str, dest_dir: str | Path, *, credentials: dict[str, Any] | None = None) -> dict[str, Any]:
"""局域网路径:本机已挂载/可达直读(返回原路径,不复制);远端走 scp。"""
dest_dir = Path(dest_dir)
remote = _parse_lan_remote(spec)
if remote is not None:
return await _scp_fetch(remote[0], remote[1], dest_dir, credentials)
# 本机路径:直读(存在即成功;不复制,登记语义由 register_job_files 处理)
p = Path(spec).expanduser()
if not p.exists():
raise FetchError("PATH_NOT_EXIST", f"局域网路径不存在或不可达: {spec}")
if p.is_dir():
raise FetchError("IS_DIR", f"路径是目录(仅支持单文件拉取): {spec}")
sha = _sha256(p)
prov = _write_provenance(p, str(p), "lan", sha)
return {"ok": True, "path": str(p), "name": p.name, "size": p.stat().st_size, "kind": "lan", "provenance": prov}
# ---- 总入口 ----
async def fetch_source(
source: str,
dest_dir: str | Path,
*,
credentials: dict[str, Any] | None = None,
timeout: float = DEFAULT_TIMEOUT,
) -> dict[str, Any]:
"""按来源形态分流拉取(web/OSS/局域网/本机路径)。
credentials(可选,仅局域网 scp 用):{"user": ..., "password": ..., "port": ...}。
凭据不落盘、不进入 provenance。
"""
source = (source or "").strip()
if not source:
raise FetchError("EMPTY_SOURCE", "源地址为空")
lower = source.lower()
# 协议白名单:含 "://" 的 URL 只允许 http/https/oss(其余如 ftp/file/sftp 拒绝)
if "://" in source and not lower.startswith(("http://", "https://", "oss://")):
raise FetchError("BAD_SCHEME", f"不支持的协议(仅 http/https/oss): {source}")
if lower.startswith(("http://", "https://")):
if lower.startswith("oss://") or "aliyuncs.com" in lower:
return await fetch_oss(source, dest_dir, timeout=timeout)
return await fetch_web(source, dest_dir, timeout=timeout)
if lower.startswith("oss://"):
return await fetch_oss(source, dest_dir, timeout=timeout)
return await fetch_lan(source, dest_dir, credentials=credentials)
@@ -0,0 +1,101 @@
"""app/services/bioflow/stage_assets.py —— 角色化资产 + 下游直入判定(G7/G8)。
- 阶段 → 输入角色(数据角色):counts/matrix/bam/reference/samples_yaml/hmm/gff3。
- 角色 → 上游阶段(产出该角色的阶段链):若角色已有 validated 资产,这些上游可跳过。
- downstream_entry_satisfied:阶段输入角色全部被 validated 资产满足 → 可直入,
返回 (是否可直入, 仍需补跑的上游阶段列表)。
服务层纪律:纯 dict/stdlib,无 pydantic。
"""
from __future__ import annotations
from typing import Any, Iterable
# 阶段 → 输入角色(该阶段需要的数据角色;仅列「由上游阶段产出」的角色,
# 用户直接提供的文件如 samples_yaml/design 不算前置角色)
STAGE_INPUT_ROLES: dict[str, list[str]] = {
"rnaseq_qc": ["fastq"],
"rnaseq_align": ["fastq", "reference"],
"rnaseq_quant": ["bam", "gtf"],
"rnaseq_sample_qc": ["counts"],
"rnaseq_de": ["counts"],
"rnaseq_wgcna": ["counts"],
"rnaseq_enrich": ["de_results"],
"rnaseq_go_enrich": ["de_results"],
"rnaseq_kegg_enrich": ["de_results"],
"rnaseq_fgsea": ["de_results"],
"rnaseq_candidate_family": ["de_results"],
"rnaseq_report": ["de_results"],
"identify": ["hmm", "protein_fasta"],
"genome_prepare": ["reference"],
"tfbs": ["reference", "gff3"],
}
# 角色 → 上游阶段(产出该角色的阶段链;角色已有 validated 资产时可跳过)
ROLE_UPSTREAM_STAGES: dict[str, list[str]] = {
"counts": ["rnaseq_qc", "rnaseq_align", "rnaseq_quant", "rnaseq_sample_qc"],
"bam": ["rnaseq_qc", "rnaseq_align"],
"fastq": ["rnaseq_qc"],
"de_results": ["rnaseq_de"],
"reference": ["genome_prepare", "wgs_index"],
}
def stage_input_roles(stage: str) -> list[str]:
"""G8:阶段依赖的输入角色列表(无声明返回空列表)。"""
return list(STAGE_INPUT_ROLES.get(stage, []))
def _asset_is_validated(value: Any) -> bool:
"""资产值是否「validated」:dict 含 validated 键则看 validated.ok,否则非空即视为有资产。"""
if isinstance(value, dict):
if "validated" in value:
return bool((value.get("validated") or {}).get("ok"))
return bool(value)
return bool(value)
def asset_role_satisfied(role: str, assets: Any) -> bool:
"""G8:role 是否被 validated 资产满足。
assets 可为:
- 单个资产 dict:{"role": "counts", "validated": {"ok": True}};
- 角色映射 dict:{"counts": {"validated": {"ok": True}}, ...};
- list/set[str](角色名集合)。
"""
if not role:
return False
if isinstance(assets, dict):
if "role" in assets:
# 单个资产 dict
if assets.get("role") != role:
return False
return bool((assets.get("validated") or {}).get("ok"))
# 角色映射 dict
return _asset_is_validated(assets.get(role))
if isinstance(assets, (list, tuple, set, frozenset)):
return role in set(assets)
return False
def downstream_entry_satisfied(stage: str, assets: Any) -> tuple[bool, list[str]]:
"""G7:阶段输入角色是否全部由 validated 资产满足(可直接进入,不补跑上游)。
返回 (是否可直入, 仍需补跑的上游阶段列表)。无角色声明 → (False, [])(保守)。
"""
roles = stage_input_roles(stage)
if not roles:
return False, []
missing_upstream: list[str] = []
for role in roles:
if not asset_role_satisfied(role, assets):
missing_upstream.extend(ROLE_UPSTREAM_STAGES.get(role, []))
return (not missing_upstream), missing_upstream
def upstream_stages_for_roles(satisfied_roles: Iterable[str]) -> set[str]:
"""已满足角色对应的上游阶段集合(这些上游可跳过,不补跑)。"""
out: set[str] = set()
for role in satisfied_roles:
out.update(ROLE_UPSTREAM_STAGES.get(role, []))
return out
@@ -0,0 +1,46 @@
"""app/services/bioflow/task_context.py —— 任务绑定上下文(G11)。
返回 session_id + job_id + plan_revision + run_id + current_stage,供 MCP
get_task_context 与 Skill 编排绑定「本次运行」到具体任务与阶段。
服务层纪律:纯 dict/stdlib,无 pydantic。
"""
from __future__ import annotations
from typing import Any
# 阶段终态(不视为「当前阶段」)
_TERMINAL = ("done", "skipped", "archived")
def current_stage(record: Any) -> str | None:
"""当前阶段:阶段链中第一个未达终态的阶段;全部终态则取末阶段。"""
chain = getattr(record, "stage_chain", None) or []
if not chain:
return None
for stage in chain:
st = (record._stage_states.get(stage, {}) or {}).get("status")
if st not in _TERMINAL:
return stage
return chain[-1]
def get_task_context(record: Any | None = None) -> dict[str, Any]:
"""G11:任务绑定上下文(session_id/job_id/plan_revision/run_id/当前阶段)。
record 缺省返回空绑定(5 字段齐全);传 JobRecord 时返回实际绑定。
"""
if record is None:
return {
"session_id": "",
"job_id": "",
"plan_revision": "1",
"run_id": "",
"current_stage": None,
}
return {
"session_id": getattr(record, "session_id", None) or "",
"job_id": getattr(record, "id", None) or "",
"plan_revision": getattr(record, "plan_revision", None) or "1",
"run_id": getattr(record, "run_id", None) or getattr(record, "id", None) or "",
"current_stage": current_stage(record),
}
@@ -0,0 +1,134 @@
"""app/services/bioflow/versioning.py —— 版本化修改(G16/G17/G18/G19)。
- G16 版本化身份:new_revision(based_on_revision) 返回 revision/based_on_revision/
asset_id/action_id;make_action_id() 生成唯一 action id。
(plan_revision/run_id 已在 JobRecord,见 runner.py + task_context.py)
- G17 影响范围:affected_stages(stage_chain, changed_stage) 返回改参数后受影响的
下游阶段(依赖图传递闭包,按 stage_chain 序,只重跑受影响节点)。
- G18 历史不覆盖:RevisionStore(root) 按 revision 物理隔离存 key→content,
新 revision 不覆盖旧 revision。
- G19 防重复:ActionDedup().try_start(action_id) 幂等(首次 True,重复 False)。
服务层纪律:纯 dataclass/dict/stdlib,无 pydantic。
"""
from __future__ import annotations
import uuid
from pathlib import Path
from typing import Any
# ---- G16 版本化身份 ----
def _next_revision(base: str | None) -> str:
"""无 base → r1;base 形如 rN 或 N → r(N+1)。"""
if not base:
return "r1"
try:
n = int(str(base).lstrip("r") or 0) + 1
except ValueError:
n = 1
return f"r{n}"
def new_plan_revision(base: str | None = None) -> str:
"""新 plan_revision(G11/G16 共用)。"""
return _next_revision(base)
def make_action_id() -> str:
"""G16:唯一 action id。"""
return uuid.uuid4().hex[:16]
def new_run_id() -> str:
"""G16:新执行标识。"""
return uuid.uuid4().hex[:16]
def new_asset_id() -> str:
"""G16:新资产标识。"""
return uuid.uuid4().hex[:16]
def new_revision(based_on_revision: str | None = None) -> dict[str, Any]:
"""G16:产生一次版本化变更身份。
返回 {"revision": str, "based_on_revision": str|None,
"asset_id": str, "action_id": str}。
"""
return {
"revision": _next_revision(based_on_revision),
"based_on_revision": based_on_revision,
"asset_id": new_asset_id(),
"action_id": make_action_id(),
}
# ---- G17 影响范围 ----
def _downstream_closure(changed_stage: str) -> set[str]:
"""依赖图反向传递闭包:changed_stage 的所有下游阶段。"""
from .catalog import STAGE_DEPENDENCIES_SNAPSHOT
rev: dict[str, list[str]] = {}
for s, deps in STAGE_DEPENDENCIES_SNAPSHOT.items():
for d in deps:
rev.setdefault(d, []).append(s)
closure: set[str] = set()
stack: list[str] = list(rev.get(changed_stage, []))
while stack:
s = stack.pop()
if s in closure:
continue
closure.add(s)
stack.extend(rev.get(s, []))
return closure
def affected_stages(stage_chain: list[str], changed_stage: str) -> list[str]:
"""G17:改 changed_stage 参数后受影响的下游阶段列表(按 stage_chain 序)。"""
closure = _downstream_closure(changed_stage)
return [s for s in stage_chain if s in closure]
# ---- G18 历史不覆盖 ----
class RevisionStore:
"""按 revision 物理隔离存储(G18):新 revision 不覆盖旧 revision。"""
def __init__(self, root: str | Path) -> None:
self.root = Path(root)
def _path(self, revision: str, key: str) -> Path:
return self.root / revision / Path(key).name
def write(self, revision: str, key: str, content: str) -> Path:
"""写 key→content 到 revision 独立路径;返回写入路径(不覆盖旧 revision)。"""
p = self._path(revision, key)
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(content, encoding="utf-8")
return p
def read(self, revision: str, key: str) -> str | None:
"""读 revision 下 key 的内容;不存在返回 None。"""
p = self._path(revision, key)
if not p.exists():
return None
return p.read_text(encoding="utf-8")
# ---- G19 防重复 ----
class ActionDedup:
"""action_id 幂等(G19):同一 action_id 重复提交不重复启动。"""
def __init__(self) -> None:
self._started: set[str] = set()
def try_start(self, action_id: str) -> bool:
"""首次 True,重复 False(幂等,不重复启动)。"""
if action_id in self._started:
return False
self._started.add(action_id)
return True
+6
View File
@@ -0,0 +1,6 @@
"""bioflowctl —— 生信业务逻辑 CLI(唯一事实源)。
无状态、幂等、直连 SQLite + 文件系统。后续 MCP 薄适配与 Skill 接入建立在本 CLI 之上。
用法:`python -m bioflowctl <子命令> ...`(cwd = bioflow_service/)。
"""
+176
View File
@@ -0,0 +1,176 @@
"""bioflowctl/__main__.py —— CLI 入口。
用法(cwd = bioflow_service/):
python -m bioflowctl source fetch <url|oss|lan> [--job-id JOB_ID] [--cred-user U] [--cred-password P] [--cred-port PORT]
python -m bioflowctl file inspect <path>
python -m bioflowctl file transform <src> --to <fmt> [--column-map JSON] [--dest PATH]
python -m bioflowctl asset register --role <role> <path>
python -m bioflowctl asset validate --role <role> <path>
python -m bioflowctl report register --path <p>
python -m bioflowctl archive <job_id> --name <n> [--description <d>]
python -m bioflowctl archive reuse --name <n>
输出:JSON(stdout);ok=False 时退出码 1。
"""
from __future__ import annotations
import argparse
import asyncio
import json
import sys
from typing import Any
from . import commands
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="bioflowctl",
description="PlantAI 生信业务逻辑 CLI(唯一事实源;无状态/幂等/直连 SQLite)",
)
sub = parser.add_subparsers(dest="group", required=True)
# source fetch
g = sub.add_parser("source", help="源文件拉取(web/oss/lan)")
ss = g.add_subparsers(dest="action", required=True)
p = ss.add_parser("fetch", help="拉取源文件到任务 input 或 _incoming")
p.add_argument("source", help="http/https URL、oss://bucket/key 或局域网路径")
p.add_argument("--job-id", dest="job_id", default=None, help="目标任务 id(缺省落 _incoming)")
p.add_argument("--cred-user", dest="cred_user", default=None, help="局域网 scp 用户名")
p.add_argument("--cred-password", dest="cred_password", default=None, help="局域网 scp 密码(仅内存)")
p.add_argument("--cred-port", dest="cred_port", default=None, help="局域网 scp 端口")
# file inspect / transform
g = sub.add_parser("file", help="文件探测 / 转换")
ss = g.add_subparsers(dest="action", required=True)
p = ss.add_parser("inspect", help="探测文件格式/预览/列结构")
p.add_argument("path")
p = ss.add_parser("transform", help="非标准文件转 fasta/gff3/tsv/samples.yaml")
p.add_argument("src")
p.add_argument("--to", dest="target", required=True, help="目标格式")
p.add_argument("--column-map", dest="column_map", default=None, help='列映射 JSON(如 {"name":"基因名","seq":"序列"})')
p.add_argument("--dest", dest="dest", default=None, help="输出路径(缺省自动命名)")
# asset register / validate
g = sub.add_parser("asset", help="角色化资产登记 / 审核")
ss = g.add_subparsers(dest="action", required=True)
p = ss.add_parser("register", help="角色化登记 + 原子发布 + 同名冲突检测")
p.add_argument("--role", dest="role", required=True, help="资产角色(如 pep.fa / gff3 / samples.yaml)")
p.add_argument("path")
p = ss.add_parser("validate", help="按角色内容审核")
p.add_argument("--role", dest="role", required=True)
p.add_argument("path")
# report register
g = sub.add_parser("report", help="报告登记")
ss = g.add_subparsers(dest="action", required=True)
p = ss.add_parser("register", help="登记报告(拒绝空路径/空文件)")
p.add_argument("--path", dest="path", required=True)
# archive / archive reuse
g = sub.add_parser("archive", help="任务存档 / 复用")
g.add_argument("target", help="任务 id,或字面 'reuse'")
g.add_argument("--name", dest="name", required=True, help="归档名")
g.add_argument("--description", dest="description", default="")
# check readiness
g = sub.add_parser("check", help="就绪检查")
ss = g.add_subparsers(dest="action", required=True)
ss.add_parser("readiness", help="四象限就绪检查(环境/参考/磁盘/预算)")
# change propose / apply
g = sub.add_parser("change", help="版本化变更")
ss = g.add_subparsers(dest="action", required=True)
p = ss.add_parser("propose", help="计算参数变更的影响范围")
p.add_argument("--stage", dest="stage", required=True)
p = ss.add_parser("apply", help="幂等应用变更")
p.add_argument("--action-id", dest="action_id", required=True)
# plan revision list
g = sub.add_parser("plan", help="规划 revision")
ss = g.add_subparsers(dest="action", required=True)
p = ss.add_parser("revision", help="列出历史 revision")
p.add_argument("--root", dest="root", required=True)
# run node-execute / stop / resume
g = sub.add_parser("run", help="执行控制")
ss = g.add_subparsers(dest="action", required=True)
p = ss.add_parser("node-execute", help="只重跑受影响节点")
p.add_argument("--stage", dest="stage", required=True)
p = ss.add_parser("stop", help="终止实际执行")
p.add_argument("--run-id", dest="run_id", required=True)
p.add_argument("--owner", dest="owner", required=True)
p = ss.add_parser("resume", help="恢复执行")
p.add_argument("--run-id", dest="run_id", required=True)
p.add_argument("--owner", dest="owner", required=True)
# delivery validate / publish
g = sub.add_parser("delivery", help="交付清单")
ss = g.add_subparsers(dest="action", required=True)
p = ss.add_parser("validate", help="校验交付清单完整性")
p.add_argument("--manifest", dest="manifest", required=True, help="清单 JSON 文件路径")
p = ss.add_parser("publish", help="发布交付清单")
p.add_argument("--manifest", dest="manifest", required=True, help="清单 JSON 文件路径")
return parser
def _dispatch(args: argparse.Namespace) -> dict[str, Any]:
if args.group == "source" and args.action == "fetch":
creds: dict[str, str] = {}
if args.cred_user:
creds["user"] = args.cred_user
if args.cred_password:
creds["password"] = args.cred_password
if args.cred_port:
creds["port"] = args.cred_port
return asyncio.run(commands.cmd_source_fetch(
args.source, job_id=args.job_id, credentials=creds or None))
if args.group == "file" and args.action == "inspect":
return commands.cmd_file_inspect(args.path)
if args.group == "file" and args.action == "transform":
column_map = None
if args.column_map:
column_map = json.loads(args.column_map)
return asyncio.run(commands.cmd_file_transform(
args.src, args.target, column_map=column_map, dest=args.dest))
if args.group == "asset" and args.action == "register":
return commands.register_asset(args.role, args.path)
if args.group == "asset" and args.action == "validate":
return commands.validate_asset(args.role, args.path)
if args.group == "report" and args.action == "register":
return commands.cmd_report_register(args.path)
if args.group == "archive" and args.target == "reuse":
return asyncio.run(commands.cmd_archive_reuse(args.name))
if args.group == "archive":
return asyncio.run(commands.cmd_archive(args.target, args.name, args.description))
if args.group == "check" and args.action == "readiness":
return commands.cmd_check_readiness()
if args.group == "change" and args.action == "propose":
return commands.cmd_change_propose(args.stage)
if args.group == "change" and args.action == "apply":
return commands.cmd_change_apply(args.action_id)
if args.group == "plan" and args.action == "revision":
return commands.cmd_plan_revision_list(args.root)
if args.group == "run" and args.action == "node-execute":
return commands.cmd_run_node_execute(args.stage)
if args.group == "run" and args.action == "stop":
return commands.cmd_run_stop(args.run_id, args.owner)
if args.group == "run" and args.action == "resume":
return commands.cmd_run_resume(args.run_id, args.owner)
if args.group == "delivery" and args.action == "validate":
return commands.cmd_delivery_validate(args.manifest)
if args.group == "delivery" and args.action == "publish":
return commands.cmd_delivery_publish(args.manifest)
return {"ok": False, "code": "USAGE", "message": "未知子命令组合"}
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
result = _dispatch(args)
print(json.dumps(result, ensure_ascii=False, indent=2))
return 0 if result.get("ok") else 1
if __name__ == "__main__":
sys.exit(main())
+439
View File
@@ -0,0 +1,439 @@
"""bioflowctl/commands.py —— 子命令实现(无状态、幂等、直连 SQLite + 文件系统)。
每个命令返回 dict({"ok": True/False, ...}),由 __main__ 序列化为 JSON 并定退出码。
复用服务层单一事实源(不复刻逻辑,只做直连编排):
- source_fetcher.fetch_source —— web/oss/lan 流式拉取(G5 已流式)
- file_transform.inspect_file / transform_file / validate_format
- archive.archive_from_job / effective_stage_params / normalize_archive_name
- reports.locate_report / extract_figure_refs / validate_report_path
- store(aiosqlite 直连 bioflow.db)
"""
from __future__ import annotations
import os
import shutil
import uuid
from pathlib import Path
from typing import Any
from app.config import get_settings
from app.db import Database
from app.services.bioflow import store
from app.services.bioflow.archive import (
archive_from_job,
effective_stage_params,
normalize_archive_name,
)
from app.services.bioflow.errors import error_result
from app.services.bioflow.file_transform import (
TransformError,
inspect_file,
transform_file,
validate_format,
)
from app.services.bioflow.reports import (
extract_figure_refs,
locate_report,
validate_report_path,
)
from app.services.bioflow.source_fetcher import FetchError, fetch_source
_COPY_CHUNK = 1024 * 1024 # 1MB 流式复制缓冲(G5)
def _err(code: str, message: str, *, evidence_refs: list[str] | None = None) -> dict[str, Any]:
"""G13:失败返回统一信封(code 顶层 + error 结构化对象)。"""
return error_result(code, message, evidence_refs=evidence_refs)
# ---------------------------------------------------------------------------
# source fetch
# ---------------------------------------------------------------------------
async def cmd_source_fetch(
source: str,
*,
job_id: str | None = None,
credentials: dict[str, Any] | None = None,
fetch_fn: Any | None = None,
) -> dict[str, Any]:
"""拉取源文件(web/oss/lan);job_id 指定时落任务 input,否则落 _incoming。
fetch_fn 供 MCP 薄适配注入(保持模块级 fetch_source 可 monkeypatch)。
"""
settings = get_settings()
if job_id:
dest_dir: Path = Path(settings.jobs_dir_abs) / job_id / "input"
else:
dest_dir = Path(settings.jobs_dir_abs) / "_incoming" / uuid.uuid4().hex[:8]
fetch = fetch_fn or fetch_source
try:
result = await fetch(source, dest_dir, credentials=credentials)
except FetchError as exc:
return _err(exc.code, exc.message)
result["job_id"] = job_id
return result
# ---------------------------------------------------------------------------
# file inspect / transform
# ---------------------------------------------------------------------------
def cmd_file_inspect(path: str) -> dict[str, Any]:
result = inspect_file(path)
if not result.get("ok"):
return _err(result.get("code") or "READ_ERROR", result.get("message") or "")
return result
async def cmd_file_transform(
src: str,
target: str,
*,
column_map: dict[str, str] | None = None,
dest: str | None = None,
llm: Any | None = None,
) -> dict[str, Any]:
"""非标准文件转换(LLM 不注入则启发式 + 显式 column_map;可测、确定性)。"""
try:
return await transform_file(
src, target, column_map=column_map, dest_path=dest, llm=llm,
)
except TransformError as exc:
evidence = [str(exc.details)] if exc.details else None
return _err(exc.code, exc.message, evidence_refs=evidence)
# ---------------------------------------------------------------------------
# asset register / validate(G2 原子发布 + 同名冲突;G8 角色化资产基础)
# ---------------------------------------------------------------------------
def _sanitize_role(role: str) -> str:
r = (role or "").strip().strip("/")
if not r or ".." in r or "/" in r or "\\" in r:
return ""
return r
def _safe_name(name: str) -> str:
return Path(name or "asset").name or "asset"
def _assets_root(assets_root: str | Path | None) -> Path:
if assets_root is not None:
return Path(assets_root)
return Path(get_settings().data_root_abs) / "assets"
def register_asset(
role: str, path: str, assets_root: str | Path | None = None
) -> dict[str, Any]:
"""角色化登记 + 原子发布 + 同名冲突检测(G2/G8 基础)。
- 流式复制到角色目录(G5:shutil.copyfileobj 1MB 缓冲,不整块进内存)。
- 临时文件 os.link 原子发布,目标已存在时显式报错,绝不静默覆盖。
"""
role = _sanitize_role(role)
if not role:
return _err("BAD_ROLE", "角色名非法(须非空、不含路径分隔符/..)")
src = Path(path).expanduser()
if not src.is_file():
return _err("NOT_FOUND", f"源文件不存在: {path}")
dest_dir = _assets_root(assets_root) / role
dest_dir.mkdir(parents=True, exist_ok=True)
name = _safe_name(src.name)
dest = dest_dir / name
if dest.exists() or dest.is_symlink():
return _err("ASSET_EXISTS", f"同名资产已存在,拒绝覆盖: {role}/{name}")
tmp = dest_dir / f".tmp-{uuid.uuid4().hex[:8]}-{name}"
try:
with src.open("rb") as fin, tmp.open("wb") as fout:
shutil.copyfileobj(fin, fout, length=_COPY_CHUNK)
# 原子发布:os.link 目标已存在时抛 FileExistsError(不覆盖)
os.link(tmp, dest)
except FileExistsError:
return _err("ASSET_EXISTS", f"同名资产已存在,拒绝覆盖: {role}/{name}")
except OSError as exc:
if dest.exists() or dest.is_symlink():
return _err("ASSET_EXISTS", f"同名资产已存在,拒绝覆盖: {role}/{name}")
return _err("PUBLISH_FAILED", f"资产发布失败: {exc}")
finally:
tmp.unlink(missing_ok=True)
return {"ok": True, "role": role, "name": name, "path": str(dest),
"size": dest.stat().st_size}
def _role_to_format(role: str) -> str | None:
"""角色名 → 期望格式(校验器键)。无法推断返回 None(走内容嗅探)。"""
r = (role or "").lower()
if "samples" in r:
return "samples.yaml"
if "gff" in r or "gtf" in r:
return "gff3"
if r.endswith(".tsv") or ".tsv" in r:
return "tsv"
if any(k in r for k in ("fasta", ".fa", ".faa", ".fna", "pep", "cds")):
return "fasta"
return None
def validate_asset(role: str, path: str) -> dict[str, Any]:
"""内容审核:按角色推断期望格式并跑格式校验器。"""
p = Path(path).expanduser()
if not p.is_file():
return _err("NOT_FOUND", f"文件不存在: {path}")
fmt = _role_to_format(role)
if fmt is None:
info = inspect_file(p)
guessed = info.get("format") if info.get("ok") else None
fmt = guessed if guessed in ("fasta", "gff3", "tsv") else None
if fmt is None:
return _err("BAD_ROLE",
f"无法从角色 {role!r} 推断期望格式(可选 fasta/gff3/tsv/samples.yaml)")
validated = validate_format(p, fmt)
if not validated.get("ok"):
return _err("VALIDATION_FAILED", validated.get("reason") or f"{fmt} 校验失败",
evidence_refs=[str(validated)])
return {"ok": True, "role": role, "format": fmt, "validated": validated}
# ---------------------------------------------------------------------------
# report register(G3:拒绝空路径/空文件)
# ---------------------------------------------------------------------------
def cmd_report_register(path: str) -> dict[str, Any]:
ok, err = validate_report_path(path)
if not ok:
return _err("EMPTY_REPORT", err)
p = Path(path)
figures = extract_figure_refs(p)
return {"ok": True, "path": str(p), "size": p.stat().st_size, "figures": figures}
# ---------------------------------------------------------------------------
# archive / archive reuse(G1 产物清单 + G6 生效参数)
# ---------------------------------------------------------------------------
async def cmd_archive(
job_id: str,
name: str,
description: str = "",
*,
db: Any | None = None,
db_path: str | Path | None = None,
jobs_dir: str | Path | None = None,
archives_dir: str | Path | None = None,
include_intermediate: bool = False,
archive_fn: Any | None = None,
record: Any | None = None,
) -> dict[str, Any]:
"""按 manifest 必需产物清单存档(G1)+ 保存各阶段实际生效参数(G6)。
- db:已连接 Database(MCP 薄适配注入 ctx.db;缺省自建连接)。
- record:内存 JobRecord(MCP 传入;缺省走 DB 查询)。
- archive_fn:注入 archive_from_job(供测试 monkeypatch)。
"""
settings = get_settings()
own_db = db is None
if db is None:
db = Database(db_path)
await db.connect()
try:
archive_name = normalize_archive_name(name)
existing = await store.find_archive_by_name(db, archive_name)
if existing is not None:
return _err("ARCHIVE_EXISTS", f"归档名已存在: {archive_name}")
if record is not None:
job = {
"id": record.id, "session_id": record.session_id,
"title": record.title, "stage_chain": record.stage_chain,
"params": record.params,
}
stage_states = getattr(record, "_stage_states", {}) or {}
job_dir = record.job_dir
else:
job = await store.get_job(db, job_id)
if job is None:
return _err("NOT_FOUND", f"任务不存在: {job_id}")
stages = await store.list_stages(db, job_id)
stage_states = {s["stage"]: {"params": s["params"]} for s in stages}
job_dir = Path(jobs_dir or settings.jobs_dir_abs) / job_id
# G6:各阶段实际生效参数(非创建时 params)
effective_params = effective_stage_params(job["stage_chain"], stage_states)
archives_root = archives_dir or settings.archives_dir_abs
report_path = locate_report(job_dir / "workspace")
afn = archive_fn or archive_from_job
try:
result = await afn(
{"id": job_id, "session_id": job["session_id"], "title": job["title"],
"stage_chain": job["stage_chain"], "params": effective_params},
job_dir,
archives_root,
name=archive_name,
description=description,
include_intermediate=bool(include_intermediate),
report_artifact=str(report_path) if report_path else None,
)
except Exception as exc: # noqa: BLE001 - 存档失败统一映射 ARCHIVE_FAILED
return _err("ARCHIVE_FAILED", f"存档失败: {type(exc).__name__}: {exc}")
await store.insert_archive(db, {
"id": result.id, "job_id": job_id, "name": archive_name,
"description": description, "stage_chain": job["stage_chain"],
"params": effective_params,
"report_artifact": result.report_artifact,
"disk_path": result.disk_path, "size_bytes": result.size_bytes,
})
return {"ok": True, "data": {
"id": result.id, "job_id": job_id, "name": archive_name,
"description": description, "disk_path": result.disk_path,
"size_bytes": result.size_bytes, "report_artifact": result.report_artifact,
}}
finally:
if own_db:
await db.close()
async def cmd_archive_reuse(
name: str, *, db: Any | None = None, db_path: str | Path | None = None
) -> dict[str, Any]:
"""按名称匹配存档,返回阶段链 + 各阶段实际生效参数(G6 读侧)。"""
own_db = db is None
if db is None:
db = Database(db_path)
await db.connect()
try:
arch = await store.find_archive_by_name(db, (name or "").strip())
if arch is None:
return _err("ARCHIVE_NOT_FOUND", f"未找到匹配存档: {name}")
return {"ok": True, "data": {
"id": arch["id"], "name": arch["name"],
"stage_chain": arch["stage_chain"],
"params": arch["params"], "description": arch["description"],
}}
finally:
if own_db:
await db.close()
# ---------------------------------------------------------------------------
# check readiness(G10:四象限就绪检查)
# ---------------------------------------------------------------------------
def cmd_check_readiness(
*, assets_root: str | Path | None = None, stages: list[str] | None = None
) -> dict[str, Any]:
"""G10:四象限就绪检查;缺项 ready=False + 结构化错误阻断。"""
from app.services.bioflow.readiness import check_readiness
root = assets_root or get_settings().data_root_abs
return check_readiness(assets_root=root, stages=stages or [])
# ---------------------------------------------------------------------------
# change propose / apply / plan revision list(G17/G18/G19)
# ---------------------------------------------------------------------------
def cmd_change_propose(
changed_stage: str, stage_chain: list[str] | None = None
) -> dict[str, Any]:
"""G17:给定参数变更,经依赖图计算受影响的下游阶段列表。"""
from app.services.bioflow.versioning import affected_stages
chain = list(stage_chain or [])
affected = affected_stages(chain + [changed_stage] if changed_stage not in chain else chain,
changed_stage)
return {"ok": True, "data": {"changed_stage": changed_stage, "affected": affected}}
def cmd_change_apply(action_id: str, *, dedup: Any = None) -> dict[str, Any]:
"""G19:change apply 幂等——同一 action_id 重复提交不重复启动。"""
from app.services.bioflow.versioning import ActionDedup
d = dedup or ActionDedup()
started = d.try_start(action_id)
return {"ok": True, "started": started, "already_applied": not started,
"action_id": action_id}
def cmd_plan_revision_list(root: str | Path, *, store: Any = None) -> dict[str, Any]:
"""G18:按 revision 列历史(新 revision 不覆盖旧 revision)。"""
from app.services.bioflow.versioning import RevisionStore
st = store or RevisionStore(root)
revs = sorted(p.name for p in Path(root).iterdir()) if Path(root).is_dir() else []
return {"ok": True, "data": {"revisions": revs}}
# ---------------------------------------------------------------------------
# run node-execute / stop / resume(G17/G20)
# ---------------------------------------------------------------------------
def cmd_run_node_execute(
changed_stage: str, stage_chain: list[str] | None = None
) -> dict[str, Any]:
"""G17:只重跑受影响节点。"""
from app.services.bioflow.versioning import affected_stages
chain = list(stage_chain or [])
affected = affected_stages(chain + [changed_stage] if changed_stage not in chain else chain,
changed_stage)
return {"ok": True, "data": {"nodes": affected}}
def cmd_run_stop(run_id: str, owner_id: str) -> dict[str, Any]:
"""G20:用户停止终止实际执行。"""
from app.services.bioflow import run_owner
return {"ok": run_owner.stop(run_id, owner_id), "run_id": run_id, "owner_id": owner_id}
def cmd_run_resume(run_id: str, owner_id: str) -> dict[str, Any]:
"""G20:恢复不与旧进程重叠。"""
from app.services.bioflow import run_owner
return {"ok": run_owner.resume(run_id, owner_id), "run_id": run_id, "owner_id": owner_id}
# ---------------------------------------------------------------------------
# delivery validate / publish(G21)
# ---------------------------------------------------------------------------
def _load_manifest(manifest: Any) -> dict[str, Any]:
"""接受 dict 或 JSON 文件路径 → manifest dict。"""
if isinstance(manifest, dict):
return manifest
import json
return json.loads(Path(manifest).read_text(encoding="utf-8"))
def cmd_delivery_validate(manifest: Any) -> dict[str, Any]:
"""G21:校验交付清单完整性(八项)。"""
from app.services.bioflow.delivery import delivery_validate
return delivery_validate(_load_manifest(manifest))
def cmd_delivery_publish(manifest: Any) -> dict[str, Any]:
"""G21:发布交付清单(校验通过置 approved)。"""
from app.services.bioflow.delivery import publish_delivery
return publish_delivery(_load_manifest(manifest))
# ---------------------------------------------------------------------------
# plan validate(G12:读单一 schema 校验参数)
# ---------------------------------------------------------------------------
def cmd_validate_plan(stage: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
"""G12:plan validate——读单一参数 schema 校验阶段参数(未知参数/choice 越界拒绝)。"""
from app.services.bioflow.catalog import stage_schema
params = dict(params or {})
schema = stage_schema(stage)
schema_params = schema["params"]
known = {p["name"] for p in schema_params}
issues: list[str] = []
for k in params:
if k not in known:
issues.append(f"未知参数: {k}")
for p in schema_params:
name = p["name"]
if name in params and p.get("type") == "choice":
if params[name] not in p.get("choices", []):
issues.append(f"参数 {name}={params[name]!r} 不在可选 {p['choices']}")
if issues:
return _err("VALIDATION_FAILED", "; ".join(issues), evidence_refs=issues)
return {"ok": True, "data": {"stage": stage, "valid": True}}
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# bioflow_service/install_skill.sh —— 安装 bioflow-analysis skill(G12)。
#
# 用法:bash install_skill.sh [skills_dir]
# - skills_dir 缺省为仓库根 .claude/skills(Claude Code 项目级 skill 目录)
# - 复制 bioflow_service/SKILL.md → skills_dir/bioflow-analysis/SKILL.md
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
SKILL_SRC="$HERE/SKILL.md"
SKILL_NAME="bioflow-analysis"
SKILLS_DIR="${1:-$HERE/../.claude/skills}"
[ -f "$SKILL_SRC" ] || { echo "缺 SKILL.md: $SKILL_SRC" >&2; exit 2; }
DEST="$SKILLS_DIR/$SKILL_NAME"
mkdir -p "$DEST"
cp "$SKILL_SRC" "$DEST/SKILL.md"
echo "[install] $SKILL_NAME -> $DEST/SKILL.md"
echo "DONE"
@@ -0,0 +1,883 @@
"""tests/test_bioflowctl_cli.py —— Phase A CLI(bioflowctl)测试用例集。
覆盖(Phase A 验收):
- CLI 命令(bioflowctl/commands.py 子命令 + 其服务层单一事实源):
source fetch / file inspect / file transform / asset register / asset validate /
report register / archive / archive reuse
- M0 六条 gap(G1-G6)正反例。
命令 → 实现映射:
source fetch → source_fetcher.fetch_source(commands.cmd_source_fetch 薄封装)
file inspect → file_transform.inspect_file
file transform→ file_transform.transform_file
asset register→ commands.register_asset(G2 原子发布 + 同名冲突)
asset validate→ commands.validate_asset(角色→格式 → 校验器)
report register→ commands.cmd_report_register(G3 空路径拒绝)
archive → commands.cmd_archive + archive.archive_from_job(G1/G6)
archive reuse → commands.cmd_archive_reuse(G6 读生效参数)
每条用例三段式(前置 / 步骤 / 预期)写在 docstring。分层标记 -m bioflow_unit。
"""
from __future__ import annotations
import asyncio
import json
import types as _types
from pathlib import Path
import httpx as _real_httpx
import pytest
from app.db import Database
from app.services.bioflow import store
from app.services.bioflow.archive import (
archive_from_job,
effective_stage_params,
normalize_archive_name,
)
from app.services.bioflow.file_transform import (
TransformError,
inspect_file,
transform_file,
validate_format,
)
from app.services.bioflow.params import archive_param_prefill
from app.services.bioflow.reports import (
extract_figure_refs,
locate_report,
validate_report_path,
)
from app.services.bioflow.runner import (
STAGE_DONE,
STAGE_WAITING_CONFIRM,
)
from app.services.bioflow.source_fetcher import (
FetchError,
_resolve_oss_url,
check_disk_space,
fetch_lan,
fetch_source,
fetch_web,
)
from bioflowctl import commands as cli
pytestmark = pytest.mark.bioflow_unit
# ---------------------------------------------------------------------------
# 辅助
# ---------------------------------------------------------------------------
def _run(coro):
"""同步跑协程(新建 event loop;与 docgen 测试同款可靠模式)。"""
return asyncio.new_event_loop().run_until_complete(coro)
def _scenario(coro_factory):
"""单 loop 跑完整 runner 场景(runner 内部 create_task 派生任务必须同 loop)。"""
loop = asyncio.new_event_loop()
try:
return loop.run_until_complete(coro_factory())
finally:
loop.close()
async def _drain(runner, seconds: float = 5.0) -> None:
"""等待 runner 派生的后台任务全部结束。"""
while runner._tasks:
await asyncio.wait_for(asyncio.gather(*list(runner._tasks)), timeout=seconds)
await asyncio.sleep(0.01)
def _fake_httpx_stream(chunks, monkeypatch, sf) -> None:
"""注入 fake httpx:stream→aiter_bytes 逐 chunk 产出(确定性,不依赖外网)。"""
total = sum(len(c) for c in chunks)
class _Resp:
status_code = 200
headers = {"content-length": str(total)}
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
async def aiter_bytes(self, chunk_size):
for c in chunks:
yield c
class _Stream:
async def __aenter__(self):
return _Resp()
async def __aexit__(self, *a):
return False
class _Client:
def __init__(self, follow_redirects=True):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
def stream(self, method, url, timeout=None):
return _Stream()
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
AsyncClient=_Client, Timeout=_real_httpx.Timeout, HTTPError=_real_httpx.HTTPError,
))
def _make_job_dir(tmp_path, with_report=True):
job_dir = tmp_path / "job" / "j1"
(job_dir / "workspace" / "final").mkdir(parents=True)
(job_dir / "logs").mkdir(parents=True)
(job_dir / "input").mkdir(parents=True)
(job_dir / "workspace" / "final" / "Ft.SPL.final_members.id").write_text(
"gene1\ngene2\n", encoding="utf-8")
(job_dir / "logs" / "identify.log").write_text("log line", encoding="utf-8")
(job_dir / "input" / "genome.pep.fa").write_text(">x\nACGT", encoding="utf-8")
(job_dir / "events.jsonl").write_text(json.dumps({"type": "job_created"}), encoding="utf-8")
if with_report:
(job_dir / "workspace" / "final" / "Ft.SPL.gene_family_report.html").write_text(
'<html><body><img src="fig://heatmap_1"/><img src="fig://tree_2"/></body></html>',
encoding="utf-8")
return job_dir
async def _seed_archives(db_path, *archives):
"""在独立 db 文件上插入存档行(供 cmd_archive_reuse 注入 db_path)。"""
db = Database(db_path)
await db.connect()
try:
for a in archives:
await store.insert_archive(db, a)
finally:
await db.close()
def _archive_row(**over):
base = {
"id": "a1", "job_id": "j1", "name": "SPL鉴定", "description": "",
"stage_chain": ["identify", "phylogeny"],
"params": {"identify": {"cpu": "8"}},
"report_artifact": None, "disk_path": "/tmp/a1", "size_bytes": 10,
}
base.update(over)
return base
# ===========================================================================
# 1. source fetch(命令: bioflowctl source fetch → source_fetcher.fetch_source)
# ===========================================================================
def test_cli_source_fetch_web_ok(tmp_path, monkeypatch):
"""前置:fake 流式 http 响应产出 2 chunk。步骤:fetch_web。
预期:ok=True,落盘 + provenance sidecar,size 正确。"""
from app.services.bioflow import source_fetcher as sf
payload = [b">seq1\nACGT\n", b">seq2\nTGCA\n"]
_fake_httpx_stream(payload, monkeypatch, sf)
r = _run(fetch_web("http://x/data.fa", tmp_path / "out"))
assert r["ok"] is True
assert r["name"] == "data.fa"
assert r["size"] == sum(len(c) for c in payload)
assert r["kind"] == "web"
assert len(r["provenance"]["sha256"]) == 64
assert (tmp_path / "out" / "data.fa.provenance.json").is_file()
def test_cli_source_fetch_bad_scheme(tmp_path):
"""前置:ftp:// 源。步骤:fetch_source。预期:FetchError BAD_SCHEME。"""
with pytest.raises(FetchError) as ei:
_run(fetch_source("ftp://host/x.fa", tmp_path))
assert ei.value.code == "BAD_SCHEME"
def test_cli_source_fetch_empty(tmp_path):
"""前置:空源。步骤:fetch_source("")。预期:FetchError EMPTY_SOURCE。"""
with pytest.raises(FetchError) as ei:
_run(fetch_source("", tmp_path))
assert ei.value.code == "EMPTY_SOURCE"
def test_cli_source_fetch_lan_local_ok(tmp_path):
"""前置:本机存在文件。步骤:fetch_lan(局域网本机路径直读)。
预期:ok=True,kind=lan,name/size/provenance 正确。"""
f = tmp_path / "genome.fa"
f.write_text(">g1\nACGT\n")
r = _run(fetch_lan(str(f), tmp_path / "out"))
assert r["ok"] is True
assert r["kind"] == "lan"
assert r["path"] == str(f)
assert r["provenance"]["sha256"]
def test_cli_source_fetch_lan_missing(tmp_path):
"""前置:路径不存在。步骤:fetch_lan。预期:FetchError PATH_NOT_EXIST。"""
with pytest.raises(FetchError) as ei:
_run(fetch_lan("/no/such/path.fa", tmp_path))
assert ei.value.code == "PATH_NOT_EXIST"
def test_cli_source_fetch_lan_dir(tmp_path):
"""前置:目录。步骤:fetch_lan。预期:FetchError IS_DIR(仅单文件)。"""
d = tmp_path / "adir"
d.mkdir()
with pytest.raises(FetchError) as ei:
_run(fetch_lan(str(d), tmp_path / "out"))
assert ei.value.code == "IS_DIR"
def test_cli_source_fetch_oss_unconfigured():
"""前置:无 BIOFLOW_OSS_ENDPOINT。步骤:_resolve_oss_url(oss://)。
预期:FetchError OSS_UNCONFIGURED。"""
with pytest.raises(FetchError) as ei:
_resolve_oss_url("oss://bucket/key.fa")
assert ei.value.code == "OSS_UNCONFIGURED"
def test_cli_source_fetch_disk_full(tmp_path):
"""前置:磁盘余量不足(1EB)。步骤:check_disk_space。预期:ok=False。"""
ok, free = check_disk_space(tmp_path, 10 ** 18)
assert ok is False
assert free < 10 ** 18
# ===========================================================================
# 2. file inspect(命令: bioflowctl file inspect → file_transform.inspect_file)
# ===========================================================================
def test_cli_file_inspect_fasta(tmp_path):
"""前置:fasta 文件。步骤:inspect_file。预期:ok,format=fasta,size>0。"""
f = tmp_path / "proteins.fa"
f.write_text(">SPL1\nMKTA\n>SPL2\nMKTB\n")
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "fasta"
assert r["size"] > 0
def test_cli_file_inspect_not_found(tmp_path):
"""前置:文件不存在。步骤:inspect_file。预期:ok=False,code=NOT_FOUND。"""
r = inspect_file(tmp_path / "nope.fa")
assert r["ok"] is False
assert r["code"] == "NOT_FOUND"
def test_cli_file_inspect_tsv_columns(tmp_path):
"""前置:tsv 文件。步骤:inspect_file。预期:format=tsv,列名提取正确。"""
f = tmp_path / "samples.tsv"
f.write_text("run\tfastq_1\tfastq_2\nSRR1\ta_1.fq\ta_2.fq\n")
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "tsv"
assert r["columns"] == ["run", "fastq_1", "fastq_2"]
def test_cli_file_inspect_garbled(tmp_path):
"""前置:含异常字节的文件。步骤:inspect_file。预期:garbled 比例 > 阈值。"""
f = tmp_path / "messy.txt"
f.write_bytes(b"\x00\x01\x02\x03\xff\xfe garbage \x00\x00")
r = inspect_file(f)
assert r["ok"] is True
assert r["garbled"] > 0.3
# ===========================================================================
# 3. file transform(命令: bioflowctl file transform → file_transform.transform_file)
# ===========================================================================
def test_cli_file_transform_csv_to_fasta(tmp_path):
"""前置:csv(gene,seq)。步骤:transform_file(fasta, 显式 column_map)。
预期:ok,format=fasta,records=2。"""
src = tmp_path / "genes.csv"
src.write_text("gene,seq\nSPL1,MKTA\nSPL2,MKTB\n", encoding="utf-8")
r = _run(transform_file(src, "fasta", column_map={"name": "gene", "seq": "seq"}))
assert r["ok"] is True
assert r["format"] == "fasta"
assert r["records"] == 2
def test_cli_file_transform_tsv_to_samples_yaml(tmp_path):
"""前置:tsv(run,r1,r2)。步骤:transform_file(samples.yaml)。
预期:ok,format=samples.yaml,records=2。"""
src = tmp_path / "samples.tsv"
src.write_text("run\tr1\tr2\nSRR1\ta_1.fq\ta_2.fq\nSRR2\tb_1.fq\tb_2.fq\n", encoding="utf-8")
r = _run(transform_file(src, "samples.yaml", column_map={"name": "run", "r1": "r1", "r2": "r2"}))
assert r["ok"] is True
assert r["format"] == "samples.yaml"
assert r["records"] == 2
def test_cli_file_transform_gff3_unsupported(tmp_path):
"""前置:tsv 源。步骤:transform_file(gff3)。预期:TransformError UNSUPPORTED。"""
src = tmp_path / "a.tsv"
src.write_text("a\tb\tc\n1\t2\t3\n", encoding="utf-8")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "gff3"))
assert ei.value.code == "UNSUPPORTED"
def test_cli_file_transform_not_found(tmp_path):
"""前置:源不存在。步骤:transform_file。预期:TransformError NOT_FOUND。"""
with pytest.raises(TransformError) as ei:
_run(transform_file(tmp_path / "nope.csv", "fasta"))
assert ei.value.code == "NOT_FOUND"
def test_cli_file_transform_bad_format(tmp_path):
"""前置:csv 源。步骤:transform_file(bed)。预期:TransformError BAD_FORMAT。"""
src = tmp_path / "a.csv"
src.write_text("a,b\n1,2\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "bed"))
assert ei.value.code == "BAD_FORMAT"
def test_cli_file_transform_empty_table(tmp_path):
"""前置:空文件。步骤:transform_file(fasta)。预期:TransformError EMPTY_TABLE。"""
src = tmp_path / "empty.csv"
src.write_text("", encoding="utf-8")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta"))
assert ei.value.code == "EMPTY_TABLE"
def test_cli_file_transform_missing_columns(tmp_path):
"""前置:表头无 name/seq 列且无 LLM。步骤:transform_file(fasta)。
预期:TransformError MISSING_COLUMNS。"""
src = tmp_path / "x.csv"
src.write_text("foo,bar\n1,2\n", encoding="utf-8")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta"))
assert ei.value.code == "MISSING_COLUMNS"
def test_cli_file_transform_validation_failed(tmp_path):
"""前置:seq 列全空(写不出有效 fasta)。步骤:transform_file(fasta)。
预期:TransformError VALIDATION_FAILED(产物删除,供追问)。"""
src = tmp_path / "genes.csv"
src.write_text("gene,seq\nSPL1,\nSPL2,\n", encoding="utf-8")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta", column_map={"name": "gene", "seq": "seq"}))
assert ei.value.code == "VALIDATION_FAILED"
# ===========================================================================
# 4. asset register(命令: bioflowctl asset register → commands.register_asset)
# ===========================================================================
def test_cli_asset_register_ok(tmp_path):
"""前置:本机文件 + 合法角色。步骤:register_asset(role, path, assets_root)。
预期:ok=True,落角色目录,role/name/size 正确。"""
f = tmp_path / "genome.fa"
f.write_text(">g1\nACGT\n")
assets = tmp_path / "assets"
r = cli.register_asset("pep.fa", str(f), assets_root=assets)
assert r["ok"] is True
assert r["role"] == "pep.fa"
assert r["name"] == "genome.fa"
assert (assets / "pep.fa" / "genome.fa").read_text() == ">g1\nACGT\n"
def test_cli_asset_register_missing_source(tmp_path):
"""前置:源不存在。步骤:register_asset。预期:ok=False,code=NOT_FOUND。"""
r = cli.register_asset("pep.fa", str(tmp_path / "nope.fa"), assets_root=tmp_path / "assets")
assert r["ok"] is False
assert r["code"] == "NOT_FOUND"
def test_cli_asset_register_dir_rejected(tmp_path):
"""前置:目录。步骤:register_asset。预期:ok=False,code=NOT_FOUND(仅文件)。"""
d = tmp_path / "dir"
d.mkdir()
r = cli.register_asset("pep.fa", str(d), assets_root=tmp_path / "assets")
assert r["ok"] is False
assert r["code"] == "NOT_FOUND"
def test_cli_asset_register_bad_role(tmp_path):
"""前置:角色名含路径分隔符或 ..。步骤:register_asset。预期:code=BAD_ROLE。"""
f = tmp_path / "x.fa"
f.write_text(">x\nACGT\n")
for bad in ("", "../pep", "a/b", "a\\b"):
r = cli.register_asset(bad, str(f), assets_root=tmp_path / "assets")
assert r["ok"] is False
assert r["code"] == "BAD_ROLE"
# ===========================================================================
# 5. asset validate(命令: bioflowctl asset validate → commands.validate_asset)
# ===========================================================================
def test_cli_asset_validate_fasta_ok(tmp_path):
"""前置:合法 fasta + 角色 pep.fa。步骤:validate_asset。预期:ok=True,format=fasta。"""
f = tmp_path / "p.fa"
f.write_text(">S1\nACGT\n")
r = cli.validate_asset("pep.fa", str(f))
assert r["ok"] is True
assert r["format"] == "fasta"
def test_cli_asset_validate_fasta_invalid(tmp_path):
"""前置:无 '>' 头行文件 + 角色 pep.fa。步骤:validate_asset。预期:ok=False。"""
f = tmp_path / "p.fa"
f.write_text("just text\n")
r = cli.validate_asset("pep.fa", str(f))
assert r["ok"] is False
def test_cli_asset_validate_samples_yaml_role(tmp_path):
"""前置:合法 samples.yaml + 角色 samples.yaml。步骤:validate_asset。
预期:ok=True,format=samples.yaml。"""
f = tmp_path / "samples.yaml"
f.write_text("samples:\n S1:\n r1: a.fq\n r2: b.fq\n")
r = cli.validate_asset("samples.yaml", str(f))
assert r["ok"] is True
assert r["format"] == "samples.yaml"
def test_cli_asset_validate_bad_role(tmp_path):
"""前置:角色无法推断格式 + txt 文件。步骤:validate_asset。预期:code=BAD_ROLE。"""
f = tmp_path / "x.txt"
f.write_text("hello\n")
r = cli.validate_asset("unknown_role", str(f))
assert r["ok"] is False
assert r["code"] == "BAD_ROLE"
def test_cli_asset_validate_missing_file(tmp_path):
"""前置:文件不存在。步骤:validate_asset。预期:code=NOT_FOUND。"""
r = cli.validate_asset("pep.fa", str(tmp_path / "nope.fa"))
assert r["ok"] is False
assert r["code"] == "NOT_FOUND"
# ===========================================================================
# 6. report register(命令: bioflowctl report register → commands.cmd_report_register)
# ===========================================================================
def test_cli_report_register_ok(tmp_path):
"""前置:报告含 fig:// 引用。步骤:cmd_report_register。预期:ok=True,
size/figures 正确。"""
f = tmp_path / "r.html"
f.write_text('<html><img src="fig://a"/></html>')
r = cli.cmd_report_register(str(f))
assert r["ok"] is True
assert r["path"] == str(f)
assert r["figures"] == ["a"]
def test_cli_report_register_missing_file(tmp_path):
"""前置:路径不存在。步骤:cmd_report_register。预期:ok=False,code=EMPTY_REPORT。"""
r = cli.cmd_report_register(str(tmp_path / "nope.html"))
assert r["ok"] is False
assert r["code"] == "EMPTY_REPORT"
# ===========================================================================
# 7. archive(命令: bioflowctl archive → commands.cmd_archive + archive.archive_from_job)
# ===========================================================================
def test_cli_archive_pack(tmp_path):
"""前置:job_dir 含 final/logs/input/events。步骤:archive_from_job。
预期:四类产物入归档;默认不含中间文件。"""
job_dir = _make_job_dir(tmp_path)
job = {"id": "j1", "title": "t", "stage_chain": ["identify"], "params": {}}
result = _run(archive_from_job(job, job_dir, tmp_path / "archives", name="SPL 鉴定"))
assert result.name == "SPL 鉴定"
dest = tmp_path / "archives" / result.id
assert (dest / "final" / "Ft.SPL.final_members.id").exists()
assert (dest / "logs" / "identify.log").exists()
assert (dest / "input" / "genome.pep.fa").exists()
assert (dest / "events.jsonl").exists()
def test_cli_archive_include_intermediate(tmp_path):
"""前置:job_dir 含中间 analysis。步骤:archive_from_job(include_intermediate=True)。
预期:中间文件入归档。"""
job_dir = _make_job_dir(tmp_path)
(job_dir / "workspace" / "identify" / "analysis").mkdir(parents=True)
(job_dir / "workspace" / "identify" / "analysis" / "domtblout").write_text("intermediate")
result = _run(archive_from_job(
{"id": "j1", "title": "t", "stage_chain": [], "params": {}},
job_dir, tmp_path / "archives", name="X", include_intermediate=True))
assert (tmp_path / "archives" / result.id / "intermediate" / "identify" / "analysis" / "domtblout").exists()
def test_cli_archive_normalize_name():
"""前置:空/带空白名。步骤:normalize_archive_name。预期:trim / 空→占位。"""
assert normalize_archive_name(" SPL 鉴定 ") == "SPL 鉴定"
assert normalize_archive_name(" ") == "未命名存档"
def test_cli_archive_cmd_not_found(tmp_path):
"""前置:空 db。步骤:cmd_archive(不存在 job)。预期:ok=False,code=NOT_FOUND。"""
r = _run(cli.cmd_archive(
"nope", "X", db_path=tmp_path / "empty.db",
jobs_dir=tmp_path / "jobs", archives_dir=tmp_path / "archives"))
assert r["ok"] is False
assert r["code"] == "NOT_FOUND"
# ===========================================================================
# 8. archive reuse(命令: bioflowctl archive reuse → commands.cmd_archive_reuse + store + params)
# ===========================================================================
def test_cli_archive_reuse_find_exact(db):
"""前置:插一条存档名 SPL鉴定。步骤:find_archive_by_name(精确)。预期:命中。"""
_run(_insert_via(db, _archive_row()))
found = _run(store.find_archive_by_name(db, "SPL鉴定"))
assert found is not None and found["id"] == "a1"
async def _insert_via(db, archive):
await store.insert_archive(db, archive)
def test_cli_archive_reuse_find_fuzzy(db):
"""前置:存档名 SPL鉴定。步骤:find_archive_by_name(模糊 SPL)。预期:命中候选。"""
_run(_insert_via(db, _archive_row()))
found = _run(store.find_archive_by_name(db, "SPL"))
assert found is not None and found["id"] == "a1"
def test_cli_archive_reuse_candidates(db):
"""前置:两条含 SPL 的存档。步骤:find_archive_candidates。预期:≥2 条。"""
_run(_insert_via(db, _archive_row()))
_run(_insert_via(db, _archive_row(id="a2", name="SPL另一条")))
cands = _run(store.find_archive_candidates(db, "SPL"))
assert len(cands) >= 2
def test_cli_archive_reuse_prefill(db):
"""前置:存档 params 含 identify。步骤:archive_param_prefill。
预期:identify 预填命中;未知阶段空。"""
_run(_insert_via(db, _archive_row()))
found = _run(store.find_archive_by_name(db, "SPL鉴定"))
assert archive_param_prefill(found, "identify") == {"cpu": "8"}
assert archive_param_prefill(found, "nope") == {}
def test_cli_archive_reuse_prefill_empty(db):
"""前置:存档 params 为空。步骤:archive_param_prefill。预期:空 dict。"""
_run(_insert_via(db, _archive_row(params={})))
found = _run(store.find_archive_by_name(db, "SPL鉴定"))
assert archive_param_prefill(found, "identify") == {}
def test_cli_archive_reuse_cmd(tmp_path):
"""前置:seed 一条存档(params 含生效参数 cpu=8)。步骤:cmd_archive_reuse。
预期:ok=True,返回 stage_chain 与生效参数。"""
db_path = tmp_path / "reuse.db"
_run(_seed_archives(db_path, _archive_row(
params={"identify": {"cpu": "8", "memory": "16G"}})))
r = _run(cli.cmd_archive_reuse("SPL鉴定", db_path=db_path))
assert r["ok"] is True
assert r["data"]["stage_chain"] == ["identify", "phylogeny"]
assert r["data"]["params"] == {"identify": {"cpu": "8", "memory": "16G"}}
def test_cli_archive_reuse_cmd_not_found(tmp_path):
"""前置:空 db。步骤:cmd_archive_reuse。预期:ok=False,code=ARCHIVE_NOT_FOUND。"""
r = _run(cli.cmd_archive_reuse("不存在", db_path=tmp_path / "empty.db"))
assert r["ok"] is False
assert r["code"] == "ARCHIVE_NOT_FOUND"
# ===========================================================================
# M0 六条 gap(G1-G6)
# ===========================================================================
# ---------------------------------------------------------------------------
# G1 归档按 manifest 必需产物复制(正例:非 final 的必需产物也归档;反例:漏归档)
# ---------------------------------------------------------------------------
def _make_job_with_manifest_required_nonfinal(tmp_path):
job_dir = tmp_path / "job" / "j1"
(job_dir / "workspace" / "final").mkdir(parents=True)
(job_dir / "workspace" / "identify" / "analysis").mkdir(parents=True)
(job_dir / "logs").mkdir(parents=True)
(job_dir / "input").mkdir(parents=True)
(job_dir / "workspace" / "final" / "Ft.SPL.final_members.id").write_text("gene1\n")
required = job_dir / "workspace" / "identify" / "analysis" / "domtblout"
required.write_text("PF00001\tSPL1\n", encoding="utf-8")
manifest = {
"manifest_version": "1", "job_id": "j1", "stage": "identify",
"status": "success", "prefix": "Ft.SPL",
"artifacts": {
"final_members": {"artifact": "final_members",
"full_path": str(job_dir / "workspace" / "final" / "Ft.SPL.final_members.id"),
"type": "table", "size": 10},
"domtblout": {"artifact": "domtblout", "full_path": str(required),
"type": "table", "size": 5},
},
"parameters": {}, "metrics": {},
}
(job_dir / "workspace" / "final" / "Ft.SPL.identify.manifest.json").write_text(
json.dumps(manifest), encoding="utf-8")
return job_dir, required
def test_g1_archive_copies_manifest_required_nonfinal_artifact(tmp_path):
"""G1 正例。前置:manifest 声明必需产物在 final/ 之外(workspace/identify/analysis)。
步骤:archive_from_job(include_intermediate=False)。
预期:该非 final 必需产物仍被复制进归档(按 manifest,而非仅 final/)。"""
job_dir, _required = _make_job_with_manifest_required_nonfinal(tmp_path)
job = {"id": "j1", "title": "t", "stage_chain": ["identify"], "params": {}}
result = _run(archive_from_job(job, job_dir, tmp_path / "archives", name="X"))
dest = tmp_path / "archives" / result.id
assert any(p.name == "domtblout" for p in dest.rglob("*"))
def test_g1_archive_does_not_miss_manifest_required_outside_jobdir(tmp_path):
"""G1 反例(漏归档)。前置:manifest 必需产物 full_path 指向 job_dir 之外(共享库文件)。
步骤:archive_from_job。预期:该必需产物被复制进归档,不遗漏。"""
job_dir = tmp_path / "job" / "j1"
(job_dir / "workspace" / "final").mkdir(parents=True)
(job_dir / "logs").mkdir(parents=True)
(job_dir / "input").mkdir(parents=True)
(job_dir / "workspace" / "final" / "Ft.SPL.final_members.id").write_text("gene1\n")
shared = tmp_path / "shared" / "Pfam-A.hmm"
shared.parent.mkdir(parents=True)
shared.write_text("HMMER3/f\nNAME Pfam-A\n", encoding="utf-8")
manifest = {
"manifest_version": "1", "job_id": "j1", "stage": "identify",
"status": "success", "prefix": "Ft.SPL",
"artifacts": {"hmm_db": {"artifact": "hmm_db", "full_path": str(shared),
"type": "db", "size": 20}},
"parameters": {}, "metrics": {},
}
(job_dir / "workspace" / "final" / "Ft.SPL.identify.manifest.json").write_text(
json.dumps(manifest), encoding="utf-8")
result = _run(archive_from_job(
{"id": "j1", "title": "t", "stage_chain": ["identify"], "params": {}},
job_dir, tmp_path / "archives", name="X"))
dest = tmp_path / "archives" / result.id
assert any(p.name == "Pfam-A.hmm" for p in dest.rglob("*"))
# ---------------------------------------------------------------------------
# G2 同名上传不覆盖、显式报错(正例:同名冲突报错不覆盖;反例:静默覆盖)
# ---------------------------------------------------------------------------
def test_g2_asset_register_same_name_errors_and_preserves(tmp_path):
"""G2 正例。前置:同角色同文件名已发布。步骤:再次 register_asset 同名。
预期:显式报错 ASSET_EXISTS,原资产不被覆盖。"""
f = tmp_path / "x.fa"
f.write_text("V1")
assets = tmp_path / "assets"
assert cli.register_asset("pep.fa", str(f), assets_root=assets)["ok"] is True
r = cli.register_asset("pep.fa", str(f), assets_root=assets)
assert r["ok"] is False
assert r["code"] == "ASSET_EXISTS"
assert (assets / "pep.fa" / "x.fa").read_text() == "V1"
def test_g2_asset_register_never_silently_overwrites(tmp_path):
"""G2 反例(静默覆盖)。前置:同名资产已存在,第二个同名但内容不同的源。
步骤:register_asset。预期:绝不静默覆盖,已发布资产内容仍为第一版。"""
d1 = tmp_path / "d1"
d2 = tmp_path / "d2"
d1.mkdir()
d2.mkdir()
(d1 / "x.fa").write_text("VERSION1")
(d2 / "x.fa").write_text("VERSION2")
assets = tmp_path / "assets"
assert cli.register_asset("pep.fa", str(d1 / "x.fa"), assets_root=assets)["ok"] is True
r = cli.register_asset("pep.fa", str(d2 / "x.fa"), assets_root=assets)
assert r["ok"] is False
assert r["code"] == "ASSET_EXISTS"
assert (assets / "pep.fa" / "x.fa").read_text() == "VERSION1"
# ---------------------------------------------------------------------------
# G3 空报告路径被拒绝(空串、空文件、None)
# ---------------------------------------------------------------------------
def test_g3_empty_report_path_rejected():
"""G3 空串。前置:无。步骤:validate_report_path("")。预期:(False, 报告路径为空)。"""
ok, err = validate_report_path("")
assert ok is False
assert err == "报告路径为空"
def test_g3_empty_file_report_path_rejected(tmp_path):
"""G3 空文件。前置:0 字节报告文件。步骤:validate_report_path(empty)。
预期:(False, 报告文件为空)。"""
empty = tmp_path / "empty.html"
empty.write_text("", encoding="utf-8")
ok, err = validate_report_path(empty)
assert ok is False
assert "报告文件为空" in err
def test_g3_none_report_path_rejected():
"""G3 None。前置:无。步骤:validate_report_path(None)。预期:(False, 报告路径为空)。"""
ok, err = validate_report_path(None)
assert ok is False
assert err == "报告路径为空"
def test_g3_cli_report_register_empty_rejected():
"""G3 CLI 层。前置:无。步骤:cmd_report_register("")。预期:ok=False,code=EMPTY_REPORT。"""
r = cli.cmd_report_register("")
assert r["ok"] is False
assert r["code"] == "EMPTY_REPORT"
# ---------------------------------------------------------------------------
# G4 partial_success 独立终态(不被当作 success)
# ---------------------------------------------------------------------------
def test_g4_partial_success_independent_terminal_state(db, events, catalog):
"""G4。前置:executor 写 status=partial_success 的 manifest 且退出码 0。
步骤:run_job(auto_continue=True)。预期:阶段不得被当作 success(done),
而是独立终态 partial_success。"""
from conftest import FakeExecutor, make_runner
executor = FakeExecutor(exit_code=0, manifest_status="partial_success")
async def scenario():
runner = make_runner(db, events, executor, catalog)
record = await runner.create_job(
"s1", "t", ["identify"], auto_continue=True, auto_start=False)
await runner.run_job(record)
await _drain(runner)
return runner, record
runner, record = _scenario(scenario)
status = runner.stage_status(record, "identify")
assert status != STAGE_DONE # 不被当作 success
assert status == "partial_success" # 独立终态
# ---------------------------------------------------------------------------
# G5 大文件流式(不整块进内存的观测方式)
# ---------------------------------------------------------------------------
def test_g5_large_download_streams_chunked(tmp_path, monkeypatch):
"""G5 正常路径(观测方式)。前置:fake 流式响应产出多 chunk。
步骤:fetch_web。预期:观测到 aiter_bytes 以 CHUNK(=1MB) 分块迭代、逐块落盘。"""
from app.services.bioflow import source_fetcher as sf
observed: dict = {}
chunks = [b"x" * 1024, b"y" * 2048, b"z" * 512]
total = sum(len(c) for c in chunks)
class _Resp:
status_code = 200
headers = {"content-length": str(total)}
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
async def aiter_bytes(self, chunk_size):
observed["chunk_size"] = chunk_size
for c in chunks:
yield c
class _Stream:
async def __aenter__(self):
return _Resp()
async def __aexit__(self, *a):
return False
class _Client:
def __init__(self, follow_redirects=True):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
def stream(self, method, url, timeout=None):
return _Stream()
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
AsyncClient=_Client, Timeout=_real_httpx.Timeout, HTTPError=_real_httpx.HTTPError,
))
r = _run(fetch_web("http://x/big.fa", tmp_path / "out"))
assert r["size"] == total
assert observed["chunk_size"] == sf.CHUNK # 1MB 分块,非整块
assert (tmp_path / "out" / "big.fa").stat().st_size == total
def test_g5_large_file_inspect_uses_preview_not_whole_read(tmp_path, monkeypatch):
"""G5 异常路径(观测方式)。前置:>4MB 大文件。步骤:inspect_file(monkeypatch
_read_text 使其被调用即失败)。预期:inspect 走 _read_text_preview(只读头)。"""
from app.services.bioflow import file_transform as ft
big = tmp_path / "big.fa"
big.write_bytes(b">s\n" + b"ACGT" * 1_200_000 + b"\n") # ≈4.8MB
def _boom(*a, **k):
raise AssertionError("大文件不应整读 _read_text")
monkeypatch.setattr(ft, "_read_text", _boom)
r = ft.inspect_file(big)
assert r["ok"] is True
assert r["format"] == "fasta"
# ---------------------------------------------------------------------------
# G6 归档复用读实际生效参数(非创建时 record.params)
# ---------------------------------------------------------------------------
def test_g6_effective_stage_params_returns_effective():
"""G6 正例。前置:stage_states 含 confirm/rerun 后的生效参数。
步骤:effective_stage_params(stage_chain, stage_states)。
预期:返回各阶段生效参数;缺失阶段为空 dict。"""
stage_chain = ["identify", "phylogeny", "missing"]
stage_states = {
"identify": {"params": {"cpu": "8", "memory": "16G"}},
"phylogeny": {"params": {"method": "ML"}},
}
eff = effective_stage_params(stage_chain, stage_states)
assert eff == {
"identify": {"cpu": "8", "memory": "16G"},
"phylogeny": {"method": "ML"},
"missing": {},
}
def test_g6_archive_reuse_cmd_reads_effective_params(tmp_path):
"""G6 正例(读侧端到端)。前置:seed 存档 params=生效参数(cpu=8)。
步骤:cmd_archive_reuse。预期:返回生效参数 cpu=8(非创建时 record.params 的 cpu=2)。"""
db_path = tmp_path / "reuse.db"
_run(_seed_archives(db_path, _archive_row(
params={"identify": {"cpu": "8", "memory": "16G"}})))
r = _run(cli.cmd_archive_reuse("SPL鉴定", db_path=db_path))
assert r["ok"] is True
assert r["data"]["params"]["identify"]["cpu"] == "8"
def test_g6_effective_params_diverge_from_creation_record(db, events, catalog):
"""G6 反例。前置:创建任务 params cpu=2(record.params),确认时覆盖 cpu=8。
步骤:confirm_stage 覆盖后观察两处 params。
预期:生效参数(_stage_states)与创建时 record.params 分叉,归档必须快照前者。"""
from conftest import FakeExecutor, make_runner
executor = FakeExecutor()
async def scenario():
runner = make_runner(db, events, executor, catalog)
record = await runner.create_job(
"s1", "t", ["identify"], params={"identify": {"cpu": "2"}},
auto_continue=False, auto_start=False)
await runner.run_job(record) # 手动模式停 waiting_confirm
await runner.confirm_stage(record, "identify", {"cpu": "8"})
await _drain(runner)
return runner, record
runner, record = _scenario(scenario)
effective = record._stage_states["identify"]["params"]
assert effective.get("cpu") == "8" # 实际生效参数
assert record.params["identify"].get("cpu") == "2" # 创建时 record.params
assert effective != record.params["identify"] # 已分叉 → 归档读 effective
@@ -0,0 +1,403 @@
"""tests/test_file_transform.py —— 非标准文件预处理(探测 + 转换 + 校验)。
分层标记 -m bioflow_unit。覆盖(S0 验收):
- inspect_file:fasta/tsv 探测、乱码检测、列结构提取。
- transform_file:xlsx→fasta(显式列映射)、csv→fasta(启发式列映射)、
samples.yaml 转换 + 校验。
- 校验失败 → TransformError VALIDATION_FAILED(追问循环依据)。
- 缺列映射且无 LLM → MISSING_COLUMNS。
"""
from __future__ import annotations
import csv
import asyncio
from pathlib import Path
import pytest
from app.services.bioflow.file_transform import (
TransformError,
_garbled_ratio,
inspect_file,
transform_file,
validate_fasta,
validate_format,
validate_gff3,
validate_samples_yaml,
validate_tsv,
)
def _run(coro):
return asyncio.new_event_loop().run_until_complete(coro)
# ---------------------------------------------------------------------------
# inspect_file 探测
# ---------------------------------------------------------------------------
def test_inspect_file_fasta(tmp_path):
f = tmp_path / "proteins.fa"
f.write_text(">SPL1\nMKTA\n>SPL2\nMKTB\n")
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "fasta"
assert r["size"] > 0
def test_inspect_file_tsv_columns(tmp_path):
f = tmp_path / "samples.tsv"
f.write_text("run\tfastq_1\tfastq_2\nSRR1\ta_1.fq\ta_2.fq\n")
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "tsv"
assert r["columns"] == ["run", "fastq_1", "fastq_2"]
def test_inspect_file_garbled(tmp_path):
f = tmp_path / "messy.txt"
f.write_bytes(b"\x00\x01\x02\x03\xff\xfe garbage \x00\x00")
r = inspect_file(f)
assert r["ok"] is True
assert r["garbled"] > 0.3
def test_garbled_ratio_clean():
assert _garbled_ratio("ATCGATCG\n>seq\n") < 0.01
# ---------------------------------------------------------------------------
# transform_file
# ---------------------------------------------------------------------------
def test_transform_xlsx_to_fasta(tmp_path):
import openpyxl
src = tmp_path / "spl_list.xlsx"
wb = openpyxl.Workbook()
ws = wb.active
ws.append(["基因名", "序列", "备注"])
ws.append(["SPL1", "ATGGCTA", "水稻"])
ws.append(["SPL2", "ATGGCTB", "水稻"])
wb.save(src)
r = _run(transform_file(src, "fasta", column_map={"name": "基因名", "seq": "序列"}))
assert r["ok"] is True
assert r["records"] == 2
assert r["validated"]["ok"] is True
text = Path(r["path"]).read_text()
assert ">SPL1" in text and "ATGGCTA" in text
def test_transform_csv_to_fasta_heuristic(tmp_path):
src = tmp_path / "genes.csv"
src.write_text("gene_id,protein_sequence\nG1,MKTA\nG2,MKTB\n")
r = _run(transform_file(src, "fasta"))
assert r["ok"] is True
assert r["records"] == 2
def test_transform_csv_to_samples_yaml(tmp_path):
src = tmp_path / "design.csv"
src.write_text("sample,fastq_1,fastq_2,condition\nA,a_1.fq,a_2.fq,ctrl\nB,b_1.fq,b_2.fq,treated\n")
r = _run(transform_file(src, "samples.yaml", column_map={"name": "sample", "r1": "fastq_1", "r2": "fastq_2"}))
assert r["ok"] is True
assert r["records"] == 2
assert r["validated"]["ok"] is True
import yaml
samples = yaml.safe_load(Path(r["path"]).read_text())["samples"]
assert "A" in samples and samples["A"]["r1"] == "a_1.fq"
def test_transform_validation_failed(tmp_path):
# seq 列全空 → 产物无序列 → 校验失败
src = tmp_path / "bad.csv"
src.write_text("name,seq\nG1,\nG2,\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta", column_map={"name": "name", "seq": "seq"}))
assert ei.value.code == "VALIDATION_FAILED"
def test_transform_missing_columns(tmp_path):
# 无列映射、无 LLM、表头无启发式命中 → MISSING_COLUMNS
src = tmp_path / "unknown.csv"
src.write_text("col_a,col_b\n1,2\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta"))
assert ei.value.code == "MISSING_COLUMNS"
# ---------------------------------------------------------------------------
# 校验器
# ---------------------------------------------------------------------------
def test_validate_fasta(tmp_path):
f = tmp_path / "ok.fa"
f.write_text(">a\nACGT\n")
assert validate_fasta(f)["ok"] is True
def test_validate_fasta_empty(tmp_path):
f = tmp_path / "bad.fa"
f.write_text("no records here\n")
assert validate_fasta(f)["ok"] is False
def test_validate_tsv_inconsistent(tmp_path):
f = tmp_path / "bad.tsv"
f.write_text("a\tb\tc\n1\t2\n")
assert validate_tsv(f)["ok"] is False
def test_validate_samples_yaml(tmp_path):
f = tmp_path / "s.yaml"
f.write_text("samples:\n A:\n r1: a.fq\n r2: a2.fq\n")
assert validate_samples_yaml(f)["ok"] is True
# ===========================================================================
# S0 补强(1)inspect_file:NOT_FOUND / xlsx 列 / csv 格式 / 大文件 / 内容嗅探
# ===========================================================================
def test_inspect_file_not_found():
r = inspect_file("/no/such/file.fa")
assert r["ok"] is False
assert r["code"] == "NOT_FOUND"
def test_inspect_file_xlsx_columns(tmp_path):
import openpyxl
f = tmp_path / "spl.xlsx"
wb = openpyxl.Workbook()
ws = wb.active
ws.append(["基因名", "序列"])
ws.append(["SPL1", "MKTA"])
wb.save(f)
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "xlsx"
assert r["columns"] == ["基因名", "序列"]
def test_inspect_file_csv_format(tmp_path):
f = tmp_path / "genes.csv"
f.write_text("id,seq\nG1,MKTA\n")
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "csv"
assert r["columns"] == ["id", "seq"]
def test_inspect_file_large_uses_preview(tmp_path):
f = tmp_path / "big.fa"
f.write_bytes(b">seq\nACGT\n" + b"A" * (4 * 1024 * 1024))
r = inspect_file(f)
assert r["ok"] is True
assert r["format"] == "fasta"
def test_guess_format_content_sniffing_fasta(tmp_path):
from app.services.bioflow.file_transform import _guess_format
assert _guess_format("unknown.dat", b">seq1\nACGT\n") == "fasta"
def test_guess_format_content_sniffing_tsv(tmp_path):
from app.services.bioflow.file_transform import _guess_format
assert _guess_format("unknown.dat", b"a\tb\tc\n1\t2\t3\n") == "tsv"
def test_garbled_ratio_empty():
assert _garbled_ratio("") == 0.0
# ===========================================================================
# S0 补强(2)transform_file:NOT_FOUND / BAD_FORMAT / EMPTY_TABLE / tsv / gff3 /
# samples.yaml 启发式 / LLM 兜底 / LLM 异常兜底
# ===========================================================================
def test_transform_not_found(tmp_path):
with pytest.raises(TransformError) as ei:
_run(transform_file(tmp_path / "nope.csv", "fasta"))
assert ei.value.code == "NOT_FOUND"
def test_transform_bad_format(tmp_path):
src = tmp_path / "x.csv"
src.write_text("a,b\n1,2\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "bogus"))
assert ei.value.code == "BAD_FORMAT"
def test_transform_empty_table(tmp_path):
src = tmp_path / "empty.csv"
src.write_text("")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta"))
assert ei.value.code == "EMPTY_TABLE"
def test_transform_to_tsv(tmp_path):
src = tmp_path / "genes.csv"
src.write_text("gene_id,protein_sequence\nG1,MKTA\nG2,MKTB\n")
r = _run(transform_file(src, "tsv"))
assert r["ok"] is True
assert r["records"] == 2
assert r["validated"]["ok"] is True
def test_transform_to_gff3_unsupported(tmp_path):
src = tmp_path / "x.csv"
src.write_text("a,b\n1,2\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "gff3"))
assert ei.value.code == "UNSUPPORTED"
def test_transform_samples_yaml_heuristic(tmp_path):
src = tmp_path / "design.csv"
src.write_text("name,r1,r2,condition\nA,a_1.fq,a_2.fq,ctrl\nB,b_1.fq,b_2.fq,trt\n")
r = _run(transform_file(src, "samples.yaml"))
assert r["ok"] is True
assert r["records"] == 2
def test_transform_samples_yaml_missing_columns(tmp_path):
src = tmp_path / "unknown.csv"
src.write_text("col_a,col_b\n1,2\n")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "samples.yaml"))
assert ei.value.code == "MISSING_COLUMNS"
def test_transform_fasta_llm_column_map(tmp_path):
src = tmp_path / "candidates.csv"
src.write_text("col_a,col_b\nSPL1,ATGGCTA\nSPL2,ATGGCTB\n")
async def fake_llm(user, system):
return {"name": "col_a", "seq": "col_b"}
r = _run(transform_file(src, "fasta", llm=fake_llm))
assert r["ok"] is True
assert r["records"] == 2
def test_transform_fasta_llm_exception_falls_back_missing(tmp_path):
src = tmp_path / "candidates.csv"
src.write_text("col_a,col_b\nSPL1,ATGGCTA\n")
async def fake_llm(user, system):
raise RuntimeError("llm down")
with pytest.raises(TransformError) as ei:
_run(transform_file(src, "fasta", llm=fake_llm))
assert ei.value.code == "MISSING_COLUMNS"
# ===========================================================================
# S0 补强(3)校验器:gff3 各失败态 / samples.yaml 缺 r1r2 / YAML 解析失败 /
# tsv 空文件 / validate_format 未知格式
# ===========================================================================
def test_validate_gff3_ok(tmp_path):
f = tmp_path / "ok.gff3"
f.write_text("chr1\tsrc\tgene\t1\t100\t.\t+\t.\tID=x\n")
assert validate_gff3(f)["ok"] is True
def test_validate_gff3_no_data(tmp_path):
f = tmp_path / "empty.gff3"
f.write_text("# just a comment\n")
assert validate_gff3(f)["ok"] is False
def test_validate_gff3_bad_columns(tmp_path):
f = tmp_path / "bad.gff3"
f.write_text("chr1\tsrc\tgene\t1\t100\n") # 5 列 ≠ 9
assert validate_gff3(f)["ok"] is False
def test_validate_gff3_bad_strand(tmp_path):
f = tmp_path / "bad.gff3"
f.write_text("chr1\tsrc\tgene\t1\t100\t.\tx\t.\tID=x\n") # 第 7 列非法
assert validate_gff3(f)["ok"] is False
def test_validate_gff3_bad_int(tmp_path):
f = tmp_path / "bad.gff3"
f.write_text("chr1\tsrc\tgene\tabc\t100\t.\t+\t.\tID=x\n") # 起止非整数
assert validate_gff3(f)["ok"] is False
def test_validate_fasta_no_sequence(tmp_path):
f = tmp_path / "bad.fa"
f.write_text(">a\n")
assert validate_fasta(f)["ok"] is False
def test_validate_samples_yaml_missing_r1r2(tmp_path):
f = tmp_path / "s.yaml"
f.write_text("samples:\n A:\n condition: ctrl\n")
assert validate_samples_yaml(f)["ok"] is False
def test_validate_samples_yaml_parse_fail(tmp_path):
f = tmp_path / "s.yaml"
f.write_text("samples: [unclosed\n")
assert validate_samples_yaml(f)["ok"] is False
def test_validate_tsv_empty(tmp_path):
f = tmp_path / "empty.tsv"
f.write_text("")
assert validate_tsv(f)["ok"] is False
def test_validate_format_unknown(tmp_path):
f = tmp_path / "x.txt"
f.write_text("x")
r = validate_format(f, "bogus")
assert r["ok"] is False
assert "不支持" in r["reason"]
# ===========================================================================
# S0 补强(4)校验器缺失文件 / 格式猜测扩展名 / samples 非 dict / 空行跳过
# ===========================================================================
def test_validators_missing_file(tmp_path):
missing = tmp_path / "nope"
assert validate_fasta(missing)["ok"] is False
assert validate_gff3(missing)["ok"] is False
assert validate_tsv(missing)["ok"] is False
assert validate_samples_yaml(missing)["ok"] is False
def test_guess_format_all_extensions():
from app.services.bioflow.file_transform import _guess_format
assert _guess_format("a.gff3", b"") == "gff3"
assert _guess_format("a.gff", b"") == "gff3"
assert _guess_format("a.gtf", b"") == "gtf"
assert _guess_format("a.vcf", b"") == "vcf"
assert _guess_format("a.vcf.gz", b"") == "vcf"
assert _guess_format("a.yaml", b"") == "yaml"
assert _guess_format("a.yml", b"") == "yaml"
assert _guess_format("a.xls", b"") == "xlsx"
assert _guess_format("a.faa", b"") == "fasta"
assert _guess_format("a.fna", b"") == "fasta"
assert _guess_format("a.pep", b"") == "fasta"
assert _guess_format("a.cds", b"") == "fasta"
assert _guess_format("a.dat", b"no marker") == "txt"
def test_validate_samples_yaml_not_dict(tmp_path):
f = tmp_path / "s.yaml"
f.write_text("samples: just_a_string\n")
assert validate_samples_yaml(f)["ok"] is False
def test_transform_samples_yaml_skips_empty_rows(tmp_path):
src = tmp_path / "d.csv"
src.write_text("name,r1,r2\n,,\nA,a.fq,a2.fq\n")
r = _run(transform_file(src, "samples.yaml"))
assert r["ok"] is True
assert r["records"] == 1
File diff suppressed because it is too large Load Diff
+414
View File
@@ -0,0 +1,414 @@
"""tests/test_phase_b_error.py —— Phase B 测试用例集。
覆盖(Phase B 验收):
1. G13 结构化错误对象:CLI 命令 + MCP 工具失败时返回统一错误对象,含 8 个字段
(error_id / category / code / summary / evidence_refs / diagnosis_status /
retryable_without_change / suggested_actions),而非裸字符串 message。
2. MCP 薄适配行为回归:fetch_source_file / inspect_file / transform_file /
archive_job / reuse_archive / register_job_files 委托 CLI/service 后行为与原实现一致。
3. reports.register_report 的 httpx.Timeout 修复:不再抛 ValueError。
## G13 契约(本文件断言的结构)
失败返回为 dict,且 `ok == False`,并携带结构化错误对象:
- 结构 A:`{"ok": False, "code": "...", "error": {8 字段}}`
- 结构 B:`{"ok": False, "code": "...", "error_id": ..., "category": ..., ...}`(字段平铺)
两种结构均被 `_error_of` 识别;`code` 须可经顶层 `result["code"]` 或 `error["code"]`
取到(与 Phase A 测试 `r["code"]` 断言兼容)。
字段类型契约:
- error_id: 非空 str(错误唯一标识)
- category: 非空 str(错误分类,如 validation/not_found/io/network)
- code: 非空 str(既有错误码,如 NOT_FOUND / ASSET_EXISTS / BAD_ROLE)
- summary: 非空 str(人读摘要,等价原 message)
- evidence_refs: list(证据引用,可为空)
- diagnosis_status: 非空 str(如 diagnosed)
- retryable_without_change: bool(不改动重试是否可能成功)
- suggested_actions: list[str](建议动作,可为空)
每条用例三段式(前置/步骤/预期)写在 docstring。分层标记 -m bioflow_unit。
"""
from __future__ import annotations
import asyncio
import types as _types
from pathlib import Path
import httpx as _real_httpx
import pytest
import app.mcp_server as mcp
from app.services.bioflow import store
from app.services.bioflow.reports import register_report
from bioflowctl import commands as cli
pytestmark = pytest.mark.bioflow_unit
G13_FIELDS = (
"error_id", "category", "code", "summary", "evidence_refs",
"diagnosis_status", "retryable_without_change", "suggested_actions",
)
def _run(coro):
return asyncio.new_event_loop().run_until_complete(coro)
# ---------------------------------------------------------------------------
# G13 结构化错误对象断言助手
# ---------------------------------------------------------------------------
def _error_of(result: dict) -> dict:
"""从失败返回中取出结构化错误对象(兼容 `error` 键与字段平铺两种结构)。"""
if isinstance(result.get("error"), dict):
return result["error"]
if "error_id" in result:
return result
raise AssertionError(f"失败返回缺结构化错误对象: keys={sorted(result)}")
def _code_of(result: dict) -> str | None:
return result.get("code") or _error_of(result).get("code")
def _assert_g13_error(result, expected_code: str) -> dict:
"""断言 result 为 G13 失败返回,且 code 匹配;返回错误对象。"""
assert isinstance(result, dict), f"失败返回非 dict: {result!r}"
assert result.get("ok") is False, f"预期失败但 ok=True: {result!r}"
err = _error_of(result)
missing = [f for f in G13_FIELDS if f not in err]
assert not missing, f"G13 错误对象缺字段 {missing}: {err}"
assert isinstance(err["error_id"], str) and err["error_id"], "error_id 须非空 str"
assert isinstance(err["category"], str) and err["category"], "category 须非空 str"
assert isinstance(err["code"], str) and err["code"], "code 须非空 str"
assert isinstance(err["summary"], str) and err["summary"], "summary 须非空 str"
assert isinstance(err["evidence_refs"], list), "evidence_refs 须 list"
assert isinstance(err["diagnosis_status"], str) and err["diagnosis_status"], "diagnosis_status 须非空 str"
assert isinstance(err["retryable_without_change"], bool), "retryable_without_change 须 bool"
assert isinstance(err["suggested_actions"], list), "suggested_actions 须 list"
assert _code_of(result) == expected_code, f"code 期望 {expected_code},实际 {_code_of(result)}"
return err
async def _seed_archive_into(db, archive: dict) -> None:
await store.insert_archive(db, archive)
def _archive_row(**over) -> dict:
base = {
"id": "a1", "job_id": "j1", "name": "SPL鉴定", "description": "",
"stage_chain": ["identify", "phylogeny"],
"params": {"identify": {"cpu": "8"}},
"report_artifact": None, "disk_path": "/tmp/a1", "size_bytes": 10,
}
base.update(over)
return base
# ===========================================================================
# 1. G13 结构化错误对象 —— 形状契约(冒烟)
# ===========================================================================
def test_g13_error_object_has_all_fields(tmp_path):
"""前置:空 db。步骤:cmd_archive_reuse(不存在的存档)。
预期:ok=False 且含 8 字段结构化错误对象,code=ARCHIVE_NOT_FOUND。"""
r = _run(cli.cmd_archive_reuse("不存在", db_path=tmp_path / "empty.db"))
_assert_g13_error(r, "ARCHIVE_NOT_FOUND")
# ===========================================================================
# 2. CLI 命令失败路径 → G13(覆盖各命令的错误码)
# ===========================================================================
# ---- source fetch ----
def test_g13_source_fetch_empty(tmp_path):
"""前置:空源。步骤:cmd_source_fetch("")。预期:G13 错误,code=EMPTY_SOURCE。"""
r = _run(cli.cmd_source_fetch(""))
_assert_g13_error(r, "EMPTY_SOURCE")
def test_g13_source_fetch_bad_scheme(tmp_path):
"""前置:ftp:// 源。步骤:cmd_source_fetch。预期:G13 错误,code=BAD_SCHEME。"""
r = _run(cli.cmd_source_fetch("ftp://host/x.fa"))
_assert_g13_error(r, "BAD_SCHEME")
def test_g13_source_fetch_lan_missing(tmp_path):
"""前置:局域网路径不存在。步骤:cmd_source_fetch。预期:G13 错误,code=PATH_NOT_EXIST。"""
r = _run(cli.cmd_source_fetch("/no/such/path.fa"))
_assert_g13_error(r, "PATH_NOT_EXIST")
# ---- file inspect ----
def test_g13_file_inspect_not_found(tmp_path):
"""前置:文件不存在。步骤:cmd_file_inspect。预期:G13 错误,code=NOT_FOUND。"""
r = cli.cmd_file_inspect(str(tmp_path / "nope.fa"))
_assert_g13_error(r, "NOT_FOUND")
# ---- file transform ----
def test_g13_file_transform_not_found(tmp_path):
"""前置:源不存在。步骤:cmd_file_transform。预期:G13 错误,code=NOT_FOUND。"""
r = _run(cli.cmd_file_transform(str(tmp_path / "nope.csv"), "fasta"))
_assert_g13_error(r, "NOT_FOUND")
def test_g13_file_transform_bad_format(tmp_path):
"""前置:csv 源,目标 bed。步骤:cmd_file_transform。预期:G13 错误,code=BAD_FORMAT。"""
src = tmp_path / "a.csv"
src.write_text("a,b\n1,2\n")
r = _run(cli.cmd_file_transform(str(src), "bed"))
_assert_g13_error(r, "BAD_FORMAT")
def test_g13_file_transform_empty_table(tmp_path):
"""前置:空文件。步骤:cmd_file_transform(fasta)。预期:G13 错误,code=EMPTY_TABLE。"""
src = tmp_path / "empty.csv"
src.write_text("", encoding="utf-8")
r = _run(cli.cmd_file_transform(str(src), "fasta"))
_assert_g13_error(r, "EMPTY_TABLE")
def test_g13_file_transform_missing_columns(tmp_path):
"""前置:表头无 name/seq。步骤:cmd_file_transform(fasta)。预期:G13 错误,code=MISSING_COLUMNS。"""
src = tmp_path / "x.csv"
src.write_text("foo,bar\n1,2\n")
r = _run(cli.cmd_file_transform(str(src), "fasta"))
_assert_g13_error(r, "MISSING_COLUMNS")
def test_g13_file_transform_validation_failed(tmp_path):
"""前置:seq 全空。步骤:cmd_file_transform(fasta)。预期:G13 错误,code=VALIDATION_FAILED。"""
src = tmp_path / "genes.csv"
src.write_text("gene,seq\nSPL1,\nSPL2,\n")
r = _run(cli.cmd_file_transform(str(src), "fasta", column_map={"name": "gene", "seq": "seq"}))
_assert_g13_error(r, "VALIDATION_FAILED")
def test_g13_file_transform_gff3_unsupported(tmp_path):
"""前置:tsv 源。步骤:cmd_file_transform(gff3)。预期:G13 错误,code=UNSUPPORTED。"""
src = tmp_path / "a.tsv"
src.write_text("a\tb\tc\n1\t2\t3\n")
r = _run(cli.cmd_file_transform(str(src), "gff3"))
_assert_g13_error(r, "UNSUPPORTED")
# ---- asset register ----
def test_g13_asset_register_not_found(tmp_path):
"""前置:源不存在。步骤:register_asset。预期:G13 错误,code=NOT_FOUND。"""
r = cli.register_asset("pep.fa", str(tmp_path / "nope.fa"), assets_root=tmp_path / "assets")
_assert_g13_error(r, "NOT_FOUND")
def test_g13_asset_register_bad_role(tmp_path):
"""前置:角色名非法。步骤:register_asset。预期:G13 错误,code=BAD_ROLE。"""
f = tmp_path / "x.fa"
f.write_text(">x\nACGT\n")
r = cli.register_asset("../pep", str(f), assets_root=tmp_path / "assets")
_assert_g13_error(r, "BAD_ROLE")
def test_g13_asset_register_exists(tmp_path):
"""前置:同名资产已发布。步骤:register_asset 再次同名。预期:G13 错误,code=ASSET_EXISTS。"""
f = tmp_path / "x.fa"
f.write_text("V1")
assets = tmp_path / "assets"
assert cli.register_asset("pep.fa", str(f), assets_root=assets)["ok"] is True
r = cli.register_asset("pep.fa", str(f), assets_root=assets)
_assert_g13_error(r, "ASSET_EXISTS")
# ---- asset validate ----
def test_g13_asset_validate_not_found(tmp_path):
"""前置:文件不存在。步骤:validate_asset。预期:G13 错误,code=NOT_FOUND。"""
r = cli.validate_asset("pep.fa", str(tmp_path / "nope.fa"))
_assert_g13_error(r, "NOT_FOUND")
def test_g13_asset_validate_bad_role(tmp_path):
"""前置:角色无法推断格式 + txt。步骤:validate_asset。预期:G13 错误,code=BAD_ROLE。"""
f = tmp_path / "x.txt"
f.write_text("hello\n")
r = cli.validate_asset("unknown_role", str(f))
_assert_g13_error(r, "BAD_ROLE")
# ---- report register ----
def test_g13_report_register_empty():
"""前置:空路径。步骤:cmd_report_register("")。预期:G13 错误,code=EMPTY_REPORT。"""
r = cli.cmd_report_register("")
_assert_g13_error(r, "EMPTY_REPORT")
# ---- archive ----
def test_g13_archive_not_found(tmp_path):
"""前置:空 db。步骤:cmd_archive(不存在 job)。预期:G13 错误,code=NOT_FOUND。"""
r = _run(cli.cmd_archive(
"nope", "X", db_path=tmp_path / "empty.db",
jobs_dir=tmp_path / "jobs", archives_dir=tmp_path / "archives"))
_assert_g13_error(r, "NOT_FOUND")
# ---- archive reuse ----
def test_g13_archive_reuse_not_found(tmp_path):
"""前置:空 db。步骤:cmd_archive_reuse。预期:G13 错误,code=ARCHIVE_NOT_FOUND。"""
r = _run(cli.cmd_archive_reuse("不存在", db_path=tmp_path / "empty.db"))
_assert_g13_error(r, "ARCHIVE_NOT_FOUND")
# ===========================================================================
# 3. MCP 薄适配 —— 行为回归 + G13(6 个工具)
# ===========================================================================
# ---- inspect_file ----
def test_mcp_inspect_file_ok(tmp_path):
"""回归。前置:fasta 文件。步骤:mcp.inspect_file。预期:ok=True,format=fasta。"""
f = tmp_path / "p.fa"
f.write_text(">S1\nMKTA\n")
r = _run(mcp.inspect_file(str(f)))
assert r["ok"] is True
assert r["format"] == "fasta"
def test_mcp_inspect_file_not_found(tmp_path):
"""G13。前置:文件不存在。步骤:mcp.inspect_file。预期:G13 错误,code=NOT_FOUND。"""
r = _run(mcp.inspect_file(str(tmp_path / "nope.fa")))
_assert_g13_error(r, "NOT_FOUND")
# ---- transform_file ----
def test_mcp_transform_file_ok(tmp_path):
"""回归。前置:csv(gene,seq)。步骤:mcp.transform_file(fasta, 显式 column_map)。
预期:ok=True,format=fasta,records=2。"""
src = tmp_path / "genes.csv"
src.write_text("gene,seq\nSPL1,MKTA\nSPL2,MKTB\n")
r = _run(mcp.transform_file(str(src), "fasta", {"name": "gene", "seq": "seq"}))
assert r["ok"] is True
assert r["format"] == "fasta"
assert r["records"] == 2
def test_mcp_transform_file_failure(tmp_path):
"""G13。前置:源不存在。步骤:mcp.transform_file。预期:G13 错误,code=NOT_FOUND。"""
r = _run(mcp.transform_file(str(tmp_path / "nope.csv"), "fasta"))
_assert_g13_error(r, "NOT_FOUND")
# ---- fetch_source_file ----
def test_mcp_fetch_source_file_lan_ok(tmp_path):
"""回归。前置:本机文件。步骤:mcp.fetch_source_file(局域网路径, job_id=None)。
预期:ok=True,kind=lan。"""
f = tmp_path / "genome.fa"
f.write_text(">g1\nACGT\n")
r = _run(mcp.fetch_source_file(str(f), job_id=None))
assert r["ok"] is True
assert r["kind"] == "lan"
def test_mcp_fetch_source_file_failure(tmp_path):
"""G13。前置:空源。步骤:mcp.fetch_source_file("")。预期:G13 错误,code=EMPTY_SOURCE。"""
r = _run(mcp.fetch_source_file("", job_id=None))
_assert_g13_error(r, "EMPTY_SOURCE")
# ---- archive_job ----
def test_mcp_archive_job_not_found(api_context):
"""G13。前置:无该任务。步骤:mcp.archive_job(不存在)。预期:G13 错误,code=NOT_FOUND。"""
r = _run(mcp.archive_job("nope", "X"))
_assert_g13_error(r, "NOT_FOUND")
# ---- reuse_archive ----
def test_mcp_reuse_archive_not_found(api_context):
"""G13。前置:无匹配存档。步骤:mcp.reuse_archive。预期:G13 错误,code=ARCHIVE_NOT_FOUND。"""
r = _run(mcp.reuse_archive("不存在"))
_assert_g13_error(r, "ARCHIVE_NOT_FOUND")
def test_mcp_reuse_archive_ok(api_context):
"""回归。前置:seed 存档。步骤:mcp.reuse_archive。预期:ok=True,返回 stage_chain+params。"""
_run(_seed_archive_into(api_context.db, _archive_row(
params={"identify": {"cpu": "8", "memory": "16G"}})))
r = _run(mcp.reuse_archive("SPL鉴定"))
assert r["ok"] is True
assert r["data"]["stage_chain"] == ["identify", "phylogeny"]
assert r["data"]["params"]["identify"]["cpu"] == "8"
# ---- register_job_files ----
def test_mcp_register_job_files_not_found(api_context):
"""G13。前置:无该任务。步骤:mcp.register_job_files(不存在, [路径])。预期:G13 错误,code=NOT_FOUND。"""
r = _run(mcp.register_job_files("nope", ["/tmp/x.fa"]))
_assert_g13_error(r, "NOT_FOUND")
# ===========================================================================
# 4. reports.register_report 的 httpx.Timeout 修复(不再抛 ValueError)
# ===========================================================================
def _fake_httpx_client_for_report(monkeypatch, rep, resp_factory):
"""注入 fake httpx:Timeout 用真实类(校验修复后传参合法),AsyncClient 用 resp_factory。"""
captured: dict = {}
class _Client:
def __init__(self, timeout=None):
captured["timeout"] = timeout
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
async def post(self, url, json=None):
return resp_factory()
monkeypatch.setattr(rep, "httpx", _types.SimpleNamespace(
AsyncClient=_Client, Timeout=_real_httpx.Timeout,
HTTPError=_real_httpx.HTTPError,
))
return captured
def test_register_report_unreachable_no_value_error(monkeypatch):
"""修复。前置:登记端点不可达(ConnectError)。步骤:register_report(register_url)。
预期:不抛 ValueError,返回 degraded=True, reason=unreachable。"""
from app.services.bioflow import reports as rep
def _resp():
raise _real_httpx.ConnectError("unreachable")
_fake_httpx_client_for_report(monkeypatch, rep, _resp)
r = _run(rep.register_report({"job_id": "j1"}, register_url="http://127.0.0.1:1/x"))
assert r == {"ok": False, "degraded": True, "reason": "unreachable"}
def test_register_report_http_error_degraded(monkeypatch):
"""修复。前置:登记端点返回 500。步骤:register_report(register_url)。
预期:不抛 ValueError,返回 degraded=True, reason=http_500。"""
from app.services.bioflow import reports as rep
class _Resp:
status_code = 500
async def json(self):
return {}
_fake_httpx_client_for_report(monkeypatch, rep, lambda: _Resp())
r = _run(rep.register_report({"job_id": "j1"}, register_url="http://127.0.0.1:1/x"))
assert r["ok"] is False
assert r["degraded"] is True
assert r["reason"] == "http_500"
+79
View File
@@ -0,0 +1,79 @@
"""tests/test_phase_c.py —— Phase C(G12 Skill 接入 + 11 工具)补充测试。
契约测试见 test_phase_c_skill.py;本文件补:MCP 注册、cmd_validate_plan(CLI 参数校验)。
分层标记 -m bioflow_unit。
"""
from __future__ import annotations
import asyncio
import pytest
from app.services.bioflow.skill_tools import (
TOOL_NAMES,
control_execution,
validate_plan,
)
from bioflowctl.commands import cmd_validate_plan
pytestmark = pytest.mark.bioflow_unit
def _run(coro):
return asyncio.new_event_loop().run_until_complete(coro)
def test_g12_eleven_tools_registered_on_mcp():
from app.mcp_server import build_mcp_servers
gf, rn, _ext = build_mcp_servers()
gf_names = {t.name for t in _run(gf.list_tools())}
rn_names = {t.name for t in _run(rn.list_tools())}
for name in TOOL_NAMES:
assert name in gf_names, f"基因家族端点缺工具 {name}"
assert name in rn_names, f"转录组端点缺工具 {name}"
def test_g12_validate_plan_known():
r = validate_plan(["identify", "phylogeny"])
assert r["ok"] is True
assert r["result"]["valid"] is True
def test_g12_validate_plan_unknown():
r = validate_plan(["bogus_stage"])
assert r["ok"] is False
assert r["error"]["code"] == "UNKNOWN_STAGE"
def test_g12_control_execution_unknown_command():
r = control_execution("", "bogus")
assert r["ok"] is False
assert r["error"]["code"] == "VALIDATION"
def test_g12_control_execution_stop_resume():
# 独立 run_id 避免模块级状态跨用例污染
r1 = control_execution("run-ctx-9", "start")
assert r1["ok"] is True and r1["result"]["done"] is True # claim
r2 = control_execution("run-ctx-9", "stop")
assert r2["ok"] is True and r2["result"]["done"] is True # 所有权匹配 → 终止
# ---- cmd_validate_plan(CLI 层,读单一 schema 校验参数)----
def test_g12_cmd_validate_plan_valid():
r = cmd_validate_plan("identify", {"cpu": "8", "memory": "32g"})
assert r["ok"] is True
def test_g12_cmd_validate_plan_unknown_param():
r = cmd_validate_plan("identify", {"bogus": "x"})
assert r["ok"] is False
assert r["code"] == "VALIDATION_FAILED"
assert "bogus" in r["message"]
def test_g12_cmd_validate_plan_choice_out_of_range():
r = cmd_validate_plan("identify", {"cpu": "99"})
assert r["ok"] is False
assert r["code"] == "VALIDATION_FAILED"
+201
View File
@@ -0,0 +1,201 @@
"""tests/test_phase_c_skill.py —— Phase C(Skill 接入 + 11 工具,G12)测试用例集。
覆盖(Phase C 验收):
1. 11 个分析控制工具:get_task_context / inspect_asset / validate_plan /
check_readiness / get_run_evidence / propose_change / apply_change /
execute_nodes / control_execution / validate_delivery / publish_delivery
—— 每个工具正确委托底层 CLI 命令,返回统一结构化结果(ok+result 或 G13 错误对象)。
2. SKILL.md 结构:frontmatter 齐全(name/description/触发词)、11 工具白名单、
对话剧本、人工边界(subfamily_confirm/rnaseq_de requires_human)、验收清单。
3. 装卸脚本:install/uninstall 正确安装/移除 skill。
## 契约说明(开发对齐用)
- 11 工具集中 `app.services.bioflow.skill_tools`,每个工具为模块级函数,签名见
SKILL_TOOL_SIGNATURES;返回统一 `{"ok": bool, "result"?: ..., "error"?: G13}`。
- SKILL.md 位于 `bioflow_service/SKILL.md`(相对仓库根)。
- 装卸脚本:`bioflow_service/install_skill.sh` 与 `bioflow_service/uninstall_skill.sh`。
未实现函数用 `_phase_c_fn` 动态导入,缺失则 skip(开发完成后须全部存在、无 skip)。
"""
from __future__ import annotations
import importlib
import re
from pathlib import Path
import pytest
pytestmark = pytest.mark.bioflow_unit
REPO_ROOT = Path(__file__).resolve().parent.parent.parent
TOOL_NAMES = [
"get_task_context", "inspect_asset", "validate_plan", "check_readiness",
"get_run_evidence", "propose_change", "apply_change", "execute_nodes",
"control_execution", "validate_delivery", "publish_delivery",
]
# 每个工具的最小调用参数(无参/空参 → 应返回统一结构,失败即 G13)
SKILL_TOOL_SIGNATURES = {
"get_task_context": dict(job_id=None),
"inspect_asset": dict(role="pep.fa", path=""),
"validate_plan": dict(stages=[]),
"check_readiness": dict(assets_root="", stages=[]),
"get_run_evidence": dict(run_id=""),
"propose_change": dict(job_id="", stage="", params={}),
"apply_change": dict(action_id="", change={}),
"execute_nodes": dict(job_id="", nodes=[]),
"control_execution": dict(run_id="", command=""),
"validate_delivery": dict(manifest={}),
"publish_delivery": dict(manifest={}),
}
def _phase_c_fn(module: str, name: str):
try:
mod = importlib.import_module(module)
return getattr(mod, name)
except (ImportError, AttributeError) as exc:
pytest.skip(f"Phase C 未实现 {module}.{name}: {exc}")
def _skill_md_path() -> Path | None:
"""定位 SKILL.md(候选路径:bioflow_service/SKILL.md)。"""
candidates = [
REPO_ROOT / "bioflow_service" / "SKILL.md",
REPO_ROOT / "bioflow_service" / "bioflowctl" / "SKILL.md",
REPO_ROOT / "SKILL.md",
]
for p in candidates:
if p.is_file():
return p
return None
def _skill_md_text() -> str | None:
p = _skill_md_path()
if p is None:
return None
return p.read_text(encoding="utf-8", errors="replace")
# ===========================================================================
# G12-1:11 个分析控制工具
# ===========================================================================
def test_g12_skill_exposes_eleven_tools():
"""G12。前置:skill_tools 模块。步骤:读 TOOL_NAMES/模块属性。预期:恰 11 工具。"""
try:
mod = importlib.import_module("app.services.bioflow.skill_tools")
except ImportError:
pytest.skip("Phase C 未实现 app.services.bioflow.skill_tools")
names = getattr(mod, "TOOL_NAMES", None)
if names is None:
# 逐个属性探测
names = [n for n in TOOL_NAMES if hasattr(mod, n)]
assert set(names) == set(TOOL_NAMES), f"工具集不一致: {names}"
@pytest.mark.parametrize("tool", TOOL_NAMES)
def test_g12_tool_returns_unified_structure(tool):
"""G12。前置:工具可调用。步骤:以最小参数调用。预期:返回 dict 含 ok;
失败时携带 G13 错误对象(非裸字符串)。"""
fn = _phase_c_fn("app.services.bioflow.skill_tools", tool)
result = fn(**SKILL_TOOL_SIGNATURES[tool])
assert isinstance(result, dict), f"{tool} 未返回 dict: {result!r}"
assert "ok" in result, f"{tool} 返回缺 ok 键"
if result["ok"] is False:
err = result.get("error") if isinstance(result.get("error"), dict) else result
assert "error_id" in err or "code" in err, f"{tool} 失败返回缺 G13 错误对象"
def test_g12_get_task_context_delegates():
"""G12。前置:get_task_context 工具。步骤:调用。预期:返回 5 字段任务上下文。"""
fn = _phase_c_fn("app.services.bioflow.skill_tools", "get_task_context")
r = fn(job_id=None)
if r.get("ok") is True:
ctx = r.get("result") or r.get("data") or {}
for k in ("session_id", "job_id", "plan_revision", "run_id", "current_stage"):
assert k in ctx, f"缺字段 {k}"
def test_g12_validate_delivery_delegates():
"""G12。前置:validate_delivery 工具。步骤:不完整 manifest。预期:ok=False + G13。"""
fn = _phase_c_fn("app.services.bioflow.skill_tools", "validate_delivery")
r = fn(manifest={})
assert isinstance(r, dict) and "ok" in r
# ===========================================================================
# G12-2:SKILL.md 结构
# ===========================================================================
def test_skill_md_exists():
"""SKILL.md。前置:文件存在。步骤:定位。预期:找到 SKILL.md。"""
assert _skill_md_path() is not None, "未找到 SKILL.md"
def test_skill_md_frontmatter():
"""SKILL.md。前置:frontmatter。步骤:解析 name/description/触发词。
预期:name/description 非空,含触发词。"""
text = _skill_md_text()
assert text is not None
m = re.match(r"^---\s*\n(.*?)\n---", text, flags=re.S)
assert m is not None, "缺 frontmatter"
fm = m.group(1)
assert "name:" in fm, "frontmatter 缺 name"
assert "description:" in fm, "frontmatter 缺 description"
# 触发词(name 或 description 或 triggers 字段)
assert any(k in fm for k in ("triggers", "触发", "生物", "基因家族", "生信")), "缺触发词"
def test_skill_md_eleven_tools_whitelist():
"""SKILL.md。前置:正文。步骤:检查 11 工具白名单。预期:11 工具名全部列出。"""
text = _skill_md_text()
assert text is not None
for tool in TOOL_NAMES:
assert tool in text, f"SKILL.md 缺工具 {tool}"
def test_skill_md_human_boundaries():
"""SKILL.md。前置:正文。步骤:检查人工边界。预期:subfamily_confirm/rnaseq_de 标 requires_human。"""
text = _skill_md_text()
assert text is not None
assert "subfamily_confirm" in text, "缺 subfamily_confirm 人工边界"
assert "rnaseq_de" in text, "缺 rnaseq_de 人工边界"
assert "requires_human" in text, "缺 requires_human 标注"
def test_skill_md_acceptance_checklist():
"""SKILL.md。前置:正文。步骤:检查验收清单。预期:含验收/检查清单。"""
text = _skill_md_text()
assert text is not None
assert any(k in text for k in ("验收", "acceptance", "检查清单", "checklist")), "缺验收清单"
# ===========================================================================
# G12-3:装卸脚本
# ===========================================================================
def test_skill_install_script_exists():
"""装卸。前置:install 脚本。步骤:定位。预期:存在 install_skill.sh。"""
p = REPO_ROOT / "bioflow_service" / "install_skill.sh"
assert p.is_file(), f"未找到 install 脚本: {p}"
def test_skill_uninstall_script_exists():
"""装卸。前置:uninstall 脚本。步骤:定位。预期:存在 uninstall_skill.sh。"""
p = REPO_ROOT / "bioflow_service" / "uninstall_skill.sh"
assert p.is_file(), f"未找到 uninstall 脚本: {p}"
def test_skill_install_uninstall_scripts_content():
"""装卸。前置:两脚本。步骤:读内容。预期:install 含安装、uninstall 含移除逻辑。"""
install = REPO_ROOT / "bioflow_service" / "install_skill.sh"
uninstall = REPO_ROOT / "bioflow_service" / "uninstall_skill.sh"
assert install.is_file(), f"未找到 install 脚本: {install}"
assert uninstall.is_file(), f"未找到 uninstall 脚本: {uninstall}"
itxt = install.read_text(encoding="utf-8", errors="replace")
utxt = uninstall.read_text(encoding="utf-8", errors="replace")
assert "SKILL" in itxt or "skill" in itxt
assert "rm" in utxt or "remove" in utxt.lower() or "uninstall" in utxt.lower()
+228
View File
@@ -0,0 +1,228 @@
"""tests/test_phase_d.py —— Phase D(G7-G11 + G14 + G15)补充测试(契约测试见 test_phase_d_contract.py)。
覆盖契约未直接断言的实现细节:
- classify_reason(runner 失败原因 + 退出码/日志证据定类)。
- auto_fill_prerequisites 的 satisfied_roles 跳过上游(G7 端到端)。
- stage_schema 单一 schema 访问点(G9)。
- RecoveryBudget.decide 的 runner 集成语义 + runner 自动重试(G15)。
- task_context.current_stage。
分层标记 -m bioflow_unit。
"""
from __future__ import annotations
import asyncio
import pytest
from app.services.bioflow.catalog import build_catalog, stage_schema
from app.services.bioflow.errors import classify_reason
from app.services.bioflow.kernel import ImageMap
from app.services.bioflow.readiness import check_readiness
from app.services.bioflow.recovery import RecoveryBudget
from app.services.bioflow.runner import JOB_FAILED, STAGE_FAILED, JobRecord
from app.services.bioflow.stage_assets import (
downstream_entry_satisfied,
stage_input_roles,
upstream_stages_for_roles,
)
from app.services.bioflow.task_context import current_stage, get_task_context
from conftest import FakeTimedOutExecutor, make_runner
pytestmark = pytest.mark.bioflow_unit
def _scenario(coro_factory):
loop = asyncio.new_event_loop()
try:
return loop.run_until_complete(coro_factory())
finally:
loop.close()
def _rnaseq_catalog():
im = ImageMap(stages={
"rnaseq_qc": {"image": "bioflow-rnaseq", "module": "cli.run_rnaseq_qc", "executable": True},
"rnaseq_align": {"image": "bioflow-rnaseq", "module": "cli.run_rnaseq_align", "executable": True},
"rnaseq_quant": {"image": "bioflow-rnaseq", "module": "cli.run_rnaseq_quant", "executable": True},
"rnaseq_sample_qc": {"image": "bioflow-rnaseq", "module": "cli.run_rnaseq_sample_qc", "executable": True},
"rnaseq_de": {"image": "bioflow-rnaseq", "module": "cli.run_rnaseq_de", "executable": True},
})
return build_catalog(im)
# ===========================================================================
# G14 classify_reason(runner 失败原因 + 证据)
# ===========================================================================
def test_g14_reason_timeout_transient():
cat, diag, retry, _ = classify_reason("timeout")
assert cat == "transient" and retry is True
def test_g14_reason_exit_code_undiagnosed():
cat, diag, retry, _ = classify_reason("exit_code")
assert cat == "adapter" and diag == "undiagnosed" and retry is False
def test_g14_reason_no_manifest_artifact():
cat, *_ = classify_reason("no_manifest")
assert cat == "artifact"
def test_g14_exit_code_137_resource():
cat, *_ = classify_reason("exit_code", exit_code=137)
assert cat == "resource"
def test_g14_log_oom_resource():
cat, *_ = classify_reason("exit_code", log_tail="Error: out of memory in container")
assert cat == "resource"
def test_g14_log_image_environment():
cat, *_ = classify_reason("exit_code", log_tail="docker: no such image")
assert cat == "environment"
def test_g14_log_scientific_precondition():
cat, *_ = classify_reason("exit_code", log_tail="no significant DEGs found")
assert cat == "scientific_precondition"
def test_g14_log_reference():
cat, *_ = classify_reason("exit_code", log_tail="reference genome index not found")
assert cat == "reference"
# ===========================================================================
# G7/G8 角色化资产 + 下游直入
# ===========================================================================
def test_g8_stage_input_roles_counts_only():
assert stage_input_roles("rnaseq_de") == ["counts"]
assert stage_input_roles("unknown") == []
def test_g7_downstream_entry_satisfied_tuple():
ok, missing = downstream_entry_satisfied("rnaseq_de", {"counts": {"validated": {"ok": True}}})
assert ok is True and missing == []
def test_g7_auto_fill_skips_counts_upstream():
catalog = _rnaseq_catalog()
ordered, _ = catalog.auto_fill_prerequisites(["rnaseq_de"])
assert set(["rnaseq_qc", "rnaseq_align", "rnaseq_quant", "rnaseq_sample_qc"]) <= set(ordered)
ordered2, added2 = catalog.auto_fill_prerequisites(["rnaseq_de"], satisfied_roles=["counts"])
assert "rnaseq_qc" not in ordered2 and "rnaseq_align" not in ordered2
assert "rnaseq_quant" not in ordered2 and "rnaseq_sample_qc" not in ordered2
assert ordered2 == ["rnaseq_de"] and added2 == []
def test_g7_upstream_stages_for_roles():
up = upstream_stages_for_roles(["counts"])
assert set(["rnaseq_qc", "rnaseq_align", "rnaseq_quant", "rnaseq_sample_qc"]) <= up
# ===========================================================================
# G9 单一参数 schema
# ===========================================================================
def test_g9_stage_schema_single_source():
schema = stage_schema("identify")
assert schema["name"] == "基因家族鉴定"
assert isinstance(schema["params"], list) and schema["params"]
assert schema["dependencies"] == []
assert schema["input_roles"] == ["hmm", "protein_fasta"]
def test_g9_stage_schema_unknown_fallback():
schema = stage_schema("no_such_stage")
assert schema["name"] == "no_such_stage"
assert schema["params"] and schema["params"][0]["name"] == "cpu"
# ===========================================================================
# G10 就绪检查(四象限)
# ===========================================================================
def test_g10_check_readiness_all_ready(tmp_path):
r = check_readiness(assets_root=tmp_path, stages=[])
assert r["ready"] is True
for q in ("environment", "reference", "space", "budget"):
assert q in r and r[q]["ok"] is True
def test_g10_check_readiness_missing_reference(tmp_path):
r = check_readiness(assets_root=tmp_path, stages=["identify"])
assert r["ready"] is False
assert r["reference"]["ok"] is False
assert r["reference"]["error"]["category"] == "reference"
# ===========================================================================
# G11 任务绑定
# ===========================================================================
def test_g11_get_task_context_empty_binding():
ctx = get_task_context()
for k in ("session_id", "job_id", "plan_revision", "run_id", "current_stage"):
assert k in ctx
def test_g11_get_task_context_with_record():
rec = JobRecord(id="j1", session_id="s1", stage_chain=["identify", "phylogeny"])
rec._stage_states = {"identify": {"status": "done"}, "phylogeny": {"status": "running"}}
ctx = get_task_context(rec)
assert ctx["session_id"] == "s1" and ctx["job_id"] == "j1"
assert ctx["current_stage"] == "phylogeny"
def test_g11_current_stage():
rec = JobRecord(id="j1", stage_chain=["a", "b", "c"])
rec._stage_states = {"a": {"status": "done"}, "b": {"status": "done"}, "c": {"status": "running"}}
assert current_stage(rec) == "c"
# ===========================================================================
# G15 恢复预算 + runner 自动重试
# ===========================================================================
def test_g15_recovery_can_retry_semantics():
b = RecoveryBudget(max_same_cause_retries=2)
assert b.can_retry("x") is True
b.record_retry("x")
assert b.can_retry("x") is True
b.record_retry("x")
assert b.can_retry("x") is False
def test_g15_recovery_decide_non_transient():
b = RecoveryBudget()
assert b.decide("s", "exit_code") is False
assert b.decide("s", "no_manifest") is False
def test_g15_recovery_decide_user_stopped():
b = RecoveryBudget()
assert b.decide("s", "timeout", user_stopped=True) is False
def test_g15_recovery_decide_resource_gate():
b = RecoveryBudget()
assert b.decide("s", "exit_code", log_tail="out of memory") is False
def test_g15_runner_auto_retries_transient(db, events, catalog):
executor = FakeTimedOutExecutor()
async def scenario():
runner = make_runner(db, events, executor, catalog)
record = await runner.create_job(
"s1", "t", ["identify"], auto_continue=True, auto_start=False)
await runner.run_job(record)
return runner, record
runner, record = _scenario(scenario)
assert runner.stage_status(record, "identify") == STAGE_FAILED
assert record.status == JOB_FAILED
assert len(executor.calls) == 3 # 1 初试 + 2 自动重试
@@ -0,0 +1,264 @@
"""tests/test_phase_d_contract.py —— Phase D(契约与就绪 + 诊断)测试用例集。
覆盖(Phase D 验收):G7 / G8 / G9 / G10 / G11 / G14 / G15。
## 契约说明(开发对齐用)
- G14 错误分类:`app.services.bioflow.errors.CATEGORIES`(8 类)+ `classify(code, summary)`。
8 类确定性分类:input / scientific_precondition / reference / environment /
transient / resource / adapter / artifact。OOM→resource、镜像缺→environment、
科学前提→scientific_precondition、参考缺失→reference。
- G10 就绪检查:`check_readiness` 四象限(environment/reference/space/budget),
缺项返回结构化错误并阻断(阻断 = ready=False)。
- G11 任务绑定:`get_task_context` 返回 session_id+job_id+plan_revision+run_id+当前阶段。
- G7/G8:`stage_input_roles(stage)` 返回阶段依赖输入角色;`asset_role_satisfied` 判断
角色是否有 validated 资产;下游直入(counts→DE)不补跑上游 QC/align/quant。
- G9 单一参数 schema:所有入口读 `catalog.get(stage).params`(无多份不一致)。
- G15 恢复预算:同因重试 ≤2;收敛条件(scientific_precondition / 无新证据 /
用户停止 / 资源门禁)终止重试。
未实现函数用 `_phase_d_fn` 动态导入,缺失则 skip(开发完成后须全部可导入、无 skip)。
"""
from __future__ import annotations
import importlib
from pathlib import Path
import pytest
from app.services.bioflow import errors
from app.services.bioflow.catalog import build_catalog
pytestmark = pytest.mark.bioflow_unit
def _phase_d_fn(module: str, name: str):
"""动态取 Phase D 函数;未实现则 skip(开发完成后须全部存在)。"""
try:
mod = importlib.import_module(module)
return getattr(mod, name)
except (ImportError, AttributeError) as exc:
pytest.skip(f"Phase D 未实现 {module}.{name}: {exc}")
# ===========================================================================
# G14 错误分类(8 类确定性分类)
# ===========================================================================
def test_g14_categories_exactly_eight():
"""前置:errors 模块。步骤:读 CATEGORIES。预期:恰 8 类。"""
assert set(errors.CATEGORIES) == {
"input", "scientific_precondition", "reference", "environment",
"transient", "resource", "adapter", "artifact",
}
def test_g14_classify_resource():
"""前置:DISK_FULL。步骤:classify。预期:category=resource,不可重试。"""
cat, diag, retry, actions = errors.classify("DISK_FULL", "")
assert cat == "resource"
assert diag == "diagnosed"
assert retry is False
assert isinstance(actions, list)
def test_g14_classify_environment():
"""前置:OSS_UNCONFIGURED。步骤:classify。预期:category=environment。"""
cat, *_ = errors.classify("OSS_UNCONFIGURED", "")
assert cat == "environment"
def test_g14_classify_transient_retryable():
"""前置:NETWORK_ERROR。步骤:classify。预期:category=transient,可重试。"""
cat, diag, retry, _ = errors.classify("NETWORK_ERROR", "")
assert cat == "transient"
assert retry is True
def test_g14_classify_input():
"""前置:NOT_FOUND。步骤:classify。预期:category=input。"""
cat, *_ = errors.classify("NOT_FOUND", "")
assert cat == "input"
def test_g14_classify_adapter():
"""前置:UNSUPPORTED。步骤:classify。预期:category=adapter。"""
cat, *_ = errors.classify("UNSUPPORTED", "")
assert cat == "adapter"
def test_g14_classify_artifact():
"""前置:ARCHIVE_EXISTS。步骤:classify。预期:category=artifact。"""
cat, *_ = errors.classify("ARCHIVE_EXISTS", "")
assert cat == "artifact"
def test_g14_classify_oom_keyword_to_resource():
"""G14 确定性:OOM→resource。前置:无枚举码、摘要含 out of memory。
步骤:classify(None, "container out of memory")。预期:category=resource。"""
cat, *_ = errors.classify(None, "container out of memory")
assert cat == "resource"
def test_g14_classify_image_keyword_to_environment():
"""G14 确定性:镜像缺→environment。前置:摘要含 image。步骤:classify。
预期:category=environment。"""
cat, *_ = errors.classify(None, "docker image not found")
assert cat == "environment"
def test_g14_classify_scientific_precondition():
"""G14 确定性:科学前提→scientific_precondition。前置:摘要含科学前提语义。
步骤:classify。预期:category=scientific_precondition(开发须补齐该映射)。"""
cat, *_ = errors.classify(None, "上游科学前提未满足,无法继续")
assert cat == "scientific_precondition"
def test_g14_classify_reference():
"""G14 确定性:参考缺失→reference。前置:摘要含参考基因组缺失。
步骤:classify。预期:category=reference(开发须补齐该映射)。"""
cat, *_ = errors.classify(None, "参考基因组缺失")
assert cat == "reference"
def test_g14_classify_category_invariant():
"""G14 不变量:任意输入分类结果 category 必在 8 类内。"""
samples = [
("DISK_FULL", ""), ("NOT_FOUND", ""), (None, "timeout"), (None, "unknown_xyz"),
("BAD_SCHEME", ""), ("ASSET_EXISTS", ""),
]
for code, summary in samples:
cat, *_ = errors.classify(code, summary)
assert cat in errors.CATEGORIES, f"分类结果 {cat!r} 不在 8 类内"
def test_g14_error_object_carries_category():
"""G14 集成:error_object 携带 classify 的 category。"""
obj = errors.error_object("DISK_FULL", "磁盘不足")
assert obj["category"] == "resource"
assert obj["retryable_without_change"] is False
# ===========================================================================
# G9 单一参数 schema(所有入口读 catalog.get(stage).params)
# ===========================================================================
def test_g9_catalog_is_single_param_source(image_map):
"""G9。前置:image_map。步骤:两次 build_catalog。预期:参数定义一致(单一来源)。"""
c1 = build_catalog(image_map)
c2 = build_catalog(image_map)
assert c1.get("identify").params == c2.get("identify").params
def test_g9_catalog_params_consistent_shape(image_map):
"""G9。前置:image_map。步骤:读 identify 参数。预期:参数为 list[dict],含 name。"""
entry = build_catalog(image_map).get("identify")
assert isinstance(entry.params, list)
if entry.params:
assert all("name" in p for p in entry.params)
# ===========================================================================
# G10 就绪检查(check_readiness 四象限:环境/参考/空间/预算)
# ===========================================================================
def test_g10_check_readiness_all_ready(tmp_path):
"""G10。前置:四象限齐备。步骤:check_readiness。预期:ready=True,四象限均 ok。"""
check_readiness = _phase_d_fn("app.services.bioflow.readiness", "check_readiness")
r = check_readiness(assets_root=tmp_path, stages=[])
assert r["ready"] is True
for q in ("environment", "reference", "space", "budget"):
assert q in r, f"缺象限 {q}"
assert r[q].get("ok") is True
def test_g10_check_readiness_missing_reference(tmp_path):
"""G10。前置:参考缺失。步骤:check_readiness。预期:ready=False,结构化错误+阻断。"""
check_readiness = _phase_d_fn("app.services.bioflow.readiness", "check_readiness")
r = check_readiness(assets_root=tmp_path, stages=["identify"])
assert r["ready"] is False
# 缺项返回结构化错误(G13 对象)
err = r.get("error") or (r.get("reference") or {}).get("error")
assert err is not None, "缺项应返回结构化错误对象"
for f in ("error_id", "category", "code", "summary", "diagnosis_status"):
assert f in err, f"缺字段 {f}"
def test_g10_check_readiness_four_quadrants_keys(tmp_path):
"""G10。前置:任意。步骤:check_readiness。预期:返回值含四象限键。"""
check_readiness = _phase_d_fn("app.services.bioflow.readiness", "check_readiness")
r = check_readiness(assets_root=tmp_path, stages=[])
for q in ("environment", "reference", "space", "budget"):
assert q in r, f"缺象限键 {q}"
# ===========================================================================
# G11 任务绑定(get_task_context)
# ===========================================================================
def test_g11_get_task_context_fields():
"""G11。前置:绑定态任务。步骤:get_task_context。预期:返回 5 字段。"""
get_task_context = _phase_d_fn("app.services.bioflow.task_context", "get_task_context")
ctx = get_task_context()
for k in ("session_id", "job_id", "plan_revision", "run_id", "current_stage"):
assert k in ctx, f"缺字段 {k}"
# ===========================================================================
# G7/G8 角色化资产 + 下游直入
# ===========================================================================
def test_g8_stage_input_roles():
"""G8。前置:阶段 rnaseq_de。步骤:stage_input_roles。预期:返回 counts 等输入角色。"""
stage_input_roles = _phase_d_fn("app.services.bioflow.stage_assets", "stage_input_roles")
roles = stage_input_roles("rnaseq_de")
assert isinstance(roles, list)
assert roles, "rnaseq_de 应声明输入角色(如 counts)"
def test_g8_role_satisfied_by_validated_asset():
"""G8。前置:validated 资产。步骤:asset_role_satisfied。预期:有 validated 资产即满足。"""
asset_role_satisfied = _phase_d_fn("app.services.bioflow.stage_assets", "asset_role_satisfied")
assert asset_role_satisfied("counts", {"role": "counts", "validated": {"ok": True}}) is True
assert asset_role_satisfied("counts", {"role": "counts", "validated": {"ok": False}}) is False
def test_g7_downstream_direct_entry_counts():
"""G7。前置:提供 counts 资产做 DE。步骤:下游直入判断。
预期:不补跑上游 QC/align/quant(上游角色由 counts 资产满足)。"""
downstream_entry_satisfied = _phase_d_fn(
"app.services.bioflow.stage_assets", "downstream_entry_satisfied")
ok, missing_upstream = downstream_entry_satisfied(
"rnaseq_de", {"counts": {"validated": {"ok": True}}})
assert ok is True
assert missing_upstream == []
# ===========================================================================
# G15 恢复预算(同因重试≤2 + 收敛条件)
# ===========================================================================
def test_g15_same_cause_retry_le_2():
"""G15。前置:恢复预算。步骤:同因连续重试。预期:≤2 次后拒绝。"""
RecoveryBudget = _phase_d_fn("app.services.bioflow.recovery", "RecoveryBudget")
b = RecoveryBudget(max_same_cause_retries=2)
assert b.can_retry("cause_x") is True # 第 1 次
b.record_retry("cause_x")
assert b.can_retry("cause_x") is True # 第 2 次
b.record_retry("cause_x")
assert b.can_retry("cause_x") is False # 第 3 次拒绝(≤2)
def test_g15_convergence_scientific_precondition():
"""G15。前置:scientific_precondition 分类。步骤:判断是否可重试。
预期:科学前提类不重试(收敛)。"""
cat, diag, retry, _ = errors.classify(None, "科学前提未满足")
assert cat == "scientific_precondition"
assert retry is False
def test_g15_convergence_resource_gate():
"""G15。前置:resource 门禁(如 DISK_FULL)。步骤:classify。
预期:资源类不可重试(资源门禁收敛)。"""
cat, diag, retry, _ = errors.classify("DISK_FULL", "")
assert cat == "resource"
assert retry is False
@@ -0,0 +1,184 @@
"""tests/test_phase_e_versioning.py —— Phase E(版本化修改)测试用例集。
覆盖(Phase E 验收):G16 / G17 / G18 / G19 / G20。
## 契约说明(开发对齐用,预期模块 + 函数签名)
- G16 版本化身份字段:plan_revision / run_id(已存在于 JobRecord + task_context.get_task_context);
新增 asset_id / based_on_revision / action_id,落在
`app.services.bioflow.versioning`:
new_revision(based_on_revision: str = None) -> dict
# 返回 {"revision": str, "based_on_revision": str|None,
# "asset_id": str, "action_id": str}
make_action_id() -> str # 唯一 action id
- G17 影响范围:`versioning.affected_stages(stage_chain: list[str], changed_stage: str) -> list[str]`
返回改参数后受影响的下游阶段(只重跑受影响节点;含 changed_stage 之后的所有下游)。
- G18 历史不覆盖:`versioning.RevisionStore(root: str|Path)`
write(revision: str, key: str, content: str) -> Path # 写到 revision 独立路径,不覆盖旧 revision
read(revision: str, key: str) -> str | None
- G19 防重复:`versioning.ActionDedup()`
try_start(action_id: str) -> bool # 首次 True,重复 False(幂等,不重复启动)
- G20 执行所有权:`app.services.bioflow.run_owner`:
claim(run_id: str, owner_id: str) -> bool # 首次 True(取得所有权),重复/他人 False
release(run_id: str, owner_id: str) -> bool
stop(run_id: str, owner_id: str) -> bool # 终止实际执行(非仅置状态)
resume(run_id: str, owner_id: str) -> bool # 已运行则 False(不重叠)
未实现函数用 `_phase_e_fn` 动态导入,缺失则 skip(开发完成后须全部存在、无 skip)。
"""
from __future__ import annotations
import importlib
import pytest
from app.services.bioflow.runner import JobRecord
from app.services.bioflow.task_context import get_task_context
pytestmark = pytest.mark.bioflow_unit
def _phase_e_fn(module: str, name: str):
"""动态取 Phase E 函数/类;未实现则 skip(开发完成后须全部存在)。"""
try:
mod = importlib.import_module(module)
return getattr(mod, name)
except (ImportError, AttributeError) as exc:
pytest.skip(f"Phase E 未实现 {module}.{name}: {exc}")
# ===========================================================================
# G16 版本化身份(plan_revision / run_id / asset_id / based_on_revision / action_id)
# ===========================================================================
def test_g16_job_record_revision_fields():
"""G16。前置:JobRecord。步骤:构造。预期:含 plan_revision + run_id 字段。"""
rec = JobRecord(id="j1")
assert rec.plan_revision == "1"
assert rec.run_id is None
def test_g16_task_context_revision_fields():
"""G16。前置:无 record。步骤:get_task_context()。预期:返回 plan_revision + run_id。"""
ctx = get_task_context()
assert "plan_revision" in ctx
assert "run_id" in ctx
def test_g16_new_revision_identity_fields():
"""G16。前置:versioning.new_revision。步骤:调用。预期:返回 4 个身份字段。"""
new_revision = _phase_e_fn("app.services.bioflow.versioning", "new_revision")
rev = new_revision(based_on_revision="1")
for k in ("revision", "based_on_revision", "asset_id", "action_id"):
assert k in rev, f"缺字段 {k}"
assert rev["based_on_revision"] == "1"
def test_g16_action_id_unique():
"""G16。前置:versioning.make_action_id。步骤:两次调用。预期:action_id 唯一。"""
make_action_id = _phase_e_fn("app.services.bioflow.versioning", "make_action_id")
a = make_action_id()
b = make_action_id()
assert a and b
assert a != b
# ===========================================================================
# G17 影响范围(改参数 → 依赖图 → 受影响下游阶段,只重跑受影响节点)
# ===========================================================================
def test_g17_affected_stages_downstream():
"""G17。前置:链 identify→phylogeny→motif。步骤:affected_stages(changed=identify)。
预期:返回下游 phylogeny/motif(不含已完成的 identify 之外无关节点)。"""
affected_stages = _phase_e_fn("app.services.bioflow.versioning", "affected_stages")
out = affected_stages(["identify", "phylogeny", "motif"], "identify")
assert out == ["phylogeny", "motif"]
def test_g17_affected_stages_no_downstream():
"""G17。前置:改末阶段 motif。步骤:affected_stages。预期:无下游,空列表。"""
affected_stages = _phase_e_fn("app.services.bioflow.versioning", "affected_stages")
out = affected_stages(["identify", "phylogeny", "motif"], "motif")
assert out == []
# ===========================================================================
# G18 历史不覆盖(新 revision 不覆盖旧 revision 的 manifest/产物/参数)
# ===========================================================================
def test_g18_revision_store_preserves_old(tmp_path):
"""G18。前置:RevisionStore。步骤:rev1 写 key,rev2 写同名 key。
预期:rev1 内容不被 rev2 覆盖。"""
RevisionStore = _phase_e_fn("app.services.bioflow.versioning", "RevisionStore")
store = RevisionStore(tmp_path)
store.write("rev1", "params.json", '{"cpu": "2"}')
store.write("rev2", "params.json", '{"cpu": "8"}')
assert store.read("rev1", "params.json") == '{"cpu": "2"}'
assert store.read("rev2", "params.json") == '{"cpu": "8"}'
def test_g18_revision_paths_isolated(tmp_path):
"""G18。前置:RevisionStore。步骤:rev1/rev2 各写产物。
预期:落不同路径(物理隔离,不互覆)。"""
RevisionStore = _phase_e_fn("app.services.bioflow.versioning", "RevisionStore")
store = RevisionStore(tmp_path)
p1 = store.write("rev1", "final.txt", "v1")
p2 = store.write("rev2", "final.txt", "v2")
assert str(p1) != str(p2)
assert p1.read_text() == "v1"
assert p2.read_text() == "v2"
# ===========================================================================
# G19 防重复(action_id 幂等,重复提交不重复启动)
# ===========================================================================
def test_g19_action_dedup_idempotent():
"""G19。前置:ActionDedup。步骤:同 action_id 两次 try_start。
预期:首次 True,第二次 False(幂等)。"""
ActionDedup = _phase_e_fn("app.services.bioflow.versioning", "ActionDedup")
dedup = ActionDedup()
assert dedup.try_start("act-1") is True
assert dedup.try_start("act-1") is False
def test_g19_duplicate_action_not_restarted():
"""G19。前置:ActionDedup。步骤:两个不同 action_id 各自启动,重复一个。
预期:不同 id 各自 True;重复 id 不重复启动。"""
ActionDedup = _phase_e_fn("app.services.bioflow.versioning", "ActionDedup")
dedup = ActionDedup()
assert dedup.try_start("act-1") is True
assert dedup.try_start("act-2") is True
assert dedup.try_start("act-1") is False # 不重复启动
# ===========================================================================
# G20 执行所有权(stop 终止实际执行 / resume 不重叠 / 重启不重复接管)
# ===========================================================================
def test_g20_claim_single_owner():
"""G20。前置:run_owner.claim。步骤:owner A 取得 run,owner B 再 claim。
预期:A True,B False(单一所有权,不重复接管)。"""
claim = _phase_e_fn("app.services.bioflow.run_owner", "claim")
assert claim("run-1", "owner-A") is True
assert claim("run-1", "owner-B") is False
def test_g20_resume_no_overlap():
"""G20。前置:run_owner.resume。步骤:已运行 run 再 resume。
预期:False(不重叠执行)。"""
claim = _phase_e_fn("app.services.bioflow.run_owner", "claim")
resume = _phase_e_fn("app.services.bioflow.run_owner", "resume")
assert claim("run-2", "owner-A") is True
assert resume("run-2", "owner-A") is False
def test_g20_stop_terminates_execution():
"""G20。前置:run_owner.claim + stop。步骤:claim 后 stop。
预期:stop 返回 True(实际终止),随后 release 允许重新 claim。"""
claim = _phase_e_fn("app.services.bioflow.run_owner", "claim")
stop = _phase_e_fn("app.services.bioflow.run_owner", "stop")
release = _phase_e_fn("app.services.bioflow.run_owner", "release")
assert claim("run-3", "owner-A") is True
assert stop("run-3", "owner-A") is True
assert release("run-3", "owner-A") is True
assert claim("run-3", "owner-B") is True # 释放后可被重新接管
+97
View File
@@ -0,0 +1,97 @@
"""tests/test_phase_f.py —— Phase F(交付)测试用例集。
覆盖:G21 交付清单(delivery_validate/publish)、G22 终报告范围(classify_report_scope)、
G23 资源预算门禁(check_download_budget / check_disk_available)。
契约测试见 test_phase_f_delivery.py;本文件补契约未直接断言的点(publish / build_report)。
分层标记 -m bioflow_unit。
"""
from __future__ import annotations
import pytest
from app.services.bioflow.delivery import (
build_report,
check_disk_available,
check_download_budget,
classify_report_scope,
delivery_validate,
publish_delivery,
)
pytestmark = pytest.mark.bioflow_unit
def _complete() -> dict:
return {
"selected_stages": ["identify", "phylogeny"],
"run_id": "run-1",
"input_sources": ["/data/genome.pep.fa"],
"output_sources": ["/data/final/Ft.SPL.final_members.id"],
"config": {"cpu": "8"},
"required_results": ["final_members.id"],
"validation_values": {"final_members": 24},
"review_status": "approved",
}
# G21 补充:publish 幂等/阻断
def test_g21_publish_valid():
m = _complete()
r = publish_delivery(m)
assert r["ok"] is True and r["published"] is True
assert m["review_status"] == "approved"
def test_g21_publish_invalid_blocked():
m = _complete()
m["run_id"] = ""
r = publish_delivery(m)
assert r["ok"] is False and r["published"] is False
def test_g21_validate_checks_eight_keys():
r = delivery_validate(_complete())
assert set(r["checks"]) == {
"selected_stages", "run_id", "input_sources", "output_sources",
"config", "required_results", "validation_values", "review_status",
}
# G22 补充:五分类边界
def test_g22_classify_five_categories():
assert classify_report_scope("done", "user") == "user_provided"
assert classify_report_scope("done", "reuse") == "reused"
assert classify_report_scope("done", None) == "user_provided"
assert classify_report_scope("waiting", None) == "not_executed"
assert classify_report_scope("queued", None) == "not_executed"
assert classify_report_scope("skipped", None) == "skipped"
assert classify_report_scope("failed", None) == "failed"
assert classify_report_scope("partial_success", None) == "not_executed"
# G22 补充:终报告结构
def test_g22_build_report_structure():
m = _complete()
report = build_report(m, stage_sources={"identify": "reused",
"phylogeny": "failed"})
for k in ("execution_scope", "data_sources", "qc", "scientific_preconditions",
"failed_skipped_uncovered", "artifacts"):
assert k in report, f"缺报告段 {k}"
assert report["failed_skipped_uncovered"]["failed"] == ["phylogeny"]
assert report["artifacts"] == ["final_members.id"]
# G23 补充
def test_g23_download_budget_boundary():
assert check_download_budget(100, 100)["ok"] is True # 恰好等于不超
assert check_download_budget(101, 100)["ok"] is False
def test_g23_disk_available_ok(tmp_path):
r = check_disk_available(tmp_path, 1)
assert r["ok"] is True
assert r["free_bytes"] > 0
@@ -0,0 +1,187 @@
"""tests/test_phase_f_delivery.py —— Phase F(交付)测试用例集。
覆盖(Phase F 验收):G21 / G22 / G23。
## 契约说明(开发对齐用,预期模块 + 函数签名)
模块:`app.services.bioflow.delivery`
- G21 交付清单完整性:
delivery_validate(manifest: dict) -> dict
manifest 八项:selected_stages / run_id / input_sources / output_sources /
config / required_results / validation_values / review_status。
返回 {"ok": bool, "checks": {item: bool},
"incomplete": [{"item": str, "reason": str, "error": {G13 对象}}]}
任一项缺失 → ok=False + incomplete 记录 item/reason/结构化错误(未完成原因)。
- G22 终报告范围:
classify_report_scope(status: str, source: str | None = None) -> str
status 终态分类:failed→"failed"、skipped→"skipped"、
非终态(waiting/queued/created/await_*)→"not_executed";
done + source="reuse"→"reused";done + source="user"(或空)→"user_provided"。
- G23 资源预算:
check_download_budget(size_bytes: int, budget_bytes: int) -> dict
# 超限 → {"ok": False, "code": "DOWNLOAD_BUDGET_EXCEEDED", "error": {...}}
check_disk_available(dest_dir, needed_bytes: int) -> dict
# 不足 → {"ok": False, "code": "DISK_FULL", "error": {...}}
两者阻断(ok=False)并返回结构化错误。
未实现函数用 `_phase_f_fn` 动态导入,缺失则 skip(开发完成后须全部存在、无 skip)。
"""
from __future__ import annotations
import importlib
import pytest
pytestmark = pytest.mark.bioflow_unit
def _phase_f_fn(module: str, name: str):
"""动态取 Phase F 函数;未实现则 skip(开发完成后须全部存在)。"""
try:
mod = importlib.import_module(module)
return getattr(mod, name)
except (ImportError, AttributeError) as exc:
pytest.skip(f"Phase F 未实现 {module}.{name}: {exc}")
def _complete_manifest() -> dict:
return {
"selected_stages": ["identify", "phylogeny"],
"run_id": "run-1",
"input_sources": ["/data/genome.pep.fa"],
"output_sources": ["/data/final/Ft.SPL.final_members.id"],
"config": {"cpu": "8"},
"required_results": ["final_members.id"],
"validation_values": {"final_members": 24},
"review_status": "approved",
}
def _g13_fields():
return ("error_id", "category", "code", "summary", "diagnosis_status")
# ===========================================================================
# G21 交付清单完整性(delivery validate)
# ===========================================================================
def test_g21_delivery_validate_complete():
"""G21。前置:八项齐备 manifest。步骤:delivery_validate。预期:ok=True,incomplete 空。"""
delivery_validate = _phase_f_fn("app.services.bioflow.delivery", "delivery_validate")
r = delivery_validate(_complete_manifest())
assert r["ok"] is True
assert r.get("incomplete") == []
def test_g21_delivery_validate_missing_run_id():
"""G21。前置:缺 run_id。步骤:delivery_validate。预期:ok=False,incomplete 含 run_id + 结构化错误。"""
delivery_validate = _phase_f_fn("app.services.bioflow.delivery", "delivery_validate")
m = _complete_manifest()
m["run_id"] = ""
r = delivery_validate(m)
assert r["ok"] is False
items = {i["item"]: i for i in r["incomplete"]}
assert "run_id" in items
err = items["run_id"].get("error")
assert err is not None
for f in _g13_fields():
assert f in err, f"缺字段 {f}"
def test_g21_delivery_validate_missing_required_results():
"""G21。前置:缺 required_results。步骤:delivery_validate。预期:ok=False,未完成原因含 required_results。"""
delivery_validate = _phase_f_fn("app.services.bioflow.delivery", "delivery_validate")
m = _complete_manifest()
m["required_results"] = []
r = delivery_validate(m)
assert r["ok"] is False
items = {i["item"]: i for i in r["incomplete"]}
assert "required_results" in items
assert items["required_results"].get("reason")
def test_g21_delivery_validate_checks_eight_items():
"""G21。前置:完整 manifest。步骤:delivery_validate。预期:checks 覆盖八项。"""
delivery_validate = _phase_f_fn("app.services.bioflow.delivery", "delivery_validate")
r = delivery_validate(_complete_manifest())
checks = r.get("checks")
assert checks is not None, "应返回 checks 覆盖八项"
for k in ("selected_stages", "run_id", "input_sources", "output_sources",
"config", "required_results", "validation_values", "review_status"):
assert k in checks, f"checks 缺项 {k}"
# ===========================================================================
# G22 终报告范围(用户提供 / 历史复用 / 未执行 / 跳过 / 失败)
# ===========================================================================
def test_g22_report_scope_user_provided():
"""G22。前置:done + source=user。步骤:classify_report_scope。预期:user_provided。"""
fn = _phase_f_fn("app.services.bioflow.delivery", "classify_report_scope")
assert fn("done", source="user") == "user_provided"
def test_g22_report_scope_reused():
"""G22。前置:done + source=reuse。步骤:classify_report_scope。预期:reused。"""
fn = _phase_f_fn("app.services.bioflow.delivery", "classify_report_scope")
assert fn("done", source="reuse") == "reused"
def test_g22_report_scope_not_executed():
"""G22。前置:非终态 waiting。步骤:classify_report_scope。预期:not_executed。"""
fn = _phase_f_fn("app.services.bioflow.delivery", "classify_report_scope")
assert fn("waiting", source=None) == "not_executed"
def test_g22_report_scope_skipped():
"""G22。前置:skipped。步骤:classify_report_scope。预期:skipped。"""
fn = _phase_f_fn("app.services.bioflow.delivery", "classify_report_scope")
assert fn("skipped", source=None) == "skipped"
def test_g22_report_scope_failed():
"""G22。前置:failed。步骤:classify_report_scope。预期:failed。"""
fn = _phase_f_fn("app.services.bioflow.delivery", "classify_report_scope")
assert fn("failed", source=None) == "failed"
# ===========================================================================
# G23 资源预算(下载预算超限 / 磁盘不足阻断,结构化错误)
# ===========================================================================
def test_g23_download_budget_exceeded():
"""G23。前置:下载超预算。步骤:check_download_budget。预期:ok=False + 结构化错误阻断。"""
fn = _phase_f_fn("app.services.bioflow.delivery", "check_download_budget")
r = fn(size_bytes=1024 * 1024 * 1024, budget_bytes=100)
assert r["ok"] is False
assert r["code"] == "DOWNLOAD_BUDGET_EXCEEDED"
err = r.get("error")
assert err is not None
for f in _g13_fields():
assert f in err, f"缺字段 {f}"
def test_g23_download_budget_ok():
"""G23。前置:下载未超预算。步骤:check_download_budget。预期:ok=True。"""
fn = _phase_f_fn("app.services.bioflow.delivery", "check_download_budget")
r = fn(size_bytes=100, budget_bytes=1024)
assert r["ok"] is True
def test_g23_disk_insufficient_blocks(tmp_path):
"""G23。前置:磁盘不足(1EB)。步骤:check_disk_available。预期:ok=False + 结构化错误。"""
fn = _phase_f_fn("app.services.bioflow.delivery", "check_disk_available")
r = fn(tmp_path, needed_bytes=10 ** 18)
assert r["ok"] is False
assert r["code"] == "DISK_FULL"
err = r.get("error")
assert err is not None
for f in _g13_fields():
assert f in err, f"缺字段 {f}"
def test_g23_disk_sufficient_ok(tmp_path):
"""G23。前置:磁盘充足(1 字节)。步骤:check_disk_available。预期:ok=True。"""
fn = _phase_f_fn("app.services.bioflow.delivery", "check_disk_available")
r = fn(tmp_path, needed_bytes=1)
assert r["ok"] is True
@@ -0,0 +1,455 @@
"""tests/test_source_fetcher.py —— 源文件拉取(web/OSS/局域网)+ 磁盘余量 + provenance。
分层标记 -m bioflow_unit。覆盖(S0 验收):
- web http 流式下载(本地 http.server 实测 httpx 通道)+ provenance 落盘。
- 协议白名单(ftp:// 拒绝 BAD_SCHEME)。
- oss:// 无 endpoint 拒绝 / 预签名 URL 直通 web / endpoint 解析。
- 局域网本机路径直读;远端 host:/path 解析。
- 磁盘余量不足 DISK_FULL(check_disk_space 纯函数)。
"""
from __future__ import annotations
import asyncio
import functools
import http.server
import threading
from pathlib import Path
import pytest
from app.services.bioflow.source_fetcher import (
FetchError,
_parse_lan_remote,
_resolve_oss_url,
check_disk_space,
fetch_lan,
fetch_source,
fetch_web,
)
def _run(coro):
return asyncio.new_event_loop().run_until_complete(coro)
# ---------------------------------------------------------------------------
# 本地 http 服务器(测 httpx 通道,不依赖外网)
# ---------------------------------------------------------------------------
@pytest.fixture()
def http_url(tmp_path):
content = b">seq1\nACGTACGT\n>seq2\nTGCATGCA\n"
(tmp_path / "test.fa").write_bytes(content)
handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=str(tmp_path))
server = http.server.HTTPServer(("127.0.0.1", 0), handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
yield f"http://127.0.0.1:{server.server_address[1]}/test.fa"
server.shutdown()
def test_fetch_web_download_and_provenance(http_url, tmp_path):
out = tmp_path / "out"
r = _run(fetch_web(http_url, out))
assert r["ok"] is True
assert r["name"] == "test.fa"
assert r["size"] > 0
assert r["kind"] == "web"
assert r["provenance"]["source"] == http_url
assert len(r["provenance"]["sha256"]) == 64
assert (out / "test.fa").read_text().startswith(">seq1")
# sidecar provenance
assert (out / "test.fa.provenance.json").is_file()
def test_fetch_web_http_error(http_url, tmp_path):
with pytest.raises(FetchError) as ei:
_run(fetch_web(http_url + "/missing", tmp_path / "out"))
assert ei.value.code == "HTTP_ERROR"
def test_fetch_source_bad_scheme(tmp_path):
with pytest.raises(FetchError) as ei:
_run(fetch_source("ftp://host/x.fa", tmp_path))
assert ei.value.code == "BAD_SCHEME"
# ---------------------------------------------------------------------------
# OSS
# ---------------------------------------------------------------------------
def test_oss_url_no_endpoint_rejected():
with pytest.raises(FetchError) as ei:
_resolve_oss_url("oss://bucket/key.fa")
assert ei.value.code == "OSS_UNCONFIGURED"
def test_oss_url_endpoint_resolve(monkeypatch):
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-cn-hangzhou.aliyuncs.com")
url = _resolve_oss_url("oss://seq-bucket/data/x.fa")
assert url == "https://seq-bucket.oss-cn-hangzhou.aliyuncs.com/data/x.fa"
def test_oss_presigned_passthrough():
url = _resolve_oss_url("https://bucket.oss-cn-hangzhou.aliyuncs.com/x.fa?sig=abc")
assert url.startswith("https://")
# ---------------------------------------------------------------------------
# 局域网 / 本机路径
# ---------------------------------------------------------------------------
def test_parse_lan_remote():
assert _parse_lan_remote("192.168.1.5:/data/x.fa") == ("192.168.1.5", "/data/x.fa")
assert _parse_lan_remote("xzq@192.168.1.5:/data/x.fa") == ("xzq@192.168.1.5", "/data/x.fa")
assert _parse_lan_remote("/local/path/x.fa") is None
assert _parse_lan_remote("http://x/y") is None
def test_fetch_lan_local_path(tmp_path):
f = tmp_path / "genome.fa"
f.write_text(">g1\nACGT\n")
r = _run(fetch_lan(str(f), tmp_path / "out"))
assert r["ok"] is True
assert r["kind"] == "lan"
assert r["path"] == str(f)
assert r["provenance"]["sha256"]
def test_fetch_lan_missing_path(tmp_path):
with pytest.raises(FetchError) as ei:
_run(fetch_lan("/no/such/path.fa", tmp_path))
assert ei.value.code == "PATH_NOT_EXIST"
# ---------------------------------------------------------------------------
# 磁盘余量
# ---------------------------------------------------------------------------
def test_check_disk_space_insufficient(tmp_path):
ok, free = check_disk_space(tmp_path, 10 ** 18) # 1EB,必不足
assert ok is False
assert free < 10 ** 18
def test_check_disk_space_sufficient(tmp_path):
ok, _free = check_disk_space(tmp_path, 1) # 1 字节
assert ok is True
# ===========================================================================
# S0 补强(1)web:BAD_SCHEME / 磁盘满 DISK_FULL / 断点 TRUNCATED / NETWORK_ERROR
# ===========================================================================
def test_fetch_web_bad_scheme(tmp_path):
with pytest.raises(FetchError) as ei:
_run(fetch_web("ftp://host/x.fa", tmp_path / "out"))
assert ei.value.code == "BAD_SCHEME"
def test_fetch_web_disk_full(http_url, tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
monkeypatch.setattr(sf, "check_disk_space", lambda dest_dir, needed: (False, 0))
with pytest.raises(FetchError) as ei:
_run(fetch_web(http_url, tmp_path / "out"))
assert ei.value.code == "DISK_FULL"
def test_fetch_web_truncated(tmp_path, monkeypatch):
"""断点/不完整下载:aiter_bytes 实收字节 < Content-Length → TRUNCATED。
用 fake httpx 确定性触发(真实服务器下 httpx 会因短读抛 HTTPError)。
"""
import types as _types
import httpx as real_httpx
from app.services.bioflow import source_fetcher as sf
class FakeResp:
status_code = 200
headers = {"content-length": "100"}
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
async def aiter_bytes(self, chunk_size):
yield b"short"
class FakeStream:
async def __aenter__(self):
return FakeResp()
async def __aexit__(self, *a):
return False
class FakeClient:
def __init__(self, follow_redirects=True):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
def stream(self, method, url, timeout=None):
return FakeStream()
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
AsyncClient=FakeClient, Timeout=real_httpx.Timeout, HTTPError=real_httpx.HTTPError,
))
with pytest.raises(FetchError) as ei:
_run(fetch_web("http://x/y.fa", tmp_path / "out"))
assert ei.value.code == "TRUNCATED"
def test_fetch_web_network_error(tmp_path, monkeypatch):
"""网络错误:httpx.HTTPError → NETWORK_ERROR(fake 确定性触发)。"""
import types as _types
import httpx as real_httpx
from app.services.bioflow import source_fetcher as sf
class FailingStream:
async def __aenter__(self):
raise real_httpx.HTTPError("connection reset")
async def __aexit__(self, *a):
return False
class FailingClient:
def __init__(self, follow_redirects=True):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *a):
return False
def stream(self, method, url, timeout=None):
return FailingStream()
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
AsyncClient=FailingClient, Timeout=real_httpx.Timeout, HTTPError=real_httpx.HTTPError,
))
with pytest.raises(FetchError) as ei:
_run(fetch_web("http://x/y.fa", tmp_path / "out"))
assert ei.value.code == "NETWORK_ERROR"
# ===========================================================================
# S0 补强(2)OSS:BAD_OSS / endpoint 解析(内网 http)/ fetch_oss 走 web + kind 修正
# ===========================================================================
def test_oss_url_bad_format():
with pytest.raises(FetchError) as ei:
_resolve_oss_url("oss://onlybucket")
assert ei.value.code == "BAD_OSS"
def test_oss_url_intranet_endpoint_http(monkeypatch):
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-internal.example.com")
url = _resolve_oss_url("oss://seq-bucket/data/x.fa")
assert url == "http://seq-bucket.oss-internal.example.com/data/x.fa"
def test_fetch_oss_marks_kind_and_provenance(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
from pathlib import Path as _Path
async def fake_web(url, dest_dir, timeout=None):
dest = _Path(dest_dir) / "x.fa"
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(">x\nACGT\n")
return {"ok": True, "path": str(dest), "name": "x.fa", "size": 4,
"kind": "web", "provenance": {"source": url, "kind": "web",
"sha256": "0" * 64, "fetched_at": "t"}}
monkeypatch.setattr(sf, "fetch_web", fake_web)
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-cn-hangzhou.aliyuncs.com")
r = _run(sf.fetch_oss("oss://b/k.fa", tmp_path / "out"))
assert r["ok"] is True
assert r["kind"] == "oss"
assert r["provenance"]["kind"] == "oss"
assert r["provenance"]["oss_spec"] == "oss://b/k.fa"
# ===========================================================================
# S0 补强(3)fetch_source:EMPTY_SOURCE / oss 路由 / aliyuncs 预签名路由
# ===========================================================================
def test_fetch_source_empty(tmp_path):
with pytest.raises(FetchError) as ei:
_run(fetch_source("", tmp_path))
assert ei.value.code == "EMPTY_SOURCE"
def test_fetch_source_oss_routing(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
calls = []
async def fake_oss(spec, dest_dir, timeout=None):
calls.append(spec)
return {"ok": True, "kind": "oss"}
monkeypatch.setattr(sf, "fetch_oss", fake_oss)
r = _run(sf.fetch_source("oss://bucket/k.fa", tmp_path))
assert r["kind"] == "oss"
assert calls == ["oss://bucket/k.fa"]
def test_fetch_source_aliyuncs_presigned_routes_to_oss(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
calls = []
async def fake_oss(spec, dest_dir, timeout=None):
calls.append(spec)
return {"ok": True, "kind": "oss"}
monkeypatch.setattr(sf, "fetch_oss", fake_oss)
presigned = "https://bucket.aliyuncs.com/x.fa?sig=abc"
r = _run(sf.fetch_source(presigned, tmp_path))
assert r["kind"] == "oss"
assert calls == [presigned]
# ===========================================================================
# S0 补强(4)局域网:目录 IS_DIR / scp 拉取 / sshpass 缺失 / scp 失败 / 解析边界
# ===========================================================================
def test_fetch_lan_dir_rejected(tmp_path):
d = tmp_path / "adir"
d.mkdir()
with pytest.raises(FetchError) as ei:
_run(fetch_lan(str(d), tmp_path / "out"))
assert ei.value.code == "IS_DIR"
def test_parse_lan_remote_relative_rejected():
assert _parse_lan_remote("host:relative/path") is None
def test_fetch_lan_scp_success(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
from pathlib import Path as _Path
async def fake_scp(*args, **kwargs):
dest = _Path(args[-1])
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(">g\nACGT\n")
class P:
returncode = 0
async def communicate(self):
return (b"", b"")
return P()
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
r = _run(fetch_lan("xzq@192.168.1.5:/data/genome.fa", tmp_path / "out",
credentials={"user": "xzq"}))
assert r["ok"] is True
assert r["kind"] == "lan"
assert r["name"] == "genome.fa"
def test_fetch_lan_scp_sshpass_missing(tmp_path, monkeypatch):
import shutil
from app.services.bioflow import source_fetcher as sf
monkeypatch.setattr(shutil, "which", lambda name: None)
with pytest.raises(FetchError) as ei:
_run(fetch_lan("host:/data/x.fa", tmp_path / "out", credentials={"password": "p"}))
assert ei.value.code == "SSHPASS_MISSING"
def test_fetch_lan_scp_failed(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
async def fake_scp(*args, **kwargs):
class P:
returncode = 1
async def communicate(self):
return (b"", b"scp: no such file")
return P()
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
with pytest.raises(FetchError) as ei:
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
assert ei.value.code == "LAN_ERROR"
def test_scp_fetch_remote_uses_port_and_user(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
from pathlib import Path as _Path
captured = {}
async def fake_scp(*args, **kwargs):
captured["args"] = list(args)
dest = _Path(args[-1])
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(">g\nACGT\n")
class P:
returncode = 0
async def communicate(self):
return (b"", b"")
return P()
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
_run(fetch_lan("host:/data/x.fa", tmp_path / "out",
credentials={"user": "u", "port": "2222"}))
# scp -P 2222 u@host:/data/x.fa <dest>
assert captured["args"][1] == "-P"
assert captured["args"][2] == "2222"
assert captured["args"][-2] == "u@host:/data/x.fa"
def test_fetch_lan_scp_timeout(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
async def fake_scp(*args, **kwargs):
class P:
returncode = 0
async def communicate(self):
return (b"", b"")
return P()
async def fake_wait_for(coro, timeout=None):
coro.close() # 未 await 的 communicate 协程关闭,避免 never-awaited 警告
raise asyncio.TimeoutError()
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
monkeypatch.setattr(asyncio, "wait_for", fake_wait_for)
with pytest.raises(FetchError) as ei:
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
assert ei.value.code == "LAN_TIMEOUT"
def test_fetch_lan_scp_no_file(tmp_path, monkeypatch):
from app.services.bioflow import source_fetcher as sf
async def fake_scp(*args, **kwargs):
# returncode 0 但未产出目标文件
class P:
returncode = 0
async def communicate(self):
return (b"", b"")
return P()
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
with pytest.raises(FetchError) as ei:
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
assert ei.value.code == "LAN_ERROR"
+19
View File
@@ -0,0 +1,19 @@
#!/usr/bin/env bash
# bioflow_service/uninstall_skill.sh —— 卸载 bioflow-analysis skill(G12)。
#
# 用法:bash uninstall_skill.sh [skills_dir]
# - 移除 skills_dir/bioflow-analysis/
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
SKILL_NAME="bioflow-analysis"
SKILLS_DIR="${1:-$HERE/../.claude/skills}"
DEST="$SKILLS_DIR/$SKILL_NAME"
if [ -d "$DEST" ]; then
rm -rf "$DEST"
echo "[uninstall] removed $DEST"
else
echo "[uninstall] $DEST 不存在(已卸载)"
fi
echo "DONE"
+91
View File
@@ -0,0 +1,91 @@
#!/bin/bash
# deploy/bioflow_switch_restart.sh —— 切分支后重启生信相关服务并验证
#
# 背景:QA(8091)/前端(5180)/bioflow_service(8098) 都是「启动时刻」快照,
# 切分支后不重启就会跑旧分支的代码/配置(曾导致「设置面板无生信开关」+
# 「规划降级」两个问题)。本脚本把「切分支→重启→验证」固化,避免再踩。
#
# 用法:bash deploy/bioflow_switch_restart.sh [分支名]
# 分支名缺省 feature/bioflow-skill-orchestration
#
# 重启:QA(8091) + bioflow_service(8098) + 前端 vite(5180)
# BFF(8090) 不重启:/api/capabilities 每次请求动态拉 QA registry,QA 重启后自动生效
# 验证:服务健康 + QA registry 生信能力 + 前端 SettingsPanel 生信三开关
#
# 关键点(勿删):
# - QA 需 source agent/qa/env/prod.env + 强制 QA_V1_ENABLE_BIOFLOW_AGENT=1(熔断开)
# - 8098 裸启动即可:main.py 已内置 load_dotenv,自动加载 bioflow_service/.env
# - 前端 pkill 用「zh/PlantAI-rag-new/ui/」精确匹配,避免误杀 5181(wjq 仓库) 的 vite
set -uo pipefail
ROOT="/home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new"
VENV="/home/xzq/5090/teamWorkspace/zh/plantai-rag-new-venv/bin/python"
BRANCH="${1:-feature/bioflow-skill-orchestration}"
PASS=0; FAIL=0
ok() { PASS=$((PASS+1)); echo " ✅ $1"; }
bad() { FAIL=$((FAIL+1)); echo " ❌ $1"; }
echo "======== 1. 切分支 ========"
cd "$ROOT" || { echo "仓库目录不存在: $ROOT"; exit 1; }
git checkout "$BRANCH" || { echo "切分支失败: $BRANCH"; exit 1; }
echo "当前分支: $(git branch --show-current)"
echo ""
echo "======== 2. 重启 QA (8091) ========"
pkill -f "scripts.qa_v1_1_73_server" 2>/dev/null; sleep 2
cd "$ROOT/agent/qa"
set -a; source env/prod.env; set +a
export QA_V1_ENABLE_BIOFLOW_AGENT=1 # 生信对话熔断开关,切生信分支必须开
nohup "$VENV" -m uvicorn scripts.qa_v1_1_73_server:app --host 127.0.0.1 --port 8091 \
> /tmp/plantai-qa-8091.log 2>&1 < /dev/null &
disown
echo ""
echo "======== 3. 重启 bioflow_service (8098) ========"
pkill -f "app.main:app.*port 8098" 2>/dev/null; sleep 2
cd "$ROOT/bioflow_service"
nohup "$VENV" -m uvicorn app.main:app --host 127.0.0.1 --port 8098 \
> /tmp/plantai-bioflow-8098.log 2>&1 < /dev/null &
disown
echo ""
echo "======== 4. 重启前端 vite (5180) ========"
pkill -f "zh/PlantAI-rag-new/ui/node_modules/.bin/vite" 2>/dev/null; sleep 2
cd "$ROOT/ui"
VITE_BFF_TARGET="http://127.0.0.1:8090" setsid npm run dev \
> /tmp/plantai-vite.log 2>&1 < /dev/null &
disown
echo ""
echo "等待服务启动 (12s)..."
sleep 12
echo ""
echo "======== 5. 验证 ========"
echo "[健康检查]"
c=$(curl -s -m 3 -o /dev/null -w "%{http_code}" http://127.0.0.1:8098/docs 2>/dev/null)
[ "$c" = "200" ] && ok "8098 bioflow_service /docs=$c" || bad "8098 bioflow_service /docs=$c(见 /tmp/plantai-bioflow-8098.log)"
c=$(curl -s -m 3 -o /dev/null -w "%{http_code}" http://127.0.0.1:8091/api/v1/qa-v1-1-73/capabilities/registry 2>/dev/null)
[ "$c" = "200" ] && ok "8091 QA registry=$c" || bad "8091 QA registry=$c(见 /tmp/plantai-qa-8091.log)"
c=$(curl -s -m 3 -o /dev/null -w "%{http_code}" http://127.0.0.1:5180/ 2>/dev/null)
[ "$c" = "200" ] && ok "5180 前端 =$c" || bad "5180 前端 =$c(见 /tmp/plantai-vite.log)"
echo "[生信能力注册]"
REG=$(curl -s -m 5 http://127.0.0.1:8091/api/v1/qa-v1-1-73/capabilities/registry 2>/dev/null)
BIO=$(echo "$REG" | grep -o '"name":"bioflow-[a-z-]*"' | wc -l)
[ "$BIO" -ge 3 ] && ok "QA registry 生信能力 $BIO 个(期望 3)" || bad "QA registry 生信能力仅 $BIO 个(期望 3)"
echo "[前端生信三开关]"
SW=$(curl -s -m 5 -H "Accept: */*" "http://127.0.0.1:5180/src/components/settings/SettingsPanel.vue" 2>/dev/null | grep -c "settingsBioflow\|bioflowCaps")
[ "$SW" -ge 2 ] && ok "前端 SettingsPanel 含「生信分析能力」三开关" || bad "前端 SettingsPanel 无生信三开关(可能旧代码,需硬刷新浏览器)"
echo ""
echo "======== 结果: $PASS 通过 / $FAIL 失败 ========"
if [ "$FAIL" -eq 0 ]; then
echo "✅ 全部就绪。浏览器请 Ctrl+Shift+R 硬刷新(清缓存),再测生信对话。"
else
echo "❌ 有失败项,请查看 /tmp/plantai-qa-8091.log /tmp/plantai-bioflow-8098.log /tmp/plantai-vite.log"
fi
exit "$FAIL"
+141
View File
@@ -0,0 +1,141 @@
#!/usr/bin/env bash
# deploy/toggle_bioflow.sh —— 生信能力按需装卸(热,不重启 QA)+ bioflow_service 启停 + 注册状态自检。
#
# 用法:
# bash deploy/toggle_bioflow.sh <gene-family|rnaseq|extension|all> <on|off>
#
# 原理:
# - 能力装卸经 BFF POST /api/capabilities/install(透传 QA install_manager,热加载生效不重启 QA)
# - on = register(manifest 校验 + 原子写 capabilities.yaml + registry 差分刷新)
# - off = unregister(按 name 移除能力段,其余能力不受影响)
# - 自检 = GET /api/capabilities 的 installed 数组是否含对应能力
# - bioflow_service 启停:全部能力关闭时停 8098;任一开启时启动 8098
# - 卸载活跃任务:三表已持久化,服务恢复后 runner.recover 续跑;QA 侧按「生信服务未启动」降级话术
set -uo pipefail
BFF="http://127.0.0.1:8090"
BIOFLOW_SERVICE_DIR="/home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new/bioflow_service"
PY="/home/xzq/5090/teamWorkspace/zh/plantai-rag-new-venv/bin/python"
BIOFLOW_PORT="${BIOFLOW_PORT:-8098}"
CAP="${1:-}"
ACTION="${2:-}"
CAPS=(gene-family rnaseq extension)
usage() {
echo "用法: $0 <gene-family|rnaseq|extension|all> <on|off>" >&2
exit 2
}
# capability key -> registry 能力名
cap_name() {
case "$1" in
gene-family) echo "bioflow-gene-family" ;;
rnaseq) echo "bioflow-rnaseq" ;;
extension) echo "bioflow-extension" ;;
*) usage ;;
esac
}
# capability key -> register manifest YAML(与 agent/qa/configs/capabilities.yaml 一致)
manifest() {
case "$1" in
gene-family) cat <<'YAML'
capabilities:
- name: bioflow-gene-family
provider: mcp
permission: read_only
description: 基因家族分析(路径规划/建任务/参数确认/文件获取与预处理/状态查询),自然语言触发
session_default: true
nl_triggers: [基因家族, 家族成员, 亚家族, 系统发育, 建树, 进化树, 保守基序, motif, 结构域, 基因结构, 共线性, Ka/Ks, 选择压力, 组合证据]
mcp:
url: http://127.0.0.1:8098/mcp/gene-family/
prompt_pack:
file: bioflow_gene_family_agent_rules.md
YAML
;;
rnaseq) cat <<'YAML'
capabilities:
- name: bioflow-rnaseq
provider: mcp
permission: read_only
description: 转录组分析(实验设计确认/样本配置/路径规划/建任务/差异表达/富集),自然语言触发
session_default: true
nl_triggers: [转录组, RNA-seq, RNAseq, rnaseq, 差异表达, 表达定量, 富集, WGCNA, GSEA, 实验设计, fastq]
mcp:
url: http://127.0.0.1:8098/mcp/rnaseq/
prompt_pack:
file: bioflow_rnaseq_agent_rules.md
YAML
;;
extension) cat <<'YAML'
capabilities:
- name: bioflow-extension
provider: mcp
permission: read_only
description: 用户自定义扩展分析(创建/转换/试运行/校验/发布/运行),自然语言触发
session_default: true
nl_triggers: [扩展, 拓展分析, 扩展分析, 自定义分析, 分析模板, 分析脚本, 分析工具]
mcp:
url: http://127.0.0.1:8098/mcp/extension/
prompt_pack:
file: bioflow_extension_agent_rules.md
YAML
;;
*) usage ;;
esac
}
[ -n "$CAP" ] && [ -n "$ACTION" ] || usage
[ "$ACTION" = "on" ] || [ "$ACTION" = "off" ] || usage
if [ "$CAP" != "all" ]; then
cap_name "$CAP" >/dev/null 2>&1 || usage
fi
# 1. 逐能力 register/unregister(热加载不重启 QA)
targets=("$CAP")
[ "$CAP" = "all" ] && targets=("${CAPS[@]}")
for key in "${targets[@]}"; do
name="$(cap_name "$key")"
if [ "$ACTION" = "on" ]; then
body="$(manifest "$key")"
resp=$(curl -s -m 30 -X POST "$BFF/api/capabilities/install" \
-H 'Content-Type: application/json' \
-d "$(python3 -c 'import json,sys; print(json.dumps({"action":"register","manifest":sys.stdin.read()}))' <<<"$body")")
else
resp=$(curl -s -m 30 -X POST "$BFF/api/capabilities/install" \
-H 'Content-Type: application/json' \
-d "{\"action\":\"unregister\",\"name\":\"$name\"}")
fi
echo "[$ACTION] $name -> $(echo "$resp" | head -c 300)"
done
# 2. 自检:GET /api/capabilities 的 installed 数组
installed=$(curl -s -m 10 "$BFF/api/capabilities")
echo "注册状态: $(echo "$installed" | python3 -c 'import json,sys; d=json.load(sys.stdin); print([i.get("name") for i in d.get("installed") or []])' 2>/dev/null || echo 'parse-fail')"
# 3. bioflow_service 按需启停:任一能力开启则启动,全部关闭则停止
any_on=false
for key in "${CAPS[@]}"; do
name="$(cap_name "$key")"
if echo "$installed" | grep -q "\"$name\""; then
any_on=true
fi
done
if [ "$any_on" = true ]; then
if ! lsof -ti tcp:$BIOFLOW_PORT -sTCP:LISTEN >/dev/null 2>&1; then
echo "启动 bioflow_service :$BIOFLOW_PORT"
( cd "$BIOFLOW_SERVICE_DIR" && setsid "$PY" -m uvicorn app.main:app --host 127.0.0.1 --port "$BIOFLOW_PORT" \
> /tmp/plantai-bioflow.log 2>&1 < /dev/null & ) && disown -a
sleep 2
fi
curl -s -m 8 "http://127.0.0.1:$BIOFLOW_PORT/api/bioflow/health" | head -c 150
echo
else
echo "全部能力已卸载:停止 bioflow_service :$BIOFLOW_PORT"
kill $(lsof -ti tcp:$BIOFLOW_PORT -sTCP:LISTEN) 2>/dev/null || true
fi
echo "DONE"
+177
View File
@@ -0,0 +1,177 @@
# 生信流程 · 对话式手工测试手册(S6 更新版)
> 供用户逐项实测。生信模块已从「卡片式交互」改造为「自然语言对话式交互 + MCP 外挂」,
> 本手册以**对话剧本**形式给出每个场景的:对话示例 → 系统应如何回应 → 通过判据。
> 环境:内网 http://192.168.0.175:8090(BFF);生信服务 8098、QA 8091 已部署;
> 三能力(基因家族/转录组/扩展)按 §0 装卸。
---
## 0. 前置条件
1. 浏览器打开 http://192.168.0.175:8090。
2. **能力安装检查**:设置面板(SettingsPanel)应能看到三个生信开关(基因家族/转录组/扩展)。
打开任意一个后,工作区出现「生信任务」「生信档案」TAB;全关则 TAB 消失(**TAB 显隐 = 能力注册状态**,非服务健康)。
3. 服务器 docker 已有 8 个 bioflow 镜像;金标数据在 `bioflow_kernel/assets/examples/test_input/`(Ft 四件套 pep/cds/genome/gff3)。
4. 熔断开关:`QA_V1_ENABLE_BIOFLOW_AGENT=1`(默认 0,关时生信句走普通 QA)。
---
## 1. 多配置开关回归(S6)
| 配置 | 操作 | 期望 | 判据 |
|---|---|---|---|
| 全关(熔断=0 + 三能力卸载) | 发「帮我做基因家族分析」 | 普通 QA 礼貌回答(说明未安装生信能力),**无**路径卡 | 无生信分流 |
| 只开基因家族 | 发「帮我做基因家族分析」 | 进入生信对话(列路径+挂路径卡) | 命中 |
| 只开基因家族 | 发「帮我做转录组差异表达」 | 走普通 QA(转录组能力未装) | 不响应 |
| 只开转录组 | 发「帮我做转录组」 | 进入生信对话 | 命中 |
| 只开转录组 | 发「帮我做基因家族分析」 | 走普通 QA | 不响应 |
| 全开但非生信提问 | 发「光合作用的电子传递链是什么」 | 走普通 QA(非生信意图不调工具) | 不调工具 |
---
## 2. 对话剧本 A:基因家族分析(卡片辅助通道)
**第 1 轮 —— 需求与路径**
> 用户:帮我做水稻 SPL 基因家族分析
> 系统(自然语言):列「核心流程(必选)+ 可选分析」,说明每个可选项生物学意义;
> **同帧挂出路径选择卡**。
> 判据:回复含路径列表 + 路径卡出现(卡片不替代自然语言)。
**第 2 轮 —— 卡片操作 → 系统复述**
> 用户(路径卡勾选「核心流程 + Ka/Ks + 启动子」,点确认;打字补充文件位置)
> 系统:先自然语言**复述**用户选择("您选择了核心流程 + Ka/Ks + 启动子分析…"),再请确认。
> 判据:系统复述内容与卡片勾选一致;建任务前必有一轮复述确认(不直接建任务)。
**第 3 轮 —— 网络文件 + 非标准文件预处理**
> 用户:文件在 `https://…/rice.fa` 与 `oss://seq-bucket/rice.gff3`,另传 `spl_list.xlsx`
> 系统:拉取 web/OSS 文件(**下载前检查磁盘余量、记录 provenance**);对 `spl_list.xlsx`
> 做内容嗅探后**自然语言说明列结构并提问**("推测第二列是蛋白序列,备注列忽略?")。
> 判据:文件获取成功;Excel 触发预处理提问(列表方式,可卡片点选也可打字回答)。
**第 4 轮 —— 建任务执行**
> 用户:都对,备注列不要;确认,开始吧
> 系统:复述整理结果("已整理为 FASTA 3 条序列")→ 创建任务 → 消息内出现**只读精简进度块**,
> 右侧「生信任务」TAB 完整滚动日志实时跟随。
> 判据:进度块只读(无按钮);TAB 日志自动滚底。
**执行中 —— 进度转述 + 随时提问**
> 用户:什么是组合证据?
> 系统:经流程知识库自然语言回答(带引用),**不打断任务**。
> 判据:提问得到引用回答,任务继续推进。
**人工确认边界(不可跳过)**
> 系统:第 8 步亚家族预测完成(6 个亚家族)。⚠ 需您人工确认,我不能代您确认。
> 用户:把 SPL3 从亚家族 A 移到 B → 系统复述修改 → 用户「确认」。
> 判据:subfamily_confirm 必停、必复述、绝不代确认。
**收尾 —— 存档**
> 系统:全部完成(19 成员 6 亚家族),同帧挂出存档卡。
> 用户(存档卡填名称点「存档」或打字「存档,叫 SPL 水稻鉴定」)
> 系统:实际执行 `archive_job` 存档(复用 REST 语义),复述存档三要素(名称/位置/大小)。
> 判据:存档成功(重名提示 ARCHIVE_EXISTS)。
---
## 3. 对话剧本 B:转录组分析(纯自然语言通道,全程不点卡片)
**第 1 轮 —— 必确认信息**
> 用户:帮我做转录组,比较干旱处理和对照
> 系统:列主流程 + 可选;**必须确认**四项(物种/参考基因组版本、FASTQ 单双端、
> 对照处理组重复数、链特异性),列表展示。
> 判据:实验设计确认边界被明确列出,未确认前不得建差异表达任务。
**第 2 轮 —— OSS 文件 + 样本信息(纯打字,不点卡)**
> 用户:数据在 oss://seq-bucket/drought/,水稻日本晴,干旱 3 重复对照 3 重复,双端非链特异
> 系统:从 OSS 拉取 12 个 FASTQ 并核对成对;复述样本设计("干旱 3 vs 对照 3,双端,非链特异"),
> 请确认基线(对照为基线、比较方向「干旱 vs 对照」)。
> 判据:纯自然语言路径走通(**卡片与打字两条通道服务端语义一致**)。
**第 3 轮 —— 执行 + 结果转述**
> 用户:对 → 系统执行 QC→比对→定量→样本质控→差异表达→富集,逐段转述(接头率/Q30/唯一比对率/差异基因数/PCA 分离)。
> 判据:每步完成后自然语言转述关键指标;样本质控 PCA 结论有转述。
---
## 4. 三种网络文件来源用例
| 来源 | 示例 | 期望 |
|---|---|---|
| web | 「文件在 https://…/x.fa」 | httpx 流式下载,不设大小上限;下载前查磁盘余量;记录 provenance |
| OSS | 「数据在 oss://bucket/key 或预签名 URL」 | 经 BIOFLOW_OSS_ENDPOINT 解析或预签名直通;无 endpoint 时提示改用预签名 |
| 局域网 | 「文件在 192.168.x.x:/data/x.fa,账号 xx 密码 xx」 | ssh/scp 拉取(凭据单次会话内存用,不落盘、审计不记明文) |
判据:文件落 `bioflow_data/jobs/{job}/input/`,先内容嗅探后登记,provenance 记录来源/哈希/时间。
---
## 5. 非标准文件预处理(Excel / 乱码文本)闭环
1. 上传 `xxx.xlsx` / 乱码 txt → 系统 `inspect_file` 探测(列结构/格式猜测/乱码检测)。
2. 系统自然语言说明 + 列表提问(哪列是序列、是否去重、是否去表头)。
3. 用户回答 → `transform_file` 转换(LLM 定规则 + 脚本执行,大文件抽样定规则)。
4. 结果过格式验证器(fasta 可解析/gff3 9 列/tsv 列一致/samples.yaml schema);
失败说明问题并追问,循环 ≤3 次后建议人工整理。
5. 转换后 `register_job_files` 登记,provenance 记「原始→规则→产物」链。
判据:全程自然语言引导;转换产物格式校验通过;原文件永不覆盖。
---
## 6. 卡片辅助 vs 纯自然语言双通道等价
| 场景 | 卡片通道 | 纯 NL 通道 | 等价判据 |
|---|---|---|---|
| 路径选择 | 路径卡勾选 | 打字「核心流程 + Ka/Ks」 | 两者复述的确认内容一致,建任务入参一致 |
| 参数确认 | 参数卡填值 | 打字「用 8 核、padj 用 0.01」 | confirm_stage 入参一致 |
| 文件提供 | 文件卡拖拽/填路径 | 打字「文件在 …」 | register_job_files / fetch_source_file 入参一致 |
| 存档 | 存档卡点存档 | 打字「存档,叫 X」 | archive_job 入参一致 |
判据:卡片操作产生的是 `card_result` 结构化消息进 QA 流(**不直连 bioflow 控制面**),
与打字在服务端语义完全一致(QA 对话状态机是唯一决策点)。
---
## 7. 旧会话兼容(Legacy 块只读)
1. 打开一个改造前创建、含旧生信块(bioflow_qa / 旧 bioflow_* 块)的会话。
2. 期望:旧块以只读形式降级渲染(标题 + 只读引用/提示),**不报错、无可交互控件**。
---
## 8. 扩展子系统对话化(第三能力单元)
1. 打开「扩展」能力开关。
2. 对话创建扩展:描述分析脚本(代码文件 + 运行命令)→ `create_extension`(draft)。
3. `build` → `trial_run` → `validate` → `confirm` → `publish` → `run` 逐步走(卡片或打字等价)。
4. 试运行失败 → 「AI 修复」触发 `repair_extension`(修复上限 20 轮);校验失败/状态不足时系统给出确定性提示。
判据:扩展生命周期全程对话化;非法状态返回清晰提示(EXT_STATUS),不报 500。
---
## 9. 后端金标验证(服务器命令行,S6 回归)
```bash
cd /home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new
BIOFLOW_DATA=/home/xzq/5090/teamWorkspace/zh/bioflow-data \
bash bioflow_kernel/tests/smoke_golden.sh bioflow-gf:dev
```
期望:4 步全绿——python 3.11.14 / ldd 0 缺库 / 无 conda / 金标 identify 24 ID diff 空(`GOLDEN_DIFF_OK 24 IDs`)。
---
## 10. 端到端主线(一次性走全流程,双通道各一遍)
1. 卡片通道:§2 全程(路径卡→文件卡→参数卡→进度块→存档卡)。
2. 纯 NL 通道:§3 全程(不点任何卡片)。
3. 含一次 Excel 预处理、一次局域网/OSS 文件拉取。
4. 任务完成后存档 → 生信档案 TAB 回看 → 复用发起。
期望:两条通道均走通,7 项引导(路径/文件/参数/报告/存档/复用/过程展示)逐项达成,
进度块实时流转、右侧日志滚底。
---
*本手册为 S6 对话式改造更新版,替代旧卡片式 `MANUAL_TEST_BIOFLOW.md` 的生信流程部分。*
@@ -1,42 +0,0 @@
# feat(review): journal_directory 启动自动 seed(部署即运行)
## 背景
source_v4.xlsx(1056 条目录)此前需手动跑 import_journals.py 入库,部署后忘跑即出现"目录数据未导入"。本次实现启动时自动检测 + 自动导入。线上当前环境已由人工完成导入(review.db 内 1056 行),本功能消除的是新环境部署/重建库时的手动步骤风险。
## 修改内容
| 文件 | 变更 |
|---|---|
| `app/services/journal_seed.py`(新增) | 共享核心:schema/清洗/解析/upsert 从 CLI 原样迁入;`extract_rows_from_xlsx()`(口径 779/277 与 id 冲突闸门,不符不写库)、`upsert_to_db()`、`ensure_journal_directory_seeded()`(空表才导,四键 dict 契约,绝不抛异常) |
| `scripts/import_journals.py` | 瘦身为薄 CLI(-431/+92 行):对账报告/退出码 0/1/2/3 全保留,核心复用 journal_seed |
| `app/config.py` | +`seed_on_startup=True`(REVIEW_SEED_ON_STARTUP 可关)、+`journal_directory_seed_path` |
| `app/main.py` | lifespan:开关门控 → `asyncio.to_thread` 导入(不堵 event loop)→ try/except 不阻塞启动;路径走 `db.path` 注入链 |
| `tests/conftest.py`(新增) | 全局关 seed,防止单测触碰真库/拖慢套件 |
| `tests/test_seed.py`(新增) | 6 用例(见下) |
不动:source_v4.xlsx、journal_directory 路由/表 schema、db.py、前端。
## 测试用例与结果(pytest 78 passed,新增 6 条)
1. 空库 → seed 成功,rows=1056(en=779/zh=277)✓
2. 非空库(1 行)→ skip,count 不变 ✓
3. xlsx 不存在 → 返回 error,不抛异常,不写库 ✓
4. 幂等:连调两次,第二次 skip ✓
5. 开关 off(lifespan)→ 表不创建 ✓
6. 开关 on(lifespan)→ tmp 库 1056 行 ✓
## 部署验证
```bash
bash review_service/restart.sh # 非空库:日志 "already has 1056 rows, skip"
curl 'http://127.0.0.1:8100/api/review/journals/directory?lang=en' # total=779
# 空库冷启动(本机已验证等价路径):日志 "seeded 1056 rows (en=779, zh=277, N.Ns)"
# 特殊部署:REVIEW_SEED_ON_STARTUP=false → 日志 "seed skipped"
```
本机已实测:真机重启走 skip 路径(`INFO:app.main:already has 1056 rows, skip`),CLI 薄壳回归 exit=0 / `en=779 zh=277 → ✓ 相符`。
## 是否需要立即部署
待管理员确认。线上已手动导入 1056 行,本改动属部署便利性补强(消除新环境忘跑导入脚本的风险),可随下次正常发版评估带上。
@@ -1,56 +0,0 @@
# fix(review): 冲突降级真影响 recommendation(major → minor 真降级)
- 分支:`feature/academic-review-assistant`
- 日期:2026-10-12
- 服务:`review_service`(:8100)/ 前端 `PresubmitPanel.vue`
- 状态:**未 commit**,待验收
## 背景
s41588(已发表 Nature Genetics 论文)真实链路里,关键词扫描器报告"3 处冲突已降级为非硬性问题",但 `recommendation` 仍为大修、`major_weaknesses` 仍 5 条。诊断确认:原降级只做了"剥 hard_rule 标签 + 记 conflicts",**条目仍在 major 列表内**,`major_count` / 引擎输入未变,故判定不变。本次把"降级"从标签维度升级为**列表维度的真降级**(major → minor)。
## 变更
**后端**
- `paper_scanner.reconcile_hard_rules`:命中冲突的 major 从列表**移出**,`hard_rule=None` + `scanner_downgraded=true`,落入 demoted 列表;`fatal=true` 条目例外(只记 conflict 不降级,须人工过目)。
- `routers/presubmit.py`:demoted 追加进 `report.minor_weaknesses`;`major_weaknesses` 为过滤后列表;explanation 文案改"已降级为小修级问题"。
- `self_review.WeaknessItem`:新增 `scanner_downgraded: bool = False`(coerce 不读、LLM 无法伪造)。
- 引擎 `recommend_engine.decide_recommendation` **零改动**——计数因列表变短而自动真实。
**前端**
- `PresubmitPanel.vue`:minor 卡片对 `scanner_downgraded` 显示黄徽标"冲突降级"(与 major 卡片"硬性"红徽标对仗);冲突明细块保留。
## 测试
- pytest **104 passed**(新增:`test_reconcile_fatal_not_demoted`、`test_http_partial_conflict_leaves_residual_soft`;改写 `test_reconcile_*` 与 `test_http_scanner_conflict_downgrades` 为真降级断言)。
- vitest **382 passed / 5 failed**(5F 为既有他人 `benchpanel`+`Composer .mode-pill` labels.js 引发,与本改动无关);`npm run build` ✓。
## 验收(真 LLM,:8100)
| 样本 | rec | rule | M/S/minor | conflicts | downgraded→minor |
|---|---|---|---|---|---|
| diag_strong | minor_revision | R5 | 0/0/4 | 0 | 无 |
| diag_weak | major_revision | R1 | 6/2/7 | 1(hic) | 有(真降级生效) |
弱稿仍大修:4 条硬性命中(busco/accession/voucher/abs)为真缺,扫描器无冲突、不降级——真缺陷不放行。
## 验收:s41588(真 LLM,:8100)
输入 `/tmp/opencode/s41588-025-02113-5.pdf`,target_journal=Genome Biology(tier 1 override)。
| 项 | 修复前 | 修复后 |
|---|---|---|
| recommendation | major_revision | **minor_revision**(R5) |
| major_count / soft / minor | 5 / — / — | **0 / 0 / 6** |
| hard_hits | 含 busco/accession… | **[]** |
| scanner signals | — | accession_found=True, busco=partial, hic_found=True |
| major_weaknesses 列表 | 5 条 | **0 条** |
| scanner_conflicts | — | **2 条保留**(busco_missing 格式不完整、accession_missing 部分未给号)→ 均真降级进 minor(`scanner_downgraded=true`) |
判定链:扫描器证伪 2 条硬性 → 真降级 → 剩余为 minor → R5(tier1 accept 门槛 minor≤1,6 条 → minor_revision)。conflicts 透明保留、explanation 明示"已降级为小修级问题,请人工核对"。
注:本次 LLM 未再把"树图注未标数值 / GWAS 校正描述"判成 major(负向清单 Prompt 生效),故 major 从 5 归 0,优于预期的"降到 2"。
## 部署
待管理员确认。属判定正确性修复(真论文误判大修),建议随下次正常发版带上。
@@ -1,81 +0,0 @@
# fix(review): presubmit 四 Bug 修复——截断保真 / 分档 / 词表 / 扫描器兜底
## 背景
真实案例:已发表 Nature Genetics 论文(s41588-026-02682-z,黄瓜泛基因组)被判
"大修",但论文实有 BUSCO 98.57%(P2)、94.18% sequences anchored(P2)、
PRJCA038097 等 accession(P17 Data Availability)。诊断出四个独立缺陷:
1. **PDF 截断**:`max_text_chars=60000` 从头切,30 页论文 P17 的 Data Availability 被截掉;
2. **期刊分档**:Nature Genetics 不在 5 本精选 → 默认 tier 2(应 tier 1);
3. **硬性规则词表过窄**:缺 PRJCA/CRA(中国 NGDC/GSA)、Hi-C 同义词只认 anchoring rate、
BUSCO 仅百分比也算缺失;
4. **无程序层兜底**:hard_rule 全靠 LLM 输出,无关键词证伪机制。
## 修改内容
| 文件 | 变更 |
|---|---|
| `app/routers/presubmit.py` | **Bug 1**:段落感知装箱 `pack_feed_text`(超预算按
Abstract→Data Availability 全文→Methods 前 60%→Results→Discussion 优先保留,
dropped_sections 入 meta);无节标题回退头+尾窗(捞页尾 availability 块)。
**双层保护**:估算 token(len/1.5)>48K → 自动降 100K 重装;LLM 400/413 → 降 80K
重试一次。**Bug 4 接线**:coerce 后扫描器证伪 → rationale.scanner_conflicts |
**Bug 2**:tier_source ∈ override/featured/jif/默认 |
| `app/services/paper_scanner.py`(新增) | 关键词扫描器(纯 stdlib):accession
词表 PRJNA/PRJEB/PRJCA/SRP/ERP/DRP/CRA/SRR/ERR/DRR/GCA_/GCF_ **只在 Data Availability
段扫**(正文引用他人数据不误判);BUSCO 三级 none/partial/full;Hi-C 同义词
(anchored to chromosome(s)/anchoring rate/pseudo-chromosome(s)/chromosome-level)。
`reconcile_hard_rules`:LLM 报缺 X + 扫描器见 X → 剥硬性标签降 soft + 记冲突
(只证伪不证实,不引入新误判) |
| `app/services/recommend_engine.py` | **Bug 2**:tier 查找优先级 override
(13 本顶刊硬编码,"The " 前缀归一化)→ featured(journals.json)→ jif 启发式
(journal_directory 只读 jif 列;jif≥15→tier1;综述刊名含 Review/Trends/Annual
Review 除外)→ 默认 tier 2 |
| `app/services/self_review.py` | **Bug 3**:Prompt 词表扩充(PRJCA/CRA/SRR…);
BUSCO 三级判定(仅百分比→minor 不打 hard_rule);Hi-C 同义词清单。
**2026-10-12 追加 soft major 收紧**(s41588 复盘):维度三系统发育树分诊重写
(完全未做→major / 已声明仅图未标数值→presentation / 附带树→minor);新增
【负向清单】节(图未标统计量、标准方法描述不全、阈值缺数值、引用排版 →
minor/presentation,严禁列 major;总纲"只有补实验/补数据/重跑分析/改核心论证
才是 major";与各维度冲突时以本节为准) |
| `app/config.py` | `max_text_chars` 60000 → **150000**(REVIEW_MAX_TEXT_CHARS 可覆盖) |
| `ui/.../PresubmitPanel.vue` | 冲突黄标块(scanner_conflicts + 降级提示);截断
dropped_sections 美化提示;截断文案改为"保摘要/数据可用性/方法/结果/讨论" |
| `tests/test_presubmit.py` | 新增 15 用例(装箱/回退/token 守卫/400 重试/扫描器
三类信号/DA 段限定/冲突降级/端到端/tier 优先级矩阵/Prompt 新词表) |
## 测试用例与结果
- **pytest:102 passed**(87 基线 + 15 新增)。
- **vitest:379 passed / 5 failed**(5F 为既有他人 i18n labels 改动引发,与本改动
无关;新增冲突黄标/截断提示 2 条全绿)。`npm run build` ✓。
- **真实链路验收**(真 LLM,target_journal=Genome Biology → tier 1 override):
| 样本 | 修复前 | 修复后 | 扫描器 |
|---|---|---|---|
| 强稿 diag_strong.pdf | minor_revision | **minor_revision**(R5,0/0/5) | accession/busco=full/hic 全命中,0 冲突 ✓ |
| 弱稿 diag_weak.pdf | major_revision | **major_revision**(R1,硬性 4 + soft 4 + minor 5) | accession/busco 未命中(真缺不降);Hi-C 冲突降级并记录 ✓ |
| **真实案例 s41588**(NG 论文) | major_revision | 见下"验收"节 | — |
## 部署验证
```bash
bash review_service/restart.sh
curl -X POST http://127.0.0.1:8100/api/review/presubmit \
-F "file=@paper.pdf" -F "target_journal=Nature Genetics"
# 检查:meta.tier=1/tier_source=override;meta.dropped_sections;
# report.recommendation_rationale.scanner_conflicts
```
## 是否需要立即部署
待管理员确认。四修复均为判定正确性问题(真实发表论文误判大修),建议随下次正常发版带上。
## 遗留
- tier jif 启发式仅覆盖 journal_directory 已录入的 1056 刊;未收录刊名走默认 tier 2。
- 扫描器 accession 仅扫 DA 段——若论文把 accession 放 Methods 数据声明处且无 DA 节,
扫描器不证伪(保守方向,LLM 硬性判定仍生效,不会误放行)。
- BUSCO full 判定要求 S: 与 D: 标记在 200 字符窗内相邻——极端排版可能判 partial
(只会更严,不误放)。
@@ -1,61 +0,0 @@
# fix(review): presubmit recommendation 判定过严修复(配额制 → 规则引擎)
## 背景
用户反馈 AI 预审稿多次测试均返回 major_revision。2026-10-12 诊断确认根因是
**配额制**而非"尺子紧":
- `self_review.py` 硬性规则要求 `major_weaknesses 3-8 条`(强制配额)→ accept/minor_revision 几乎不可达;
- LLM 直接输出 recommendation,无程序层规则;JSON 示例锚定 `major_revision`、coerce 缺省回落 `major_revision`;
- 对照实验:近 GB 完备强稿(BUSCO 96.8% / Hi-C 98.7% / PRJNA+GCA+figshare 齐 / UFBoot 1000)仍被凑出 6 条 major,与弱稿(7 条)几乎无区分度;
- `target_journal` 传入但 Prompt 零分档标准。
## 修改内容
| 文件 | 变更 |
|---|---|
| `app/services/recommend_engine.py`(新增) | 纯函数规则引擎:R0 fatal→reject / R1 硬性 major→大修(不可协商,与 tier 无关)/ R2' tier3 仅 1-2 条 soft 可降小修 / R2 soft≥1→大修 / R4 0 major 且 minor≤tier 阈值→接收 / R5→小修;rationale 模板拼装(rule/hard_hits/计数/tier/explanation,含 tier 降级提示句);`resolve_tier` 按 journals.json `review_tier` 匹配,失败默认 tier 2 |
| `app/services/self_review.py` | Prompt 重写:严重分级标准(major/minor/presentation),**三类均不设下限、可 0 条、严禁凑数**;硬性规则 5 项打标签(busco/hic/accession/voucher/abs_missing);删除 recommendation 输出字段与示例锚定;coerce 不再读 recommendation(防注入),hard_rule 白名单过滤 |
| `app/routers/presubmit.py` | coerce 后接引擎;meta 增加 tier/tier_label/tier_source |
| `data/journals/journals.json` | 精选 5 本加 `review_tier`(GB/MP=1;BMC PB/FPS/Rice=2)+ notes 一行 |
| `ui/.../PresubmitPanel.vue` | 判定依据块(explanation + 硬性 chips);判定档位徽标;reject 红标"建议人工复核";顶部 AI 免责行;major 卡片"硬性/fatal"徽标 |
| `tests/test_presubmit.py` | 用例 1-10(引擎分支全覆盖 + tier 对照 + 注入忽略 + rationale 字段) |
判定链与旧链路唯一对外契约变化:`report.recommendation` 语义不变(四档枚举),
新增 `report.recommendation_rationale` 与 `meta.tier*`。
## 测试用例与结果
- **pytest:87 passed**(78 基线 + 9 新增/重写),含用例 1-10:
强稿形态不判大修、弱稿形态判大修且 hard_hits 正确、0 major 0/多 minor 分支、
soft major 分支、fatal→reject、tier3 降级与硬性豁免、tier1 accept 门槛、
LLM recommendation 注入被忽略、hard_rule 白名单过滤、rationale 含 tier/explanation。
- **vitest:377 passed / 5 failed**(5F 为既有他人 i18n labels 改动引发的 benchpanel
失败,与本改动无关;新增 4 条 rationale/档位/fatal 展示用例全绿)。
- **真实链路验收**(诊断同款样本、同 target_journal=Genome Biology,真 LLM 调用):
| 样本 | 修复前 | 修复后 | 依据 |
|---|---|---|---|
| 弱稿 diag_weak.pdf | major_revision(7 major) | **major_revision**(R1;硬性 5 命中:BUSCO/Hi-C/Accession/voucher/ABS + soft 2 + minor 6) | 真缺陷不漏判 ✓ |
| 强稿 diag_strong.pdf | major_revision(凑出 6 major) | **minor_revision**(R5;0 major / 5 minor) | 区分度恢复 ✓ |
## 部署验证
```bash
bash review_service/restart.sh
# 上传强/弱样本对比(或任意 PDF):
curl -X POST http://127.0.0.1:8100/api/review/presubmit \
-F "file=@paper.pdf" -F "target_journal=Genome Biology"
# 检查 report.recommendation_rationale.rule / meta.tier;前端看"判定依据"块
```
## 是否需要立即部署
待管理员确认。属判定质量修复(误报大修影响用户体验与信任),建议随下次正常发版带上。
## 遗留(Phase 2 候选)
- 程序层兜底 `_sanity_check_missing_hard_rules`:正文无 PRJNA/GCA_/SRP 关键词但 LLM
未报 accession_missing → 追加 warning(不进 recommendation 计算);先观察 LLM 是否
"躺平"再决定。
- tier 数据仅覆盖精选 5 本;扩展到 journal_directory 1056 条需人工核验后另行入库。
@@ -1,161 +0,0 @@
# PR 报告:Rebuttal Phase 1(审稿回复助手)
- 日期:2026-10-12
- 分支:`feature/academic-review-assistant`(未 commit,按用户约定统一处理)
- 范围:`review_service/` 新增 5 文件 + main 注册;`ui/` 新增面板与 store/api 扩展;无 sdk/、agent/、vstore/ 改动;无新增依赖。
## 功能概述
粘贴审稿意见文本(主入口)/ 上传批注版 PDF(次要入口)→ LLM+正则拆条(Reviewer→Comment)
→ 逐条填回复要点(5 类下拉,无 auto)→ 生成草稿(反幻觉校验前置)
→ 逐条勾确认 → 拼装 response letter → 下载 `.md`(human_ack 门槛)。docx 属 Phase 2。
## 10 个 API 端点(review_service:8100;BFF 8095 通用透传已验证可用)
| 方法 | 路径 | 说明 |
|---|---|---|
| POST | /api/review/rebuttal/jobs | 文本粘贴主入口 |
| POST | /api/review/rebuttal/jobs/pdf | PDF 次要入口(vstore 解析;非 PDF/超限→VALIDATION) |
| GET | /api/review/rebuttal/jobs/{id} | 全量还原 |
| PATCH | .../comments/{cid}/response | 保存 parts(改后旧 draft/validation 作废) |
| POST | .../comments/{cid}/draft | 生成草稿(校验C前置;MISSING_RESPONSE 时不调 LLM) |
| PATCH | .../comments/{cid}/structure | 手工 merge(≥2)/ split(≥2 非空子项) |
| PATCH | .../comments/{cid}/confirm | 逐条勾;后端重算 human_ack(全勾才 true,空 job 不置位) |
| POST | .../letter | 拼装 letter_md |
| GET | .../letter.md | 下载(human_ack=false → 400 HUMAN_ACK_REQUIRED) |
| DELETE | .../jobs/{id} | 手动删(替代 TTL) |
信封错误复用约定:HTTP 200 + `{ok:false,error:{code,message,degraded}}`。
## 数据模型
- Job:`{id,status,source:{kind,filename,raw_text},template:'generic',reviewers:[...],letter_md,validation,human_ack,created_at,updated_at}`
- Comment:`{id,group_id,severity,title,body,sub_items[],cross_refs[],response:{parts[]},draft,validation,human_confirmed}`
- part.type 枚举:`new_data | reanalysis | location | clarification | rebuttal`(决策 2:无 auto)
- SQLite 表 `rebuttal_jobs` 落在既有 `review_service/review.db`(自含短连接,不动 app/db.py);
启动幂等建表(main lifespan)。
## 关键实现决策(方案未写清处,按"保守、可测、不猜测")
1. **LLM 注入形态**:parse/draft 用 `async (messages, temperature) → dict` 回调,路由内适配
`llm.complete_json(system, user)`;单测直接 monkeypatch `llm.complete_json`。
2. **拆条双轨**:正则先切骨架(Reviewer 段头 + 编号行,兼容行内 `. 2.` 编号);
LLM 逐段出 `{title,severity,sub_items[]}`。LLM 失败/结构非法 → 该段回退正则
(title=首 60 字,severity=major)。`sub_items ≥2 且 max confidence ≥0.7` 才拆
(决策 3),拆出条目共享 group_id;未达阈值多子问题并回一条且 **body 保留段原文**
(不采信 LLM 改写)。
3. **交叉引用**:`see comment N / 参见第 N 条 / 同上` → 解析目标 id;解析不到
`target_id=null` 保留 raw(不猜)。
4. **反幻觉校验(负向词表收窄版)**:
- location:`shows that/as shown` 且无 `added/新增` → `LOCATION_EVIDENCE_WEAK` FAIL;
- rebuttal:命中 `we agree/accept/concede/you are right/we were wrong` →
`REBUTTAL_NEGATIVE` FAIL;无立场词(maintain/disagree/respectfully…)→
`REBUTTAL_NO_STANCE` FAIL;
- 其余三类 Phase 1 仅查非空。校验在 draft 端点前置,FAIL 不调 LLM。
5. **human_ack**:confirm 端点改完后由 store 重算(所有 comment 全 true 才置位;
空 job 不置位防平凡真)。前端无法直接置位。
6. **letter.md 下载**:纯文本端点(非信封),门槛失败返回 400 + 明文
`HUMAN_ACK_REQUIRED:…`;前端 `rebuttalDownloadLetterMd` 适配成 ApiError。
7. **MISSING_RESPONSE**:draft 端点 parts 为空或全空内容 → 信封错误,LLM 调用次数=0
(单测断言)。
8. **PDF 入口**:复用 vstore `parse_pdf_bytes`;校验顺序 = 非 .pdf → 空文件 → 超
`max_pdf_bytes`(50MB)→ 解析。与任务清单"PDF 超限/非 PDF"信封一致。
9. **前端状态**:`store.rebuttal` 纯内存(切面板不丢、刷新丢——与任务"状态持久化"
不同诉求,Phase 1 未要求 localStorage);draft 失败信封里的 job/validation 一并镜像。
10. **不改禁区**:sdk/agent/vstore/未动;PresubmitPanel 等既有面板未动;无新增依赖。
## 测试
### 后端 pytest(9 例新增,全量 113 passed)
```
tests/test_rebuttal.py::test_single_reviewer_single_comment PASSED
tests/test_rebuttal.py::test_multi_reviewer_attribution PASSED
tests/test_rebuttal.py::test_subitem_split_shares_group_id PASSED
tests/test_rebuttal.py::test_location_validation_fail_and_pass PASSED
tests/test_rebuttal.py::test_rebuttal_validation PASSED
tests/test_rebuttal.py::test_missing_response_no_llm_call PASSED
tests/test_rebuttal.py::test_cross_ref_unresolved_keeps_raw PASSED
tests/test_rebuttal.py::test_envelope_errors PASSED
tests/test_rebuttal.py::test_human_ack_gate_on_letter_download PASSED
全量: 113 passed in 10.05s(104 基线 + 9 新增)
```
### 前端 vitest(5 例新增)
```
rebuttal.panel.test.js 5 passed(5 类下拉无 auto / location hint 条件显示 /
未填禁用生成 / 未全确认禁用下载+横幅 / merge-split 仅 active 表单)
全量: 387 passed | 5 failed(失败全部为已知 labels.js 引发的 benchpanel×4 +
Composer .mode-pill×1 预存失败,与本次改动无关;基线 382P/5F → 387P/5F)
```
### build
`npm run build` ✓ built in 3.04s
## 手工冒烟(curl,8100 真 LLM + 真 SQLite)
```
POST /jobs(真实双 Reviewer 意见)→ ok=True job=rbt_ed245cd63b3a reviewers=2 comments=4
titles: 缺少FDR校正与q值报告 / Figure 3 blurry… / Missing coordinates… / Overstated novelty…
POST draft(未填) → ok=False code=MISSING_RESPONSE(LLM 0 调用)
PATCH response(location 弱) + POST draft
→ ok=False VALIDATION_FAILED findings=[LOCATION_EVIDENCE_WEAK]
PATCH response(location 合法) + POST draft → ok=True(真 LLM 草稿)
PATCH response(rebuttal 让步词) + draft → VALIDATION_FAILED findings=[REBUTTAL_NEGATIVE]
3 条合理回复(new_data/clarification/rebuttal)→ draft 全 ok=True
GET letter.md(未全勾) → HTTP 400 HUMAN_ACK_REQUIRED
PATCH confirm ×3 → human_ack=False(c0001 漏勾,恰好验证逐条门槛)
PATCH confirm c0001 → human_ack=True
POST letter → ok=True letter_md len=2731
GET letter.md → HTTP 200(/tmp/opencode/letter.md 落盘,含
Response to Reviewers + 4 条原文/回复)
DELETE job → ok=True meta={deleted:…};GET → NOT_FOUND
BFF 8095 探测 /rebuttal/jobs/nonexistent → HTTP 200(透传信封,REVIEW_ENABLED=1)
```
## 交付物清单
| 文件 | 动作 |
|---|---|
| review_service/app/services/rebuttal_store.py | 新增(SQLite CRUD + human_ack 重算) |
| review_service/app/services/rebuttal_parse.py | 新增(正则骨架 + LLM 拆分 + 交叉引用) |
| review_service/app/services/rebuttal_draft.py | 新增(5 类 Prompt + 反幻觉校验 + letter) |
| review_service/app/routers/rebuttal.py | 新增(10 端点) |
| review_service/app/main.py | +import +include_router +lifespan 建表(约 8 行) |
| review_service/tests/test_rebuttal.py | 新增(9 例) |
| ui/src/api/review.js | +10 API 封装 + rebuttal 错误码未进 CODE_HINTS(后端原话直达) |
| ui/src/stores/edu.js | +SECTIONS 'rebuttal' + rebuttal 状态块 + 10 actions |
| ui/src/components/edu/panels/RebuttalPanel.vue | 新增(拆条树/逐条表单/校验红条/draft 预览/手工拆合/letter) |
| ui/src/components/edu/EduSideNav.vue | +1 行菜单(审稿回复助手 💬) |
| ui/src/components/edu/EduWorkspace.vue | +import +panelMap 挂载 |
| tests/frontend/unit/edu/rebuttal.panel.test.js | 新增(5 例) |
| docs/pr-reports/2026-10-12-rebuttal-phase1.md | 本报告 |
## git status --short(新增部分)
```
?? docs/pr-reports/2026-10-12-rebuttal-phase1.md
?? review_service/app/routers/rebuttal.py
?? review_service/app/services/rebuttal_draft.py
?? review_service/app/services/rebuttal_parse.py
?? review_service/app/services/rebuttal_store.py
?? review_service/tests/test_rebuttal.py
?? tests/frontend/unit/edu/rebuttal.panel.test.js
?? ui/src/components/edu/panels/RebuttalPanel.vue
M review_service/app/main.py
M ui/src/api/review.js
M ui/src/components/edu/EduSideNav.vue
M ui/src/components/edu/EduWorkspace.vue
M ui/src/stores/edu.js
(此前 AI 预审稿修复的未 commit 改动保持原样,未混入本报告范围)
```
## 已知边界(Phase 1 明确不做)
- docx 导出(Phase 2,python-docx 已在 lock 备用);
- job TTL 自动清理(决策 5:手动 DELETE 替代);
- 跨 Reviewer 合并(merge 限同 Reviewer 内);
- 刷新页面后 job 恢复(纯内存态;GET /jobs/{id} 已备,Phase 2 可接列表入口)。
-13
View File
@@ -1,13 +0,0 @@
Reviewer #1:
1. The GWAS analysis lacks multiple-testing correction. Please re-run with FDR control and report q-values for all significant loci. In addition, the sample size justification is missing — please add a power analysis or at least discuss the detectable effect size.
2. Figure 3 is blurry and the axis labels are unreadable. As noted in Comment 3, the same issue applies to Figure S2. Please replace all figures with high-resolution versions.
3. Minor: please add the standard error estimates to Supplementary Table 2.
Reviewer #2:
1. The claim of novelty overstates the advance relative to Zhang et al. 2023. The introduction should more clearly delineate what is new.
2. How were the batch effects between the two sequencing centers corrected? This is essential to support the conclusions.
-27
View File
@@ -1,27 +0,0 @@
# review_service/.env.example —— 运行时配置模板
# 用法: cp .env.example .env 后填写(.env 已被仓库 .gitignore 忽略,勿提交)
# restart.sh 会 `set -a; source .env` 导出给 uvicorn 进程。
# ---- LLM(DeepSeek;REVIEW_LLM_* 优先 → DOCGEN_LLM_* 回退 → 内置默认,见 services/llm.py)----
# 最简做法:从部署树抄一行即可复用同一把 key:
# grep '^DOCGEN_LLM_API_KEY=' /home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new/sdk/.env >> .env
REVIEW_LLM_BASE_URL=https://api.deepseek.com/v1
REVIEW_LLM_API_KEY=
REVIEW_LLM_MODEL=deepseek-chat
#REVIEW_LLM_TIMEOUT=300
# ---- 上游(只读消费)----
#REVIEW_VSTORE_BASE_URL=http://127.0.0.1:8094
#REVIEW_METHODOLOGY_BASE_URL=http://127.0.0.1:8092 # 预留(MinerU 重链路,当前未用)
# ---- PDF 解析(经 vstore)----
#REVIEW_PARSE_TIMEOUT=180 # ingest→取文→清理 总读超时(秒)
#REVIEW_PARSE_DELETE_AFTER=1 # 1=解析后删除 vstore 临时文档(0=保留便于排查)
# ---- 自审稿 ----
#REVIEW_PRESUBMIT_READ_TIMEOUT=360 # LLM 长调用读超时(秒)
#REVIEW_MAX_TEXT_CHARS=60000 # 送 LLM 的正文截断上限
#REVIEW_MAX_PDF_BYTES=52428800 # 上传 PDF 上限 50MB
# ---- 服务(restart.sh 读取)----
#REVIEW_BIND_HOST=127.0.0.1
-7
View File
@@ -1,7 +0,0 @@
"""review_service —— 科研投稿与智能审稿助手(8100)。
独立部署树(模式同 paper_writer_service 8097 / bioflow_service 8098):
- BFF(sdk 8090) 经 routers/review_proxy.py 条件反代 /api/review/*(REVIEW_ENABLED)。
- 上游只读消费:vstore 8094(PDF 解析)、DeepSeek(LLM,REVIEW_LLM_*/DOCGEN_LLM_*)。
- 约束见根目录 AGENTS.md:审稿相关新代码只允许落在本目录。
"""
-1
View File
@@ -1 +0,0 @@
"""review 服务(8100)应用包。"""
-112
View File
@@ -1,112 +0,0 @@
"""config.py —— review 服务(8100)环境变量驱动配置(pydantic-settings)。
全部键可用环境变量覆盖(前缀 REVIEW_)。LLM 沿用 docgen 的裸键模式:
优先 REVIEW_LLM_*,回退 DOCGEN_LLM_*(与 paper_writer_service 共享同一套
DeepSeek 凭据,由 services/llm.py 直接读 os.environ)。零硬编码宿主路径。
"""
from __future__ import annotations
from functools import lru_cache
from pathlib import Path
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
env_prefix="REVIEW_",
)
app_name: str = "plantai-review-service"
version: str = "0.1.0"
# ---- 上游(只读消费;不修改 vstore/methodology 任何代码)----
# vstore 8094:POST /api/vstore/ingest + GET/DELETE /api/vstore/documents/{id}
# (pymupdf 段落解析,同步秒级;methodology 8092 的 MinerU 链路是
# 分钟级异步 job,不适配同步自审稿,仅预留地址供后续扩展)
vstore_base_url: str = "http://127.0.0.1:8094"
methodology_base_url: str = "http://127.0.0.1:8092"
# ---- PDF 解析 ----
parse_timeout: float = 180.0 # vstore ingest→fetch→delete 读超时(秒)
parse_delete_after: bool = True # 解析完成后删除 vstore 临时文档(不污染私有库)
# ---- 自审稿(presubmit)----
presubmit_read_timeout: float = 360.0 # LLM 长调用读超时(秒)
max_text_chars: int = 150000 # 送 LLM 的正文截断上限(REVIEW_MAX_TEXT_CHARS 覆盖;
# token 超 48K 自动降 100K,LLM 400/413 再降 80K 重试)
max_pdf_bytes: int = 50 * 1024 * 1024 # 上传 PDF 上限 50MB(对齐 extraction FR-EXTRACT-004)
# ---- 期刊情报与投稿推荐(recommend)----
db_path: str = "review.db" # 独立 SQLite(相对 review_service/;REVIEW_DB_PATH 覆盖)
journals_seed_path: str = "data/journals/journals.json" # 种子(相对 review_service/)
journal_directory_seed_path: str = "data/journals/source_v4.xlsx" # 目录源 xlsx(相对 review_service/)
seed_on_startup: bool = True # 启动自动 seed journal_directory(空表才导;false 关闭)
vstore_match_timeout: float = 30.0 # scope 入库/向量检索读超时(秒)
recommend_read_timeout: float = 60.0 # LLM 重排读超时(秒)
recommend_query_chars: int = 3000 # 送 vstore 的检索 query 截断(防嵌入模型超长)
recommend_scope_top_k: int = 10 # search-chunks top_k(vstore 上限 20)
# ---- 论文重排版(reformat,P1:.tex 源码级变换,不编译)----
max_tex_bytes: int = 2 * 1024 * 1024 # 上传 .tex 上限 2MB(PDF 的 50MB 不适用)
reformat_dir: str = "data/reformat_jobs" # 产物落盘(相对 review_service/)
# ---- LaTeX 编译 PDF(compile,P3:tectonic 单文件二进制)----
tectonic_path: str = "tools/tectonic" # 相对 review_service/;也可绝对路径
tectonic_cache_dir: str = "data/tectonic-cache" # 宏包缓存(部署时预热,运行期只读命中)
compile_timeout_s: float = 60.0 # 单次编译墙钟上限,超时整进程组 kill
max_compile_pdf_bytes: int = 50 * 1024 * 1024 # 产物 PDF 上限 50MB
compile_queue_limit: int = 5 # 等待槽位上限(>5 → QUEUE_FULL 429)
compile_retention_days: int = 7 # 编译产物保留天数(后台清扫)
@property
def _service_root(self) -> "Path":
# review_service/app/config.py → review_service/(相对本文件定位,不依赖 cwd)
return Path(__file__).resolve().parent.parent
@property
def db_path_abs(self) -> "Path":
# 绝对 db_path 会被 Path 除法原样返回;相对路径落在 review_service/ 下
return self._service_root / self.db_path
@property
def journals_seed_abs(self) -> "Path":
return self._service_root / self.journals_seed_path
@property
def journal_directory_seed_abs(self) -> "Path":
return self._service_root / self.journal_directory_seed_path
@property
def reformat_dir_abs(self) -> "Path":
p = Path(self.reformat_dir)
return p if p.is_absolute() else self._service_root / p
@property
def tectonic_abs(self) -> "Path":
p = Path(self.tectonic_path)
return p if p.is_absolute() else self._service_root / p
@property
def tectonic_cache_abs(self) -> "Path":
p = Path(self.tectonic_cache_dir)
return p if p.is_absolute() else self._service_root / p
@property
def vstore_base(self) -> str:
return self.vstore_base_url.rstrip("/")
@property
def methodology_base(self) -> str:
return self.methodology_base_url.rstrip("/")
@lru_cache(maxsize=1)
def get_settings() -> Settings:
return Settings()
settings = get_settings()
-191
View File
@@ -1,191 +0,0 @@
"""app/db.py —— review 服务独立 SQLite(aiosqlite + WAL,无 ORM):journals 表。
- 单连接 + asyncio 写锁(模式对齐 paper_writer_service/bioflow,但表结构与种子
逻辑为本模块自研);路径默认 review_service/review.db(REVIEW_DB_PATH 覆盖)。
- 种子:data/journals/journals.json → journals 表(幂等 upsert,vstore_doc_id 列
永不被种子覆盖——该列记录 scope_desc 在 vstore 8094 的 document_id)。
- 只在 review_service/ 内使用,不触碰 BFF/生产服务数据库。
"""
from __future__ import annotations
import asyncio
import json
from pathlib import Path
from typing import Any, Iterable
import aiosqlite
from .config import get_settings
_SCHEMA = """
CREATE TABLE IF NOT EXISTS journals (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
scope_desc TEXT NOT NULL,
impact_factor REAL,
review_time_weeks REAL,
acceptance_rate REAL,
submission_deadline TEXT,
official_url TEXT,
last_updated TEXT NOT NULL,
vstore_doc_id TEXT
);
"""
# 种子 upsert 列(刻意不含 vstore_doc_id:种子刷新不得清掉向量库映射)
_SEED_COLS = (
"id",
"name",
"scope_desc",
"impact_factor",
"review_time_weeks",
"acceptance_rate",
"submission_deadline",
"official_url",
"last_updated",
)
_UPSERT_SQL = (
"INSERT INTO journals (" + ", ".join(_SEED_COLS) + ") VALUES ("
+ ", ".join("?" for _ in _SEED_COLS) + ") "
+ "ON CONFLICT(id) DO UPDATE SET "
+ ", ".join(f"{c}=excluded.{c}" for c in _SEED_COLS if c != "id")
)
class SeedError(Exception):
"""种子文件缺失/损坏(journals.json 不可读)。"""
class Database:
"""aiosqlite 单连接封装(WAL;写串行化)。"""
def __init__(self, path: str | Path | None = None) -> None:
self._path: Path | None = Path(path) if path is not None else None
self._conn: aiosqlite.Connection | None = None
self._lock = asyncio.Lock()
@property
def path(self) -> Path:
if self._path is None:
self._path = get_settings().db_path_abs
return self._path
def set_path(self, path: str | Path) -> None:
"""测试注入:在进入 TestClient lifespan 之前预置库路径(须尚未 connect)。"""
if self._conn is not None:
raise RuntimeError("Database 已连接,无法再改路径(先 close_db())")
self._path = Path(path)
@property
def connected(self) -> bool:
return self._conn is not None
async def connect(self) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
self._conn = await aiosqlite.connect(self.path)
self._conn.row_factory = aiosqlite.Row
await self._conn.execute("PRAGMA journal_mode=WAL")
await self._conn.execute("PRAGMA busy_timeout=5000")
await self._conn.executescript(_SCHEMA)
await self._conn.commit()
async def close(self) -> None:
if self._conn is not None:
await self._conn.close()
self._conn = None
@property
def conn(self) -> aiosqlite.Connection:
if self._conn is None:
raise RuntimeError("Database 未初始化:先 await init_db()")
return self._conn
async def fetchall(self, sql: str, params: Iterable[Any] = ()) -> list[dict[str, Any]]:
async with self.conn.execute(sql, tuple(params)) as cur:
rows = await cur.fetchall()
return [dict(r) for r in rows]
async def fetchone(self, sql: str, params: Iterable[Any] = ()) -> dict[str, Any] | None:
async with self.conn.execute(sql, tuple(params)) as cur:
row = await cur.fetchone()
return dict(row) if row is not None else None
async def execute(self, sql: str, params: Iterable[Any] = ()) -> int:
async with self._lock:
cur = await self.conn.execute(sql, tuple(params))
await self._conn.commit() # type: ignore[union-attr]
return cur.lastrowid or 0
async def executemany(self, sql: str, seq: list[tuple[Any, ...]]) -> None:
async with self._lock:
await self.conn.executemany(sql, seq)
await self._conn.commit() # type: ignore[union-attr]
db = Database()
async def init_db(path: str | Path | None = None) -> None:
"""幂等初始化:已连接则跳过(测试可在 lifespan 前 set_path 注入路径)。"""
if path is not None:
db.set_path(path)
if db.connected:
return
await db.connect()
async def close_db() -> None:
await db.close()
def set_path(path: str | Path) -> None:
"""同步注入库路径(测试 fixture 用;须在 init_db 之前调用)。"""
db.set_path(path)
async def ensure_journals_seeded() -> int:
"""装载种子期刊(幂等)。返回 journals 表行数。"""
row = await db.fetchone("SELECT COUNT(*) AS c FROM journals")
if row and int(row["c"]) > 0:
return int(row["c"])
seed_file = get_settings().journals_seed_abs
try:
payload = json.loads(Path(seed_file).read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise SeedError(f"期刊种子文件不存在:{seed_file}") from exc
except (OSError, ValueError) as exc:
raise SeedError(f"期刊种子文件不可读:{exc}") from exc
journals = payload.get("journals") if isinstance(payload, dict) else None
if not isinstance(journals, list) or not journals:
raise SeedError(f"期刊种子文件缺少 journals 数组:{seed_file}")
rows: list[tuple[Any, ...]] = []
for item in journals:
if not isinstance(item, dict) or not item.get("id") or not item.get("name"):
continue
rows.append(tuple(item.get(c) for c in _SEED_COLS))
if not rows:
raise SeedError(f"期刊种子文件无有效条目:{seed_file}")
await db.executemany(_UPSERT_SQL, rows)
return len(rows)
async def list_journals() -> list[dict[str, Any]]:
"""全部期刊(impact_factor 降序,NULL 靠后)。"""
return await db.fetchall(
"SELECT * FROM journals ORDER BY impact_factor DESC, name ASC"
)
async def get_journal(journal_id: str) -> dict[str, Any] | None:
return await db.fetchone("SELECT * FROM journals WHERE id = ?", (journal_id,))
async def set_vstore_doc_id(journal_id: str, doc_id: str) -> None:
"""记录 scope_desc 在 vstore 的 document_id(向量白名单检索用)。"""
await db.execute(
"UPDATE journals SET vstore_doc_id = ? WHERE id = ?", (doc_id, journal_id)
)
-124
View File
@@ -1,124 +0,0 @@
"""main.py —— review 服务应用入口(uvicorn 入口)。
- 独立部署树 review_service/(科研投稿与智能审稿助手,端口 8100)。
- 端点:/health(GET,BFF 探活反代)+ /api/review/*(自审稿等业务路由)。
- 不依赖 BFF/生产服务(8090 等绝不探测/连接);上游仅消费 vstore 8094。
"""
from __future__ import annotations
import asyncio
import logging
import time
from contextlib import asynccontextmanager
import httpx
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from .config import get_settings
from .db import close_db, db, init_db
from .routers import (
compile,
conferences,
format_check,
journal_directory,
journals,
presubmit,
reformat,
rebuttal,
)
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def create_app() -> FastAPI:
settings = get_settings()
app = FastAPI(title=settings.app_name, version=settings.version)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=False,
allow_methods=["*"],
allow_headers=["*"],
)
@asynccontextmanager
async def _lifespan(_app: FastAPI):
# journals 推荐依赖独立 SQLite:启动即建表(幂等;测试可先 set_path 注入
# tmp 路径,本函数沿用注入路径)
await init_db()
# rebuttal Phase 1:独立表,启动幂等建表(自含连接,不走 db.py)
try:
from .services.rebuttal_store import ensure_table
ensure_table()
except Exception: # noqa: BLE001
logger.warning("rebuttal 建表失败(忽略)", exc_info=True)
# 编译产物过期清扫:启动跑一次(失败不阻塞启动)
try:
from .services.latex_compile import sweep_expired
sweep_expired()
except Exception: # noqa: BLE001
logger.warning("compile sweep 启动清扫失败(忽略)", exc_info=True)
# journal_directory 自动 seed:空表才导(xlsx 解析约 1-3s,不堵 event loop;
# 路径沿用 db.path 注入链——生产=settings.db_path_abs,测试=tmp)
if settings.seed_on_startup:
try:
from .services.journal_seed import ensure_journal_directory_seeded
_t0 = time.time()
_result = await asyncio.to_thread(
ensure_journal_directory_seeded, db.path)
if _result.get("seeded"):
_c = _result.get("counts") or {}
logger.info(
"seeded %s rows (en=%s, zh=%s, %.1fs)",
_result["rows"], _c.get("en"), _c.get("zh"),
time.time() - _t0)
elif _result.get("error"):
logger.warning(
"journal_directory 启动 seed 失败(忽略,不阻塞启动): %s",
_result["error"])
else:
logger.info("%s", _result.get("reason") or "journal_directory seed skipped")
except Exception: # noqa: BLE001
logger.warning("journal_directory 启动 seed 失败(忽略,不阻塞启动)", exc_info=True)
else:
logger.info("journal_directory seed skipped (REVIEW_SEED_ON_STARTUP=false)")
try:
yield
finally:
await close_db()
app.router.lifespan_context = _lifespan
@app.get("/health")
async def health() -> dict[str, object]:
vstore_ok = False
vstore_error: str | None = None
try:
async with httpx.AsyncClient(timeout=2.0) as client:
resp = await client.get(f"{settings.vstore_base}/api/vstore/health")
resp.raise_for_status()
vstore_ok = bool(resp.json().get("ok"))
except Exception as exc: # noqa: BLE001
vstore_error = str(exc)
return {
"ok": True,
"status": "ok" if vstore_ok else "degraded",
"upstreams": {"vstore": {"connected": vstore_ok, "error": vstore_error}},
"version": settings.version,
}
app.include_router(presubmit.router)
app.include_router(rebuttal.router)
app.include_router(journals.router)
app.include_router(format_check.router)
app.include_router(conferences.router)
app.include_router(journal_directory.router)
app.include_router(reformat.router)
app.include_router(compile.router)
return app
app = create_app()
-1
View File
@@ -1 +0,0 @@
"""review 服务路由包。"""
-184
View File
@@ -1,184 +0,0 @@
"""routers/compile.py —— LaTeX 编译 PDF(P3a:tectonic)。
端点:
POST /api/review/compile multipart:file(.tex 可选) | job_id(可选,二选一)
→ {ok, compile_id, pdf_url, log_url, meta}
GET /api/review/compile/{cid} 编译元信息(meta.json)
GET /api/review/compile/{cid}/pdf output.pdf(FileResponse)
GET /api/review/compile/{cid}/log output.log(text/plain)
信封:业务错误 = HTTP 200 + {ok:false,error:{code,message,detail?,degraded:false}}
(QUEUE_FULL 例外:HTTP 429 + 同款信封体,前端仍按 ok/error 单一口径处理)。
detail = log 尾 200 行(COMPILE_FAILED / TIMEOUT)。
"""
from __future__ import annotations
import json
import logging
import re
import uuid
from pathlib import Path
from typing import Any
from fastapi import APIRouter, File, Form, UploadFile
from fastapi.responses import FileResponse, JSONResponse, PlainTextResponse
from pydantic import BaseModel
from ..config import get_settings
from ..services import latex_compile as lc
from ..services.latex_compile import (
CompileError,
QueueFullError,
ToolUnavailableError,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-compile"])
_CID_RE = re.compile(r"^[0-9a-f]{12}$")
class CompileResponse(BaseModel):
ok: bool
compile_id: str | None = None
pdf_url: str | None = None
log_url: str | None = None
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
def _error(code: str, message: str, detail: str = "") -> CompileResponse:
err: dict[str, Any] = {"code": code, "message": message, "degraded": False}
if detail:
err["detail"] = detail
return CompileResponse(ok=False, error=err)
def _jobs_root() -> Path:
d = get_settings().reformat_dir_abs
d.mkdir(parents=True, exist_ok=True)
return d
@router.post("/compile", response_model=CompileResponse)
async def compile_tex(
file: UploadFile | None = File(None, description="LaTeX 源码(.tex,与 job_id 二选一)"),
job_id: str = Form("", description="P1 重排版任务 id(优先于 file)"),
) -> CompileResponse:
settings = get_settings()
# ---- 输入解析:job_id 优先,否则直接上传的 .tex ----
jid = (job_id or "").strip()
if jid:
if not _CID_RE.fullmatch(jid):
return _error("VALIDATION", "job_id 格式非法(应为 12 位 hex)")
src = _jobs_root() / f"{jid}.tex"
if not src.is_file():
return _error("VALIDATION", "job_id 对应的重排版结果不存在,请重新排版")
try:
source = src.read_text("utf-8")
except UnicodeDecodeError:
return _error("VALIDATION", "任务源码不是合法 UTF-8")
anchor_id = jid
else:
if file is None:
return _error("VALIDATION", "必须提供 file(.tex)或 job_id")
filename = file.filename or "paper.tex"
if Path(filename).name != filename or ".." in filename:
return _error("VALIDATION", "非法文件名")
if not filename.lower().endswith(".tex"):
return _error("VALIDATION", "仅支持 .tex 文件")
content = await file.read()
if not content:
return _error("VALIDATION", "上传文件为空")
if len(content) > settings.max_tex_bytes:
return _error("VALIDATION", "源码超过 2MB 上限")
try:
source = content.decode("utf-8")
except UnicodeDecodeError:
return _error("VALIDATION", "源码必须为 UTF-8 编码")
anchor_id = uuid.uuid4().hex[:12]
# ---- 编译(隔离目录;失败信封带 log 尾)----
lc.prepare_input(anchor_id, source)
try:
result = await lc.compile_tex(anchor_id)
except ToolUnavailableError as exc:
return _error("TOOL_UNAVAILABLE", str(exc))
except QueueFullError as exc:
return JSONResponse(
status_code=429,
content={
"ok": False,
"compile_id": None, "pdf_url": None, "log_url": None, "meta": None,
"error": {"code": "QUEUE_FULL", "message": str(exc), "degraded": False},
},
)
except CompileError as exc:
return _error(exc.code, str(exc), detail=exc.log_tail)
meta_path = lc.compile_dir(anchor_id) / "meta.json"
try:
meta = json.loads(meta_path.read_text("utf-8"))
except (OSError, ValueError):
meta = {"compile_id": anchor_id, "seconds": result["seconds"]}
return CompileResponse(
ok=True,
compile_id=anchor_id,
pdf_url=f"/api/review/compile/{anchor_id}/pdf",
log_url=f"/api/review/compile/{anchor_id}/log",
meta=meta,
)
def _cid_dir(cid: str) -> Path | None:
if not _CID_RE.fullmatch(cid or ""):
return None
d = lc.compile_dir(cid)
return d if d.is_dir() else None
@router.get("/compile/{compile_id}", response_model=CompileResponse)
async def get_compile(compile_id: str) -> CompileResponse:
cdir = _cid_dir(compile_id)
meta_path = cdir / "meta.json" if cdir else None
if meta_path is None or not meta_path.is_file():
return _error("NOT_FOUND", "编译任务不存在或已过期,请重新编译")
try:
meta = json.loads(meta_path.read_text("utf-8"))
except (OSError, ValueError):
return _error("NOT_FOUND", "编译产物损坏,请重新编译")
return CompileResponse(
ok=True,
compile_id=compile_id,
pdf_url=f"/api/review/compile/{compile_id}/pdf",
log_url=f"/api/review/compile/{compile_id}/log",
meta=meta,
)
@router.get("/compile/{compile_id}/pdf", response_model=None)
async def download_compile_pdf(compile_id: str) -> FileResponse | JSONResponse:
cdir = _cid_dir(compile_id)
pdf = cdir / "output.pdf" if cdir else None
if pdf is None or not pdf.is_file():
return JSONResponse(
status_code=200,
content={"ok": False, "error": {
"code": "NOT_FOUND", "message": "PDF 不存在或已过期", "degraded": False}},
)
return FileResponse(pdf, media_type="application/pdf", filename=f"{compile_id}.pdf")
@router.get("/compile/{compile_id}/log", response_model=None)
async def download_compile_log(compile_id: str) -> PlainTextResponse | JSONResponse:
cdir = _cid_dir(compile_id)
log = cdir / "output.log" if cdir else None
if log is None or not log.is_file():
return JSONResponse(
status_code=200,
content={"ok": False, "error": {
"code": "NOT_FOUND", "message": "日志不存在或已过期", "degraded": False}},
)
return PlainTextResponse(log.read_text("utf-8", errors="replace"))
-209
View File
@@ -1,209 +0,0 @@
"""routers/conferences.py —— 会议投稿时间线(只读静态种子)。
- GET /api/review/conferences?domain=&ccf=&upcoming_days=180
data/journals/conferences.json → 过滤/排序 → 列表 + meta。
- 数据诚实性铁律:每条会议必须带 deadline_source ∈ {placeholder, official}、
last_verified 与 deadline_confidence ∈ {high, medium, low},缺失/非法条目加载时
跳过并记日志(宁缺毋造);本端点无任何生成日期的逻辑,日期只来自种子常量。
v2 共 27/27 全 placeholder,meta.data_source_note 显式声明占位与置信度,
由前端渲染「⚠ 日期待核验」及置信度提示。
- 过滤语义:domain 对 domain_tags 精确匹配(大小写不敏感);ccf ∈ {A,B,C};
upcoming_days=0 不过滤,>0 仅保留 deadline_full_paper ∈ [UTC now, now+N 天],
缺失/不可解析截稿日期在过滤开启时排除。排序:deadline_full_paper 升序,
缺失/不可解析恒排最后。
- 只读无写接口;文件不可读 → ok=false + DATA_UNAVAILABLE 信封;参数非法 →
VALIDATION 信封(均 HTTP 200,对齐本服务"错误信封永远 200"约定)。
- 静态文件首次请求加载进内存(lru_cache),进程内 O(1);不触碰 SQLite/BFF。
"""
from __future__ import annotations
import json
import logging
from datetime import datetime, timedelta, timezone
from functools import lru_cache
from pathlib import Path
from typing import Any
from fastapi import APIRouter
from pydantic import BaseModel
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-conferences"])
# review_service/app/routers/ → parents[2] = review_service(对齐 format_check 写法)
_CONFERENCES_PATH = (
Path(__file__).resolve().parents[2] / "data" / "journals" / "conferences.json"
)
_CCF_OK = frozenset({"A", "B", "C"})
_REQUIRED = ("id", "name", "deadline_source", "last_verified", "deadline_confidence")
_CONFIDENCE_OK = frozenset({"high", "medium", "low"})
_SOURCE_NOTE = (
"会议截稿/通知日期首版均为占位数据(deadline_source=placeholder),"
"接入官方渠道前须人工核验;deadline_confidence 标注日期置信度"
"(high=官方已公布/历年极稳定,medium=按历年规律推算,low=待核验);"
"所有时间按 AoE (UTC-12) 时区。"
)
# ---- 请求/响应模型 ----
class ConferenceItem(BaseModel):
"""会议公开字段(种子全字段白名单;未知键丢弃)。"""
id: str
name: str
full_name: str = ""
domain_tags: list[str] = []
application_tags: list[str] = []
ccf_rank: str | None = None
deadline_abstract: str | None = None
deadline_full_paper: str | None = None
notification_date: str | None = None
conference_date: str | None = None
location: str | None = None
official_url: str | None = None
timezone_note: str | None = None
deadline_source: str
last_verified: str
deadline_confidence: str = ""
class ConferencesResponse(BaseModel):
ok: bool
items: list[ConferenceItem] = []
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
def _error(code: str, message: str) -> ConferencesResponse:
return ConferencesResponse(
ok=False, items=[], meta=None,
error={"code": code, "message": message, "degraded": False},
)
# ---- 种子加载(首次调用读盘,进程内缓存)----
@lru_cache(maxsize=1)
def _load_conferences() -> list[dict[str, Any]] | None:
"""读 conferences.json → 合法会议列表;文件不可读 → None(DATA_UNAVAILABLE)。"""
try:
payload = json.loads(_CONFERENCES_PATH.read_text("utf-8"))
except (OSError, ValueError) as exc:
logger.warning("conferences: 种子文件不可读 %s: %s", _CONFERENCES_PATH, exc)
return None
raw = payload.get("conferences") if isinstance(payload, dict) else None
if not isinstance(raw, list):
logger.warning("conferences: 种子缺少 conferences 数组:%s", _CONFERENCES_PATH)
return None
items: list[dict[str, Any]] = []
for conf in raw:
if not isinstance(conf, dict):
continue
missing = [k for k in _REQUIRED if not conf.get(k)]
if missing:
logger.warning("conferences: 条目缺字段 %s(id=%s),跳过", missing, conf.get("id"))
continue
if conf.get("deadline_source") not in ("placeholder", "official"):
logger.warning(
"conferences: deadline_source 非法(id=%s),跳过", conf.get("id")
)
continue
if str(conf.get("deadline_confidence") or "").strip().lower() not in _CONFIDENCE_OK:
logger.warning(
"conferences: deadline_confidence 非法(id=%s),跳过", conf.get("id")
)
continue
items.append(conf)
return items
def _parse_deadline(value: Any) -> datetime | None:
"""截稿时间 → aware datetime(naive 视为 UTC);非法 → None。"""
if not isinstance(value, str) or not value.strip():
return None
try:
dt = datetime.fromisoformat(value.strip())
except ValueError:
return None
return dt.replace(tzinfo=timezone.utc) if dt.tzinfo is None else dt
# ---- GET /api/review/conferences ----
@router.get("/conferences", response_model=ConferencesResponse)
async def conferences_list(
domain: str = "",
ccf: str = "",
upcoming_days: int = 180,
) -> ConferencesResponse:
"""只读:会议投稿时间线(无写/删/改接口)。
- ?domain= 对 domain_tags 精确匹配(大小写不敏感),空=不过滤;
- ?ccf= ccf_rank ∈ A/B/C,空=不过滤,非法值 → VALIDATION 信封;
- ?upcoming_days= int≥0,默认 180;0=不过滤,>0 仅保留截稿在
[UTC now, now+N 天] 内的会议,负数 → VALIDATION 信封;
- 排序按 deadline_full_paper 升序,缺失/不可解析恒排最后;
- 空结果 → items=[] + meta.count=0(HTTP 200,不报错)。
"""
ccf_norm = (ccf or "").strip().upper()
if ccf_norm and ccf_norm not in _CCF_OK:
return _error("VALIDATION", f"不支持的 ccf={ccf!r},可选:A / B / C")
if upcoming_days < 0:
return _error(
"VALIDATION", f"upcoming_days 必须 ≥ 0,收到 {upcoming_days}"
)
all_items = _load_conferences()
if all_items is None:
return _error("DATA_UNAVAILABLE", "会议种子数据不可读,请联系管理员")
items = list(all_items)
dom = (domain or "").strip().casefold()
if dom:
items = [
c for c in items
if any(
str(t).strip().casefold() == dom
for t in (c.get("domain_tags") or [])
)
]
if ccf_norm:
items = [
c for c in items
if str(c.get("ccf_rank") or "").strip().upper() == ccf_norm
]
if upcoming_days > 0:
now = datetime.now(timezone.utc)
limit = now + timedelta(days=upcoming_days)
items = [
c for c in items
if (dl := _parse_deadline(c.get("deadline_full_paper"))) is not None
and now <= dl <= limit
]
def _sort_key(c: dict[str, Any]) -> tuple[bool, datetime]:
dl = _parse_deadline(c.get("deadline_full_paper"))
if dl is None:
return (True, datetime.max.replace(tzinfo=timezone.utc))
return (False, dl)
items.sort(key=_sort_key)
out = [ConferenceItem(**c) for c in items]
verified = max(
(str(c.get("last_verified") or "") for c in all_items), default=""
)
meta: dict[str, Any] = {
"count": len(out),
"last_verified": verified or None,
"data_source_note": _SOURCE_NOTE,
"query": {
"domain": (domain or "").strip(),
"ccf": ccf_norm,
"upcoming_days": upcoming_days,
},
}
return ConferencesResponse(ok=True, items=out, meta=meta, error=None)
@@ -1,81 +0,0 @@
"""routers/format_check.py —— 版式与规范检查(POST /api/review/format-check)。
纯规则引擎(services/format_check.py):PDF/Word 本地解档取版式元数据 +
journals.json 规则集(format_rules_default + 每条 format_rules 带 rules_source)。
无 LLM / 无 vstore,同步执行通常 <1s;恒 200 + {ok, checks, summary, meta|error} 信封。
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, File, Form, UploadFile
from pydantic import BaseModel
from ..config import get_settings
from ..services import format_check
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-format"])
class CheckItem(BaseModel):
key: str
name: str
status: str # pass | warn | fail
detail: str
suggestion: str = ""
class FormatCheckResponse(BaseModel):
ok: bool
checks: list[CheckItem] | None = None
summary: dict[str, int] | None = None
meta: dict[str, object] | None = None
error: dict[str, object] | None = None
def _error(code: str, message: str, *, degraded: bool = False) -> FormatCheckResponse:
return FormatCheckResponse(
ok=False, error={"code": code, "message": message, "degraded": degraded},
)
@router.post("/format-check", response_model=FormatCheckResponse)
async def format_check_endpoint(
file: UploadFile = File(..., description="待检查的 PDF / Word(.docx) / LaTeX(.tex) 文档"),
target_journal: str = Form("", description="目标期刊(可选,传 journals 的 id)"),
) -> FormatCheckResponse:
settings = get_settings()
filename = file.filename or ""
low = filename.lower()
if low.endswith(".pdf"):
kind = "pdf"
elif low.endswith(".docx"):
kind = "docx"
elif low.endswith(".tex"):
kind = "tex"
else:
return _error("VALIDATION", "仅支持 PDF / Word(.docx)/ LaTeX(.tex)文件")
content = await file.read()
if not content:
return _error("VALIDATION", "上传文件为空")
if len(content) > settings.max_pdf_bytes:
return _error("VALIDATION", "上传文件超过 50MB 上限")
try:
result = format_check.check_document(content, kind, (target_journal or "").strip())
except format_check.FormatCheckError as exc:
logger.warning("format-check: 解档失败:%s", exc)
return _error("PARSE_FAILED", str(exc))
meta = dict(result.get("meta") or {})
meta["filename"] = filename
return FormatCheckResponse(
ok=True,
checks=[CheckItem(**c) for c in result["checks"]],
summary=result["summary"],
meta=meta,
)
@@ -1,280 +0,0 @@
"""routers/journal_directory.py —— 期刊目录浏览(1056 本,只读分页)。
- GET /api/review/journals/directory:journal_directory 表(scripts/import_journals.py
导入的 v4 目录:2025 中科院分区 + 2025 JIF(2026 年 JCR 发布)+ 年报 2025 版
(中文统计年 2024)+ 官网网址/核对状态)→ 过滤/排序/分页 → 列表 + meta。
- 与 GET /journals(journals.json 精选 5 本,供 FormatPanel 下拉/recommend 语义)物理
分离(方案 A);本端点绝不读写 journals 表,recommend/journal_kb/journal_rank 零触碰。
- 读路径只 SELECT;表由导入脚本拥有(db.py 零改动);表不存在 → data_imported=false
优雅降级(HTTP 200 空列表)。
读时归一化(拍板细节 1-3,均在 API 层做,不动脚本/DB 原值):
细节1 lang=zh → cas_partition 恒 null + cas_partition_note="不按中科院分区口径列示";
细节2 jif_note 源文本映射(v4 值域已是用户可读文本,旧值映射保留兜底);
细节3 缺值即矛盾清零:en 且 jif=null → jif_source_url=null。
(zh 的 IF 载体是 composite/comprehensive,其 jif_source_url 本恒 null。)
决策2:响应不下发"来源"文本字段(partition_source/jif_source/data_source_label
已移除);jif_source_url/data_source_url 保留供展开行链接。
决策4(来源展示优化):partition_source_url 不再下发——DB 原值分毫未动
(仍 SELECT * 读出后被模型丢弃),响应改为常量字段 partition_source_note
文案,前端以灰色说明文字替代原"分区来源"超链接。
新增 official_check_status / jcr_release_year。
"""
from __future__ import annotations
import json
import logging
import sqlite3
from typing import Any
from fastapi import APIRouter
from pydantic import BaseModel
from ..db import db
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-directory"])
_LANGS = frozenset({"en", "zh"})
# 排序列白名单(列名不可参数化 → 白名单防注入)
_FIELDS = frozenset({
"name",
"jif",
"composite_impact_factor",
"comprehensive_impact_factor",
"research_direction",
"cas_partition",
})
_ORDERS = frozenset({"asc", "desc"})
_PARTS = {"1": "1区", "2": "2区", "3": "3区", "4": "4区"}
_MAX_SIZE = 200
# 细节2:源核对状态 → 用户可读 note
_JIF_NOTE_MAP = {
"源表标N/A": "未被2024 JCR收录",
"所用年度来源未检得": "本次汇编未查到,建议查阅最新 JCR",
}
_ZH_PART_NOTE = "不按中科院分区口径列示"
# 决策4:分区来源改为常量说明文案(不下发 partition_source_url;DB 原值不动)
_PARTITION_SOURCE_NOTE = (
"2025 中科院官方表(2026 年起停更;官方入口 fenqubiao.com 需机构登录)"
)
_NOTE_EN = (
"分区:2025年中科院公开表(多表比对,2026年起停止更新);"
"JIF:2025年度指标(2026年JCR发布,两份公开汇编一致或已核对出版社官方指标);"
"整理日期2026-10-10。空白=来源未给值≠0;研究方向/期刊标记/文章类型为导航标签,非官方评级。"
)
_NOTE_ZH = (
"指标:国内自然科学期刊年报2025版(统计年2024,official,含复合/综合IF两口径);"
"中文期刊不按中科院分区口径列示;国内IF与JIF统计源不同,不可混排。"
)
_UNIMPORTED_NOTE = "目录数据未导入(请先运行 scripts/import_journals.py)"
# ---- 响应模型 ----
class DirectoryItem(BaseModel):
"""目录公开字段(DB 全列白名单;imported_at 等内部列丢弃)。"""
id: str
lang: str
name: str
research_direction: str | None = None
cas_partition: str | None = None
cas_partition_note: str | None = None
cas_partition_year: int | None = None
jif: float | None = None
jif_year: int | None = None
jif_note: str | None = None
jif_check_status: str | None = None
composite_impact_factor: float | None = None
comprehensive_impact_factor: float | None = None
metric_year: int | None = None
metric_report_year: int | None = None
metric_caliber: str | None = None
annual_report_page: int | None = None
top_flag: str | None = None
publisher_series: str | None = None
journal_mark: str | None = None
article_type: str | None = None
issn: str | None = None
eissn: str | None = None
jcr_disciplines: list[str] = []
supplement_note: str | None = None
evidence_note: str | None = None
official_url: str | None = None
official_check_status: str | None = None
jcr_release_year: int | None = None
jif_source_url: str | None = None
partition_source_note: str = _PARTITION_SOURCE_NOTE # 决策4:常量,替代分区来源链接
data_source_url: str | None = None
class DirectoryResponse(BaseModel):
ok: bool
items: list[DirectoryItem] = []
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
def _error(code: str, message: str) -> DirectoryResponse:
return DirectoryResponse(
ok=False, items=[], meta=None,
error={"code": code, "message": message, "degraded": False},
)
# ---- 读时归一化(拍板细节 1-3)----
def _present(raw: dict[str, Any], lang: str) -> dict[str, Any]:
d = dict(raw)
# jcr_disciplines:JSON 数组字符串 → list
jcr = d.get("jcr_disciplines")
if jcr:
try:
parsed = json.loads(jcr)
d["jcr_disciplines"] = parsed if isinstance(parsed, list) else []
except (TypeError, ValueError):
d["jcr_disciplines"] = []
else:
d["jcr_disciplines"] = []
# 细节2:jif_note 源文本 → 用户可读
note = d.get("jif_note")
if note in _JIF_NOTE_MAP:
d["jif_note"] = _JIF_NOTE_MAP[note]
# 细节1:zh 分区归一(DB 原值"不按此口径列示"只留在库里)
if lang == "zh":
d["cas_partition"] = None
d["cas_partition_note"] = _ZH_PART_NOTE
# 细节3:缺值 → 来源链接清零(消除 null 值却挂来源链接的矛盾;
# 分区来源链接已按决策4 移出响应,改为常量 partition_source_note)
if lang == "en":
if d.get("jif") is None:
d["jif_source_url"] = None
return d
def _escape_like(s: str) -> str:
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
# ---- GET /api/review/journals/directory ----
@router.get("/journals/directory", response_model=DirectoryResponse)
async def journals_directory(
lang: str = "en",
domain: str = "",
cas_partition: str = "",
q: str = "",
page: int = 1,
size: int = 50,
field: str = "",
order: str = "",
) -> DirectoryResponse:
"""只读:期刊目录分页浏览(无写接口)。
- ?lang=en|zh(默认 en;**强制单语,保证国际/中文永不混排**);
- ?domain= 研究方向精确匹配;?cas_partition=1|2|3|4(仅 lang=en,zh 忽略);
- ?q= 期刊名模糊(ASCII 大小写不敏感);?page≥1 ?size 1-200(默认 50);
- ?field= 排序白名单(name/jif/composite_impact_factor/
comprehensive_impact_factor/research_direction/cas_partition),?order=asc|desc;
未传 field → 默认排序 en=jif 降序 / zh=复合IF 降序(NULL 恒排最后);
- 参数非法 → VALIDATION 信封;表不存在 → data_imported=false 空列表(HTTP 200)。
"""
lang_n = (lang or "").strip().lower()
if lang_n not in _LANGS:
return _error("VALIDATION", f"不支持的 lang={lang!r},可选:en / zh")
if page < 1:
return _error("VALIDATION", f"page 必须 ≥ 1,收到 {page}")
if size < 1 or size > _MAX_SIZE:
return _error("VALIDATION", f"size 必须在 1-{_MAX_SIZE},收到 {size}")
field_n = (field or "").strip()
if field_n and field_n not in _FIELDS:
return _error(
"VALIDATION",
f"不支持的 field={field!r},可选:{', '.join(sorted(_FIELDS))}",
)
order_n = (order or "").strip().lower()
if order_n and order_n not in _ORDERS:
return _error("VALIDATION", f"不支持的 order={order!r},可选:asc / desc")
part_q = (cas_partition or "").strip()
if part_q:
part_q = _PARTS.get(part_q, _PARTS.get(part_q.rstrip("区").strip(), ""))
if not part_q:
return _error(
"VALIDATION",
f"不支持的 cas_partition={cas_partition!r},可选:1 / 2 / 3 / 4",
)
# 过滤条件(白名单拼接;参数全部占位符)
where = ["lang = ?"]
params: list[Any] = [lang_n]
domain_n = (domain or "").strip()
if domain_n:
where.append("research_direction = ?")
params.append(domain_n)
if part_q and lang_n == "en": # zh 不分区:忽略(meta 回显)
where.append("cas_partition = ?")
params.append(part_q)
q_n = (q or "").strip()
if q_n:
where.append("name LIKE ? ESCAPE '\\'")
params.append(f"%{_escape_like(q_n)}%")
where_sql = " AND ".join(where)
# 排序(列名出自白名单;NULL 恒排最后)
if field_n:
dir_sql = "DESC" if order_n == "desc" else "ASC"
order_sql = f"{field_n} IS NULL, {field_n} {dir_sql}"
else:
col = "jif" if lang_n == "en" else "composite_impact_factor"
order_sql = f"{col} IS NULL, {col} DESC"
try:
total_row = await db.fetchone(
f"SELECT COUNT(*) AS c FROM journal_directory WHERE {where_sql}", params # noqa: S608
)
total = int(total_row["c"]) if total_row else 0
rows = await db.fetchall(
f"SELECT * FROM journal_directory WHERE {where_sql} " # noqa: S608
f"ORDER BY {order_sql} LIMIT ? OFFSET ?",
[*params, size, (page - 1) * size],
)
dom_rows = await db.fetchall(
"SELECT research_direction AS d, COUNT(*) AS c FROM journal_directory "
"WHERE lang = ? AND research_direction IS NOT NULL "
"GROUP BY 1 ORDER BY c DESC, d ASC",
[lang_n],
)
except sqlite3.OperationalError as exc:
# 表不存在(脚本未跑)→ 优雅降级,HTTP 200
logger.warning("journal_directory: 表不可用(目录未导入?): %s", exc)
return DirectoryResponse(
ok=True, items=[],
meta={
"total": 0, "page": page, "size": size, "lang": lang_n,
"domain": domain_n, "cas_partition": part_q, "q": q_n,
"domains": [], "data_source_note": _UNIMPORTED_NOTE,
"data_imported": False,
},
error=None,
)
items = [DirectoryItem(**_present(dict(r), lang_n)) for r in rows]
meta: dict[str, Any] = {
"total": total,
"page": page,
"size": size,
"lang": lang_n,
"domain": domain_n,
"cas_partition": part_q if lang_n == "en" else "",
"q": q_n,
"domains": [r["d"] for r in dom_rows if r["d"]],
"data_source_note": _NOTE_EN if lang_n == "en" else _NOTE_ZH,
"data_imported": True,
}
return DirectoryResponse(ok=True, items=items, meta=meta, error=None)
-354
View File
@@ -1,354 +0,0 @@
"""routers/journals.py —— 期刊情报与智能投稿推荐。
- GET /api/review/journals 只读:journals 表全量列表(?field=/&order= 排序),
meta 含 count / last_updated / data_source_note。无任何写接口;空表返回
空数组 + meta.count=0(HTTP 200,绝不 500)。
- POST /api/review/recommend 智能投稿推荐:vstore 向量匹配(摘要 vs
scope_desc,白名单检索)→ LLM 按 id 白名单重排(幻觉期刊丢弃、缺漏规则
补位)→ 三档输出:挑战型 / 适配型 / 稳妥型,附匹配理由与风险提示。
降级哲学(同 presubmit):向量失败 → 全量候选 + vector_ok=false;
LLM 失败 → 规则分档 + llm_ok=false;推荐永不 5xx。仅入参非法返回
VALIDATION 错误信封(HTTP 200,与本服务其余端点一致)。
"""
from __future__ import annotations
import json
import logging
from functools import lru_cache
from pathlib import Path
from typing import Any, Literal
from fastapi import APIRouter
from pydantic import BaseModel, Field
from ..config import get_settings
from ..db import SeedError, ensure_journals_seeded, list_journals
from ..services import journal_kb, journal_rank, llm
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-journals"])
# ---- 请求/响应模型 ----
class Preferences(BaseModel):
max_review_weeks: float | None = Field(default=None, ge=1, le=104)
min_impact_factor: float | None = Field(default=None, ge=0, le=200)
tier_focus: Literal["all", "challenge", "fit", "safe"] = "all"
class RecommendRequest(BaseModel):
abstract: str
preferences: Preferences = Preferences()
class JournalItem(BaseModel):
id: str
name: str
impact_factor: float | None = None
review_time_weeks: float | None = None
acceptance_rate: float | None = None
official_url: str | None = None
similarity: float | None = None
tier: str
match_reason: str
risk_note: str
class TierBox(BaseModel):
label: str
items: list[JournalItem]
class RecommendResponse(BaseModel):
ok: bool
tiers: dict[str, TierBox] | None = None
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
# ---- GET /api/review/journals(只读列表)----
class JournalBrief(BaseModel):
"""期刊公开字段(journals 表全列;不含 vstore_doc_id 内部映射)。
标签/分区 6 字段不进 SQLite:GET 时按 id 从 journals.json 内存缓存合并;
JSON 缺该 id → 空数组/null 默认值。
"""
id: str
name: str
scope_desc: str = ""
impact_factor: float | None = None
review_time_weeks: float | None = None
acceptance_rate: float | None = None
submission_deadline: str | None = None
official_url: str | None = None
last_updated: str = ""
scope_tags: list[str] = []
application_tags: list[str] = []
cas_partition: str | None = None
jcr_quartile: str | None = None
partition_source: str | None = None
cross_domain: list[str] = []
class JournalsListResponse(BaseModel):
ok: bool
items: list[JournalBrief] = []
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
# ?field= 排序白名单(列名无法参数化,白名单防注入;非法值 → VALIDATION 信封)
_SORT_FIELDS = frozenset({
"id",
"name",
"impact_factor",
"review_time_weeks",
"acceptance_rate",
"last_updated",
})
_DATA_SOURCE_NOTE = (
"IF/审稿周期为公开近似值,acceptance_rate 为 null 表示暂无公开数据,需人工核验。"
"CAS/JCR 分区为首版占位数据,需核验。"
)
# journals.json 新增字段(标签/分区;刻意不进 SQLite,见 C-2 方案)
_TAG_FIELDS = (
"scope_tags",
"application_tags",
"cas_partition",
"jcr_quartile",
"partition_source",
"cross_domain",
)
@lru_cache(maxsize=1)
def _load_tag_extras() -> dict[str, dict[str, Any]]:
"""journals.json 标签/分区字段 → {id: {6 新字段}}。
首次请求读盘后进程内缓存,后续 GET O(1) 按 id 合并(不做每请求磁盘 IO)。
JSON 缺该 id → 合并时用空数组/null 默认;文件不可读 → 空 dict 降级(不 500)。
"""
seed = get_settings().journals_seed_abs
try:
payload = json.loads(Path(seed).read_text("utf-8"))
except (OSError, ValueError) as exc:
logger.warning("journals: 标签扩展文件不可读,合并降级为默认值:%s", exc)
return {}
raw = payload.get("journals") if isinstance(payload, dict) else None
if not isinstance(raw, list):
return {}
out: dict[str, dict[str, Any]] = {}
for j in raw:
if isinstance(j, dict) and j.get("id"):
out[str(j["id"])] = {k: j.get(k) for k in _TAG_FIELDS}
return out
def _tag_fields(extra: dict[str, Any] | None) -> dict[str, Any]:
"""缓存条目 → JournalBrief 关键字参数(JSON 缺该 id → 空数组/null 默认)。"""
ex = extra or {}
return {
"scope_tags": list(ex.get("scope_tags") or []),
"application_tags": list(ex.get("application_tags") or []),
"cas_partition": ex.get("cas_partition"),
"jcr_quartile": ex.get("jcr_quartile"),
"partition_source": ex.get("partition_source"),
"cross_domain": list(ex.get("cross_domain") or []),
}
def _sort_rows(rows: list[dict[str, Any]], field: str, order: str) -> list[dict[str, Any]]:
"""按白名单字段排序;NULL 恒排最后;次序不稳定时按 db 默认序(稳定排序保留)。"""
present = [r for r in rows if r.get(field) is not None]
missing = [r for r in rows if r.get(field) is None]
present.sort(key=lambda r: r.get(field), reverse=(order == "desc"))
return present + missing
def _list_error(message: str) -> JournalsListResponse:
return JournalsListResponse(
ok=False, items=[], meta=None,
error={"code": "VALIDATION", "message": message, "degraded": False},
)
@router.get("/journals", response_model=JournalsListResponse)
async def journals_list(
field: str = "name",
order: str = "asc",
) -> JournalsListResponse:
"""只读:journals 表全量列表(无写/删/改接口)。
- ?field= 排序字段白名单:id/name/impact_factor/review_time_weeks/
acceptance_rate/last_updated,默认 name;
- ?order=asc|desc,默认 asc;
- 空表 → items=[] + meta.count=0(HTTP 200,不报错)。种子装载由
POST /recommend 首次调用幂等完成,本端点严格只读。
"""
col = (field or "").strip()
if col not in _SORT_FIELDS:
return _list_error(
f"不支持的排序字段 field={field!r},可选:{', '.join(sorted(_SORT_FIELDS))}"
)
ord_norm = (order or "asc").strip().lower()
if ord_norm not in ("asc", "desc"):
return _list_error(f"不支持的排序方向 order={order!r},可选:asc, desc")
rows = await list_journals()
rows = _sort_rows(rows, col, ord_norm)
tag_extras = _load_tag_extras()
items = [
JournalBrief(
id=str(r.get("id") or ""),
name=str(r.get("name") or ""),
scope_desc=str(r.get("scope_desc") or ""),
impact_factor=r.get("impact_factor"),
review_time_weeks=r.get("review_time_weeks"),
acceptance_rate=r.get("acceptance_rate"),
submission_deadline=r.get("submission_deadline"),
official_url=r.get("official_url"),
last_updated=str(r.get("last_updated") or ""),
**_tag_fields(tag_extras.get(str(r.get("id") or ""))),
)
for r in rows
]
latest = max((str(r.get("last_updated") or "") for r in rows), default="")
meta: dict[str, Any] = {
"count": len(items),
"last_updated": latest or None,
"data_source_note": _DATA_SOURCE_NOTE,
}
return JournalsListResponse(ok=True, items=items, meta=meta, error=None)
def _error(code: str, message: str, *, degraded: bool = False) -> RecommendResponse:
return RecommendResponse(
ok=False, tiers=None, meta=None,
error={"code": code, "message": message, "degraded": degraded},
)
def _item_sort_key(it: JournalItem) -> tuple[int, float, float]:
return (
1 if it.similarity is None else 0,
-(it.similarity or 0.0),
-(it.impact_factor or 0.0),
)
def _apply_soft_preferences(item: JournalItem, prefs: Preferences) -> None:
"""软偏好:不满足只加风险提示,不删条目。"""
weeks = item.review_time_weeks
if prefs.max_review_weeks is not None and weeks is not None and weeks > prefs.max_review_weeks:
prefix = f"⚠ 审稿约 {weeks:g} 周,超出你 {prefs.max_review_weeks:g} 周的周期偏好;"
item.risk_note = prefix + ("无" if item.risk_note == "无" else item.risk_note)
inf = item.impact_factor
if prefs.min_impact_factor is not None:
if inf is None:
item.risk_note = "⚠ 影响因子未收录,无法确认满足你的要求;" + item.risk_note
elif inf < prefs.min_impact_factor:
prefix = f"⚠ 影响因子 {inf:g} 低于你 {prefs.min_impact_factor:g} 的要求;"
item.risk_note = prefix + ("无" if item.risk_note == "无" else item.risk_note)
@router.post("/recommend", response_model=RecommendResponse)
async def recommend(req: RecommendRequest) -> RecommendResponse:
settings = get_settings()
# ---- 入参校验(手动,错误信封与本服务其余端点一致)----
abstract = (req.abstract or "").strip()
if len(abstract) < 50:
return _error("VALIDATION", "abstract 至少需要 50 个字符(请提供摘要或正文片段)")
if len(abstract) > 20000:
return _error("VALIDATION", "abstract 超过 20000 字符上限")
prefs = req.preferences
# ---- 种子 + 期刊元数据 ----
try:
await ensure_journals_seeded()
journals = await list_journals()
except SeedError as exc:
logger.error("recommend: 种子装载失败:%s", exc)
return _error("SEED_UNAVAILABLE", str(exc), degraded=True)
if not journals:
return _error("SEED_UNAVAILABLE", "期刊库为空", degraded=True)
# ---- ① vstore 向量匹配(摘要 vs scope_desc)----
try:
candidates, vector_ok = await journal_kb.match(abstract, journals)
except journal_kb.MatchError as exc:
logger.warning("recommend: 向量匹配降级:%s", exc)
candidates, vector_ok = [], False
if not candidates:
# 无有效语义分 → 全量候选(similarity=None),LLM 仍可按 scope 文本重排
candidates = [{"journal": j, "similarity": None} for j in journals]
vector_ok = False
# ---- ② LLM 重排(id 白名单硬校验)----
prefs_dict = prefs.model_dump(exclude_none=True)
llm_ok = True
dropped = 0
repaired = 0
try:
rankings, dropped, repaired = await journal_rank.rerank(abstract, candidates, prefs_dict)
except (llm.LLMError, journal_rank.RankError, ValueError) as exc:
# ValueError:complete_json 最终 json.loads 修复仍失败(同 presubmit 宽容口径)
logger.warning("recommend: LLM 重排降级为规则分档:%s", exc)
rankings = journal_rank.fallback_rankings(candidates)
llm_ok = False
# ---- ③ 三档组装(名称/指标全部来自 DB,不采信 LLM 文本)----
by_id = {str(c["journal"]["id"]): c for c in candidates}
tiers: dict[str, list[JournalItem]] = {t: [] for t in journal_rank.TIERS}
for r in rankings:
cand = by_id.get(r.journal_id)
if cand is None:
continue
j = cand["journal"]
sim = cand.get("similarity")
item = JournalItem(
id=str(j["id"]),
name=str(j.get("name") or j["id"]),
impact_factor=j.get("impact_factor"),
review_time_weeks=j.get("review_time_weeks"),
acceptance_rate=j.get("acceptance_rate"),
official_url=j.get("official_url"),
similarity=round(float(sim), 4) if sim is not None else None,
tier=r.tier,
match_reason=r.match_reason,
risk_note=r.risk_note,
)
_apply_soft_preferences(item, prefs)
tiers.setdefault(r.tier, []).append(item)
# 档内排序:相似度(None 靠后)→ 影响因子
for box in tiers.values():
box.sort(key=_item_sort_key)
# tier_focus:软展示聚焦(默认 all;聚焦时其余档 items 置空,label 保留)
out_tiers: dict[str, TierBox] = {}
for t in journal_rank.TIERS:
items = tiers.get(t, [])
if prefs.tier_focus != "all" and t != prefs.tier_focus:
items = []
out_tiers[t] = TierBox(label=journal_rank.TIER_LABELS[t], items=items)
meta: dict[str, Any] = {
"model": llm._llm_config()["model"],
"vector_ok": vector_ok,
"llm_ok": llm_ok,
"seed_count": len(journals),
"candidate_count": len(candidates),
"dropped_hallucinated": dropped,
"repaired": repaired,
"preferences": prefs_dict,
"abstract_chars": len(abstract),
}
return RecommendResponse(ok=True, tiers=out_tiers, meta=meta, error=None)
-269
View File
@@ -1,269 +0,0 @@
"""routers/presubmit.py —— 投稿前自审稿(POST /api/review/presubmit)。
流程:上传 PDF → paper_parse(vstore 8094 解析)→ 段落感知装箱 + token 双层保护
→ LLM 结构化审稿(DeepSeek json_object,System Prompt 为植物基因组学六维专属
清单,见 services/self_review.py)→ coerce 归一化 → 关键词扫描器证伪硬性缺项
(services/paper_scanner.py)→ 规则引擎计算 recommendation。
"""
from __future__ import annotations
import logging
import re
from fastapi import APIRouter, File, Form, UploadFile
from pydantic import BaseModel
from ..config import get_settings
from ..services import llm, paper_parse, paper_scanner, self_review
from ..services.recommend_engine import decide_recommendation, resolve_tier
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-presubmit"])
# ---- token 双层保护(2026-10-12)----
# 第一层:估算 token 超阈值 → 自动降到 TOKEN_GUARD_REPACK_CHARS 字符重装箱
# 第二层:LLM 返回 HTTP 400/413 → 降到 LLM_REJECT_RETRY_CHARS 字符重试一次
_EST_TOKENS_DIVISOR = 1.5 # 粗略上界:英文 ~4 字符/token,1.5 偏保守
_TOKEN_GUARD_THRESHOLD = 48000
_TOKEN_GUARD_REPACK_CHARS = 100000
_LLM_REJECT_RETRY_CHARS = 80000
# 无节标题可辨时的尾窗(捞页尾 Data Availability 块)
_TAIL_FALLBACK_CHARS = 5000
# 节标题(行首、允许编号前缀、独占一行);键 → 装箱分组名
_SECTION_HEADER_RE = re.compile(
r"^[ \t]*(?:\d+(?:\.\d+)*\.?\s+)?"
r"(Abstract|Summary|Introduction|Results|Discussion|Methods"
r"|Materials\s+and\s+Methods"
r"|Data\s+(?:and\s+(?:code\s+)?)?[Aa]vailability"
r"|Availability\s+of\s+[Dd]ata"
r"|Data\s+and\s+[Mm]aterials\s+[Aa]vailability"
r"|References|Acknowledg(?:e)?ments?)\s*$",
re.IGNORECASE | re.MULTILINE,
)
_SECTION_GROUP = {
"abstract": "abstract", "summary": "abstract",
"introduction": "introduction",
"results": "results",
"discussion": "discussion",
"methods": "methods", "materials and methods": "methods",
"data availability": "data_availability",
"data and code availability": "data_availability",
"data availability statement": "data_availability",
"availability of data": "data_availability",
"data and materials availability": "data_availability",
"references": "references", "acknowledgements": "acknowledgements",
"acknowledgments": "acknowledgements",
}
# 装箱优先级(首条命中优先;methods 限 60% 预算——六维清单大量依赖 Methods)
_PACK_PRIORITY = ("abstract", "data_availability", "methods", "results", "discussion")
_METHODS_CAP_RATIO = 0.6
def estimate_tokens(text: str) -> int:
"""粗略 token 上界估计(len/1.5),无需 tokenizer 依赖。"""
return int(len(text) / _EST_TOKENS_DIVISOR)
def _split_segments(text: str) -> list[tuple[str, str]] | None:
"""按节标题切段 → [(group, content)...]。无可辨标题 → None。"""
matches = list(_SECTION_HEADER_RE.finditer(text))
if not matches:
return None
segments: list[tuple[str, str]] = []
if matches[0].start() > 0:
segments.append(("preamble", text[:matches[0].start()]))
for i, m in enumerate(matches):
end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
group = _SECTION_GROUP.get(m.group(1).casefold(), m.group(1).casefold())
segments.append((group, text[m.start():end]))
return segments
def pack_feed_text(text: str, budget: int) -> tuple[str, list[str]]:
"""段落感知装箱 → (feed_text, dropped_sections)。
预算内:原文返回。超预算:按优先级保留
① Abstract ② Data Availability 全文 ③ Methods 前 60% ④ Results
⑤ Discussion,其余按原序填满;未收段记入 dropped。
无节标题可辨 → 头 + 尾窗拼接(捞页尾 availability 块),不优于现状亦不劣于。
"""
if len(text) <= budget:
return text, []
segments = _split_segments(text)
if not segments:
sep = "\n[...]\n"
head = max(0, budget - _TAIL_FALLBACK_CHARS - len(sep))
return (
text[:head] + sep + text[-_TAIL_FALLBACK_CHARS:],
["[无节标题回退:保留头+尾窗]"],
)
methods_cap = int(budget * _METHODS_CAP_RATIO)
chosen: dict[int, str] = {}
used = 0
for group in _PACK_PRIORITY:
for i, (g, content) in enumerate(segments):
if g != group or i in chosen:
continue
room = budget - used
if room <= 0:
break
piece = content[:methods_cap] if group == "methods" else content
if len(piece) > room:
piece = piece[:room]
if not piece:
break
chosen[i] = piece
used += len(piece)
break
for i, (_, content) in enumerate(segments):
if i in chosen or used >= budget:
continue
room = budget - used
chosen[i] = content if len(content) <= room else content[:room]
used += len(chosen[i])
feed = "".join(chosen[i] for i in sorted(chosen))
dropped: list[str] = []
for i, (g, _) in enumerate(segments):
if i not in chosen and g not in dropped:
dropped.append(g)
return feed, dropped
def _fit_feed_text(text: str, budget: int) -> tuple[str, list[str]]:
"""装箱 + 第一层 token 保护(估算超阈值自动降到 100K 字符重装)。"""
feed, dropped = pack_feed_text(text, budget)
if estimate_tokens(feed) > _TOKEN_GUARD_THRESHOLD and len(feed) > _TOKEN_GUARD_REPACK_CHARS:
logger.info(
"presubmit: token 估算 %d > %d,降到 %d 字符重装箱",
estimate_tokens(feed), _TOKEN_GUARD_THRESHOLD, _TOKEN_GUARD_REPACK_CHARS,
)
feed, dropped = pack_feed_text(text, _TOKEN_GUARD_REPACK_CHARS)
return feed, dropped
class PresubmitResponse(BaseModel):
ok: bool
report: self_review.SelfReviewReport | None = None
error: dict[str, object] | None = None
meta: dict[str, object] | None = None
def _error(code: str, message: str, *, degraded: bool = False) -> PresubmitResponse:
return PresubmitResponse(
ok=False, report=None,
error={"code": code, "message": message, "degraded": degraded},
)
@router.post("/presubmit", response_model=PresubmitResponse)
async def presubmit_review(
file: UploadFile = File(..., description="待审稿 PDF"),
target_journal: str = Form("", description="目标期刊(可选,影响格式类意见)"),
) -> PresubmitResponse:
settings = get_settings()
filename = file.filename or "paper.pdf"
if not filename.lower().endswith(".pdf"):
return _error("VALIDATION", "仅支持 PDF 文件")
content = await file.read()
if not content:
return _error("VALIDATION", "上传文件为空")
if len(content) > settings.max_pdf_bytes:
return _error("VALIDATION", "PDF 超过 50MB 上限")
# ---- 解析(vstore 8094)----
try:
text = await paper_parse.parse_pdf_bytes(content, filename=filename)
except paper_parse.ParseError as exc:
logger.warning("presubmit: 解析失败:%s", exc)
return _error("PARSE_FAILED", str(exc), degraded=True)
truncated = len(text) > settings.max_text_chars
feed_text, dropped_sections = _fit_feed_text(text, settings.max_text_chars)
# ---- LLM 结构化自审(植物基因组学六维清单)----
# 第二层保护:HTTP 400/413(疑似超长/形态拒绝)→ 降 80K 字符重试一次
user_prompt = self_review.build_user_prompt(feed_text, (target_journal or "").strip())
try:
try:
payload = await llm.complete_json(
self_review.SYSTEM_PROMPT,
user_prompt,
temperature=0.2,
max_tokens=8000,
timeout=settings.presubmit_read_timeout,
)
except llm.LLMError as exc:
msg = str(exc)
if ("HTTP 400" in msg or "HTTP 413" in msg) and len(feed_text) > _LLM_REJECT_RETRY_CHARS:
logger.warning("presubmit: LLM 拒绝输入(%s),降到 %d 字符重试",
msg[:120], _LLM_REJECT_RETRY_CHARS)
feed_text, dropped_sections = pack_feed_text(text, _LLM_REJECT_RETRY_CHARS)
truncated = True
payload = await llm.complete_json(
self_review.SYSTEM_PROMPT,
self_review.build_user_prompt(feed_text, (target_journal or "").strip()),
temperature=0.2,
max_tokens=8000,
timeout=settings.presubmit_read_timeout,
)
else:
raise
except llm.LLMError as exc:
logger.error("presubmit: LLM 审稿失败:%s", exc)
return _error("UPSTREAM_DOWN", f"审稿模型调用失败:{exc}", degraded=True)
report = self_review.coerce_report(payload)
# ---- 关键词扫描器:证伪 LLM 的硬性缺项(有 X 却报缺 X → 真降级至 minor)----
# A2(2026-10-12):被冲突的 major 不再留在 major 列表里只剥标签,而是移出
# major、append 到 minor(scanner_downgraded=True)——major_count/引擎输入真实
# 减少;fatal 条目例外(只记冲突不降级,须人工过目)。
signals = paper_scanner.scan_paper_signals(feed_text)
kept_majors, demoted_minors, scanner_conflicts = paper_scanner.reconcile_hard_rules(
report.major_weaknesses, signals
)
report.major_weaknesses = kept_majors
if demoted_minors:
report.minor_weaknesses = list(report.minor_weaknesses) + list(demoted_minors)
# recommendation 由规则引擎计算(LLM 输出的同名字段已被 coerce 忽略)
tier, tier_source = resolve_tier((target_journal or "").strip())
decision = decide_recommendation(
majors=report.major_weaknesses,
minors=report.minor_weaknesses,
presentations=report.presentation_format,
tier=tier,
)
rationale = decision["rationale"]
rationale["scanner_conflicts"] = scanner_conflicts
if scanner_conflicts:
rationale["explanation"] += (
f" 关键词扫描器与审稿意见存在 {len(scanner_conflicts)} 处冲突,"
f"已降级为小修级问题,请人工核对。"
)
report.recommendation = decision["recommendation"]
report.recommendation_rationale = rationale
return PresubmitResponse(
ok=True,
report=report,
meta={
"filename": filename,
"target_journal": (target_journal or "").strip(),
"text_chars": len(feed_text),
"truncated": truncated,
"dropped_sections": dropped_sections,
"scanner": {
"accession_found": bool((signals.get("accession") or {}).get("found")),
"busco_level": (signals.get("busco") or {}).get("level"),
"hic_found": bool((signals.get("hic") or {}).get("found")),
},
"model": llm._llm_config()["model"],
"tier": tier,
"tier_label": decision["rationale"]["tier_label"],
"tier_source": tier_source,
},
)
-288
View File
@@ -1,288 +0,0 @@
"""routers/rebuttal.py —— Rebuttal Phase 1(审稿回复助手,10 端点)。
信封与其余 review 端点一致:业务错误 = HTTP 200 + {ok:false,error:{code,message,degraded}}。
- 文本粘贴主入口 / PDF 次要入口(PDF 经 vstore 8094 解析);
- 逐条 response → draft(校验C前置)→ confirm(后端重算 human_ack)→ letter 拼装;
- letter.md 下载门槛 = human_ack;DELETE 手动清(无 TTL,决策 5)。
"""
from __future__ import annotations
import logging
from typing import Any, Literal
from fastapi import APIRouter, File, UploadFile
from fastapi.responses import PlainTextResponse
from pydantic import BaseModel, Field
from ..config import get_settings
from ..services import llm, paper_parse
from ..services.rebuttal_draft import PART_TYPES, assemble_letter, generate_draft, validate_parts
from ..services.rebuttal_parse import parse_review_text
from ..services import rebuttal_store
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review/rebuttal", tags=["review-rebuttal"])
_MAX_RAW_CHARS = 200_000 # 粘贴文本上限(审稿意见通常 <20K,保守给大)
# ---------- 信封 ----------
class RebuttalResponse(BaseModel):
ok: bool
job: dict[str, Any] | None = None
error: dict[str, Any] | None = None
meta: dict[str, Any] | None = None
def _error(code: str, message: str) -> RebuttalResponse:
return RebuttalResponse(ok=False, error={"code": code, "message": message, "degraded": False})
def _ok(job: dict[str, Any], **meta: Any) -> RebuttalResponse:
return RebuttalResponse(ok=True, job=job, meta=meta or None)
# ---------- 请求体 ----------
class JobCreateBody(BaseModel):
text: str = Field(min_length=1)
class PartBody(BaseModel):
type: Literal["new_data", "reanalysis", "location", "clarification", "rebuttal"]
content: str = ""
location_hint: str | None = None
class ResponseBody(BaseModel):
parts: list[PartBody] = Field(default_factory=list)
class StructureBody(BaseModel):
action: Literal["merge", "split"]
# merge: comment_ids(含当前 cid,≥2 条);split: items [{title, body}]
comment_ids: list[str] | None = None
items: list[dict[str, str]] | None = None
class ConfirmBody(BaseModel):
confirmed: bool
# ---------- 内部工具 ----------
async def _messages_llm(messages: list[dict[str, str]], *, temperature: float = 0.3) -> dict[str, Any]:
"""适配 rebuttal_parse/draft 的 messages 形态 → llm.complete_json。"""
system = next((m["content"] for m in messages if m.get("role") == "system"), "")
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
return await llm.complete_json(system, user, temperature=temperature)
def _find_comment(job: dict[str, Any], cid: str) -> tuple[dict, dict] | None:
for r in job["reviewers"]:
for c in r["comments"]:
if c["id"] == cid:
return r, c
return None
async def _build_job(raw_text: str, *, kind: str, filename: str | None) -> RebuttalResponse:
raw_text = (raw_text or "").strip()
if not raw_text:
return _error("VALIDATION", "审稿意见文本为空")
if len(raw_text) > _MAX_RAW_CHARS:
return _error("VALIDATION", f"文本超过 {_MAX_RAW_CHARS} 字符上限")
reviewers = await parse_review_text(raw_text, _messages_llm)
if not reviewers:
return _error("PARSE_FAILED", "未能从文本中解析出任何审稿意见条目")
job = {
"id": rebuttal_store.new_id("rbt_"),
"status": "ready",
"source": {"kind": kind, "filename": filename, "raw_text": raw_text},
"template": "generic",
"reviewers": reviewers,
"letter_md": None,
"validation": None,
"human_ack": False,
}
rebuttal_store.insert_job(job)
n_comments = sum(len(r["comments"]) for r in reviewers)
return _ok(job, reviewers=len(reviewers), comments=n_comments)
# ---------- 端点 ----------
@router.post("/jobs", response_model=RebuttalResponse)
async def create_job(body: JobCreateBody) -> RebuttalResponse:
"""文本粘贴主入口。"""
return await _build_job(body.text, kind="text", filename=None)
@router.post("/jobs/pdf", response_model=RebuttalResponse)
async def create_job_pdf(file: UploadFile = File(...)) -> RebuttalResponse:
"""PDF 次要入口(批注版审稿 PDF;经 vstore 8094 解析)。"""
settings = get_settings()
filename = file.filename or "review.pdf"
if not filename.lower().endswith(".pdf"):
return _error("VALIDATION", "仅支持 PDF 文件(推荐直接粘贴文本)")
data = await file.read()
if not data:
return _error("VALIDATION", "上传文件为空")
if len(data) > settings.max_pdf_bytes:
return _error("VALIDATION", f"PDF 超过 {settings.max_pdf_bytes // (1024 * 1024)}MB 上限")
try:
text = await paper_parse.parse_pdf_bytes(data, filename=filename)
except Exception as exc: # noqa: BLE001
return _error("UPSTREAM_DOWN", f"PDF 解析失败:{exc}")
return await _build_job(text, kind="pdf", filename=filename)
@router.get("/jobs/{job_id}", response_model=RebuttalResponse)
async def get_job(job_id: str) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
return _ok(job)
@router.patch("/jobs/{job_id}/comments/{cid}/response", response_model=RebuttalResponse)
async def patch_response(job_id: str, cid: str, body: ResponseBody) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
found = _find_comment(job, cid)
if found is None:
return _error("NOT_FOUND", f"comment {cid} 不存在")
_, comment = found
for p in body.parts:
if p.type not in PART_TYPES:
return _error("VALIDATION", f"未知 part.type: {p.type}")
comment["response"] = {"parts": [p.model_dump() for p in body.parts]}
# 改回复 → 旧 draft/validation 作废(保守:不保留过期草稿)
comment["draft"] = None
comment["validation"] = None
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
return _ok(job)
@router.post("/jobs/{job_id}/comments/{cid}/draft", response_model=RebuttalResponse)
async def post_draft(job_id: str, cid: str) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
found = _find_comment(job, cid)
if found is None:
return _error("NOT_FOUND", f"comment {cid} 不存在")
_, comment = found
parts = (comment.get("response") or {}).get("parts") or []
if not parts or not any(str(p.get("content") or "").strip() for p in parts):
# 决策 4:生成前必须填 user_response;LLM 调用次数=0
return _error("MISSING_RESPONSE", "请先填写回复要点(user_response)再生成草稿")
validation = validate_parts(parts)
comment["validation"] = validation
if not validation["passed"]:
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
return RebuttalResponse(
ok=False, job=job,
error={"code": "VALIDATION_FAILED", "message": "反幻觉校验未通过",
"degraded": False},
meta={"validation": validation},
)
comment["draft"] = await generate_draft(_messages_llm, comment)
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
return _ok(job, validation=validation)
@router.patch("/jobs/{job_id}/comments/{cid}/structure", response_model=RebuttalResponse)
async def patch_structure(job_id: str, cid: str, body: StructureBody) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
found = _find_comment(job, cid)
if found is None:
return _error("NOT_FOUND", f"comment {cid} 不存在")
reviewer, comment = found
comments = reviewer["comments"]
idx = comments.index(comment)
if body.action == "merge":
ids = body.comment_ids or []
if cid not in ids:
ids = [cid] + ids
if len(ids) < 2:
return _error("VALIDATION", "merge 至少需要 2 条 comment")
targets = [c for c in comments if c["id"] in ids]
if len(targets) < 2:
return _error("VALIDATION", "merge 目标中有效 comment 不足 2 条")
merged_body = "\n\n".join(c["body"] for c in targets)
comment["body"] = merged_body
comment["title"] = f"{comment['title']} (+{len(targets) - 1} merged)"
reviewer["comments"] = [
c for c in comments
if c is comment or c["id"] not in ids
]
else: # split
items = body.items or []
items = [i for i in items if str(i.get("body") or "").strip()]
if len(items) < 2:
return _error("VALIDATION", "split 至少需要 2 个非空子项")
group_id = comment.get("group_id") or f"g_{cid}"
new_comments = []
for n, it in enumerate(items, start=1):
c = dict(comment)
c["id"] = rebuttal_store.new_id(f"{cid}s{n}")
c["group_id"] = group_id
c["title"] = str(it.get("title") or f"{comment['title']} ({n})")
c["body"] = str(it.get("body") or "").strip()
c["sub_items"] = []
c["response"] = {"parts": []}
c["draft"] = None
c["validation"] = None
c["human_confirmed"] = False
new_comments.append(c)
comments[idx:idx + 1] = new_comments
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
return _ok(job)
@router.patch("/jobs/{job_id}/comments/{cid}/confirm", response_model=RebuttalResponse)
async def patch_confirm(job_id: str, cid: str, body: ConfirmBody) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
found = _find_comment(job, cid)
if found is None:
return _error("NOT_FOUND", f"comment {cid} 不存在")
_, comment = found
comment["human_confirmed"] = bool(body.confirmed)
# human_ack 由 store 重算(决策 4:全勾才置位,不信前端)
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
return _ok(job, human_ack=job["human_ack"])
@router.post("/jobs/{job_id}/letter", response_model=RebuttalResponse)
async def post_letter(job_id: str) -> RebuttalResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
job = rebuttal_store.update_job(job_id, letter_md=assemble_letter(job)) or job
return _ok(job)
@router.get("/jobs/{job_id}/letter.md")
async def download_letter(job_id: str) -> PlainTextResponse:
job = rebuttal_store.get_job(job_id)
if job is None:
return PlainTextResponse(
"job not found", status_code=404, media_type="text/plain; charset=utf-8")
if not job.get("human_ack"):
# 信封口径与下载端点适配:GET 下载返回纯文本,业务拒绝用 400 + 明文
return PlainTextResponse(
"HUMAN_ACK_REQUIRED: 请逐条勾选确认后再下载 letter.md",
status_code=400, media_type="text/plain; charset=utf-8")
md = job.get("letter_md") or assemble_letter(job)
return PlainTextResponse(md, media_type="text/markdown; charset=utf-8")
@router.delete("/jobs/{job_id}", response_model=RebuttalResponse)
async def delete_job(job_id: str) -> RebuttalResponse:
if not rebuttal_store.delete_job(job_id):
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
return RebuttalResponse(ok=True, job=None, meta={"deleted": job_id})
-165
View File
@@ -1,165 +0,0 @@
"""routers/reformat.py —— 论文自动排版 P1(.tex 源码级重排版,不编译)。
端点:
POST /api/review/reformat multipart 上传 .tex + target_journal →
修改后源码 + 逐行 diff 报告(job 落盘)
GET /api/review/reformat/{id} 取完整报告(重启后仍在)
GET /api/review/reformat/{id}/file 下载修改后的 .tex
信封与其余 review 端点一致:业务错误 = HTTP 200 + {ok:false,error:{code,message,degraded:false}},
前端经 review.js unwrap 归一为 ApiError。本次改动无上游依赖(零 LLM/零 vstore),
degraded 恒 false。
"""
from __future__ import annotations
import hashlib
import json
import logging
import re
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from fastapi import APIRouter, File, Form, UploadFile
from fastapi.responses import FileResponse, JSONResponse
from pydantic import BaseModel
from ..config import get_settings
from ..services.reformat import ReformatError, reformat_tex, resolve_rules
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/review", tags=["review-reformat"])
_JOB_ID_RE = re.compile(r"^[0-9a-f]{12}$")
_ENGINE = "reformat-p1/1.0"
class ReformatResponse(BaseModel):
ok: bool
job_id: str | None = None
modified_tex: str | None = None
changes: list[dict[str, Any]] | None = None
warnings: list[dict[str, Any]] | None = None
meta: dict[str, Any] | None = None
error: dict[str, Any] | None = None
def _error(code: str, message: str) -> ReformatResponse:
return ReformatResponse(
ok=False,
error={"code": code, "message": message, "degraded": False},
)
def _jobs_dir() -> Path:
d = get_settings().reformat_dir_abs
d.mkdir(parents=True, exist_ok=True)
return d
@router.post("/reformat", response_model=ReformatResponse)
async def reformat(
file: UploadFile = File(..., description="LaTeX 源码(.tex 单文件)"),
target_journal: str = Form("", description="目标期刊 id/名称,或 default=通用规则"),
) -> ReformatResponse:
settings = get_settings()
# ---- 入参校验(拍板 5:只收 .tex,docx/pdf 走 format-check 审核不重排)----
filename = file.filename or "paper.tex"
if not filename.lower().endswith(".tex"):
return _error("VALIDATION", "仅支持 .tex 文件(docx/pdf 请走格式审核)")
target = (target_journal or "").strip()
if not target:
return _error("VALIDATION", "必须指定目标期刊(或选择「通用规则」)")
content = await file.read()
if not content:
return _error("VALIDATION", "上传文件为空")
if len(content) > settings.max_tex_bytes:
return _error("VALIDATION", "源码超过 2MB 上限")
try:
source = content.decode("utf-8")
except UnicodeDecodeError:
return _error("VALIDATION", "源码必须为 UTF-8 编码")
# ---- 规则解析(journals.json / format_cache / journal_directory 只读回链)----
resolved = await resolve_rules(target)
# ---- 变换(纯规则引擎;结构畸形 → PARSER_FAILED)----
try:
result = reformat_tex(source, resolved["rules"])
except ReformatError as exc:
logger.warning("reformat: 结构守卫拒收 %s: %s", filename, exc)
return _error(exc.code, str(exc))
# ---- 落盘(.tex + .json 报告;重启后 GET 仍可用)----
job_id = uuid.uuid4().hex[:12]
base = _jobs_dir()
(base / f"{job_id}.tex").write_text(result["modified_tex"], encoding="utf-8")
meta: dict[str, Any] = {
"journal_id": resolved["journal_id"],
"journal_name": resolved["journal_name"],
"rules": resolved["rules"],
"rules_source": resolved["rules_source"],
"rules_note": resolved["rules_note"],
"filename": filename,
"original_sha256": hashlib.sha256(content).hexdigest(),
"engine": _ENGINE,
"generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
}
payload = {
"ok": True,
"job_id": job_id,
"modified_tex": result["modified_tex"],
"changes": result["changes"],
"warnings": result["warnings"],
"meta": meta,
}
(base / f"{job_id}.json").write_text(
json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
)
logger.info(
"reformat: %s → job %s(%d 处修改, %d 条提示, 规则 %s)",
filename, job_id,
sum(1 for c in result["changes"] if c["action"] == "modified"),
sum(1 for c in result["changes"] if c["action"] == "warning"),
resolved["rules_source"],
)
return ReformatResponse(**payload)
def _load_job(job_id: str) -> dict[str, Any] | None:
if not _JOB_ID_RE.fullmatch(job_id or ""):
return None
path = _jobs_dir() / f"{job_id}.json"
try:
payload = json.loads(path.read_text("utf-8"))
except (OSError, ValueError):
return None
return payload if isinstance(payload, dict) and payload.get("ok") else None
_NOT_FOUND = {"ok": False, "error": {"code": "NOT_FOUND",
"message": "任务不存在或已过期,请重新上传源码",
"degraded": False}}
@router.get("/reformat/{job_id}", response_model=ReformatResponse)
async def get_reformat_job(job_id: str) -> ReformatResponse:
payload = _load_job(job_id)
if payload is None:
return _error("NOT_FOUND", "任务不存在或已过期,请重新上传源码")
return ReformatResponse(**payload)
@router.get("/reformat/{job_id}/file", response_model=None)
async def download_reformat_file(job_id: str) -> FileResponse | JSONResponse:
"""下载修改后的 .tex;缺件/非法 id → 200 信封(前端单一错误口径)。"""
valid = bool(_JOB_ID_RE.fullmatch(job_id or ""))
tex = _jobs_dir() / f"{job_id}.tex" if valid else None
if tex is None or not tex.is_file():
return JSONResponse(status_code=200, content=_NOT_FOUND)
return FileResponse(tex, media_type="application/x-tex", filename=f"{job_id}.tex")
-1
View File
@@ -1 +0,0 @@
"""review 服务服务层包。"""
-788
View File
@@ -1,788 +0,0 @@
"""services/format_check.py —— 版式与规范规则引擎(⑤ 论文排版与规范)。
纯规则、无 LLM、无 vstore、同步执行(通常 <1s):
- PDF → pymupdf 本地解档(AGENTS.md 约束 6 例外:vstore 只回文本,字体/边距/
行距/页数等版式元数据拿不到;正文提取仍走 vstore,本服务不碰正文语义);
- Word → python-docx(页边距/行距精确,页数取 docProps/app.xml 的 <Pages>);
- 规则集 → data/journals/journals.json:
顶层 format_rules_default(通用默认值,rules_source="default")
每条目可选 format_rules(按期刊覆盖,必须带 rules_source: placeholder|official|default)
⚠ 诚实原则:不知道期刊真实投稿指南 → 该条只写 {"rules_source": "placeholder"},
所有具体数字回退 default 通用值,绝不编造期刊专属数字。
输出:{checks[{name,key,status,detail,suggestion}], summary{pass,warn,fail}, meta{...}}
status ∈ pass|warn|fail。参考文献样式宽容化:仅"明确不符"(如 [J] 标引但规则要求 APA)
才 fail,"疑似但不确定/检测不到"一律 warn,避免误伤。
"""
from __future__ import annotations
import io
import json
import re
import statistics
import zipfile
from pathlib import Path
from typing import Any
import fitz # pymupdf(venv 已装 1.28.x)
from docx import Document
from docx.shared import Length
# review_service/app/services/ → parents[2] = review_service
_JOURNALS_PATH = Path(__file__).resolve().parents[2] / "data" / "journals" / "journals.json"
_MAX_SCAN_PAGES = 80 # 版式扫描页数上限(正文/字体/边距;稿件通常远小于此)
_MAX_REFS_SCAN = 40 # 从文末向前找参考文献标题的页数窗口
_MAX_PARAS = 8000 # docx 段落扫描上限
_PT_TO_MM = 25.4 / 72.0
_EMU_TO_MM = 1.0 / 36000.0
_HEADING_RE = re.compile(
r"^\s*(references|bibliography|literature cited|works cited|参考文献|文献)\s*[::]?\s*$",
re.IGNORECASE,
)
_FIG_RE = re.compile(r"(?im)^\s*(figure|fig\.?|图)\s*\d+")
_TAB_RE = re.compile(r"(?im)^\s*(table|表)\s*\d+")
_GB_TAG_RE = re.compile(r"\[[A-Z]{1,3}\d*\]") # [J] [M] [D] [C] …(GB/T 7714 标引)
_PAREN_YEAR_RE = re.compile(r"\((?:19|20)\d{2}\)") # (2024) 作者-年份制
_BRACKET_NUM_RE = re.compile(r"^\[\d+\]") # [1] 编号制
_DOI_RE = re.compile(r"doi:?\.?org|doi:\s*1\.", re.IGNORECASE)
_STYLE_LABEL = {
"gb7714": "GB/T 7714",
"author_year": "作者-年份制(类 APA)",
"numbered": "编号制(顺序编码)",
"unknown": "无法确定",
}
class FormatCheckError(Exception):
"""解档/规则加载失败(路由器转 PARSE_FAILED)。"""
# ---------------------------------------------------------------- 规则集加载
def load_ruleset(target_journal: str) -> tuple[dict[str, Any], dict[str, Any]]:
"""返回 (rules, ruleset_meta)。target_journal 传 id 或 name 均可;未知 → default。"""
try:
data = json.loads(_JOURNALS_PATH.read_text("utf-8"))
except Exception as exc: # noqa: BLE001
raise FormatCheckError(f"规则集加载失败:{exc}") from exc
default = data.get("format_rules_default") or {}
jid = (target_journal or "").strip()
entry = None
if jid:
for j in data.get("journals") or []:
if j.get("id") == jid or j.get("name") == jid:
entry = j
break
if entry is None:
rules = dict(default)
meta = {
"journal_id": None,
"journal_name": None,
"ruleset_source": "default",
"rules_source": str(default.get("rules_source", "default")),
}
else:
journal_rules = entry.get("format_rules") or {}
rules = {**default, **journal_rules}
meta = {
"journal_id": entry.get("id"),
"journal_name": entry.get("name"),
"ruleset_source": "journal",
# 占位规则(无官方核验)→ 前端黄字提示
"rules_source": str(journal_rules.get("rules_source", "placeholder")),
}
rules.setdefault("rules_source", "default")
return rules, meta
# ---------------------------------------------------------------- PDF 解档(pymupdf)
def _norm_font(name: str) -> str:
n = (name or "").split("+")[-1].strip()
return n or "unknown"
def extract_profile_pdf(data: bytes) -> dict[str, Any]:
try:
doc = fitz.open(stream=data, filetype="pdf")
except Exception as exc: # noqa: BLE001
raise FormatCheckError(f"PDF 解档失败:{exc}") from exc
if getattr(doc, "needs_pass", False):
doc.close()
raise FormatCheckError("PDF 已加密,无法解析")
pages_total = doc.page_count
scan = min(pages_total, _MAX_SCAN_PAGES)
margin_rows: list[tuple[float, float, float, float]] = [] # t r b l (mm)
font_w: dict[str, int] = {}
size_w: dict[float, int] = {}
ratios: list[float] = []
img_ids: set[int] = set()
fig_caps = tab_caps = 0
for pno in range(scan):
page = doc[pno]
rect = page.rect
blocks = page.get_text("blocks")
content = [b for b in blocks if b[4].strip()]
if content:
left = min(b[0] for b in content)
top = min(b[1] for b in content)
right = rect.width - max(b[2] for b in content)
bottom = rect.height - max(b[3] for b in content)
margin_rows.append(
(
max(0.0, top) * _PT_TO_MM,
max(0.0, right) * _PT_TO_MM,
max(0.0, bottom) * _PT_TO_MM,
max(0.0, left) * _PT_TO_MM,
)
)
for img in page.get_images(full=True):
img_ids.add(img[0])
text = page.get_text()
fig_caps += len(_FIG_RE.findall(text))
tab_caps += len(_TAB_RE.findall(text))
d = page.get_text("dict")
for block in d.get("blocks", []):
if block.get("type") != 0:
continue
line_meta: list[tuple[float, float]] = [] # (基线 y, 字号)
for line in block.get("lines", []):
spans = [s for s in line.get("spans", []) if s.get("text", "").strip()]
if not spans:
continue
row_size = max(s.get("size", 0.0) for s in spans)
if row_size <= 0:
continue
for s in spans:
size = round(float(s.get("size", 0.0)), 1)
chars = len(s.get("text", ""))
if size > 0:
norm = _norm_font(s.get("font", ""))
font_w[norm] = font_w.get(norm, 0) + chars
size_w[size] = size_w.get(size, 0) + chars
# 行距基准用 span.origin 基线 y,不用 bbox 顶缘:bbox 顶随字形高低
# 漂移,表格/图注等同基线并列文本会产出假性小 delta,混入中位数会把
# 行距拉成 0.1x 的不可能值。
origin = spans[0].get("origin")
line_meta.append((origin[1] if origin else line["bbox"][1], row_size))
for i in range(1, len(line_meta)):
delta = line_meta[i][0] - line_meta[i - 1][0]
size = line_meta[i][1]
# 行距倍数 = 相邻行基线 y 差 / 当前行字号;
# 过滤物理不可能值:<0.5(行重叠/同基线并列表格)、>4.0(异常结构)
ratio = delta / size
if 0.5 <= ratio <= 4.0:
ratios.append(ratio)
refs_text, refs_found = _find_refs_pdf(doc, pages_total)
doc.close()
if margin_rows:
med = [statistics.median(col) for col in zip(*margin_rows)]
margins = {"top": med[0], "right": med[1], "bottom": med[2], "left": med[3]}
else:
margins = None
return {
"source_kind": "pdf",
"pages": pages_total,
"margins_mm": margins,
"font_weights": font_w,
"size_weights": size_w,
"line_spacing_median": statistics.median(ratios) if ratios else None,
"figures": max(len(img_ids), fig_caps),
"tables": tab_caps,
"refs_text": refs_text,
"refs_found": refs_found,
}
def _find_refs_pdf(doc: "fitz.Document", pages_total: int) -> tuple[str | None, bool]:
"""从文末向前找参考文献标题行;找到 → 标题行至文末文本;找不到 → (None, False)。"""
buf: list[str] = []
start = max(0, pages_total - _MAX_REFS_SCAN)
for pno in range(pages_total - 1, start - 1, -1):
page_text = doc[pno].get_text()
lines = page_text.splitlines()
for idx in range(len(lines) - 1, -1, -1):
if _HEADING_RE.match(lines[idx].strip()):
tail = "\n".join(lines[idx + 1:])
joined = "\n".join([tail] + buf).strip()
return (joined if len(joined) >= 50 else None), bool(joined)
buf.insert(0, page_text)
return None, False
# ---------------------------------------------------------------- Word 解档(python-docx)
def _docx_app_pages(data: bytes) -> int | None:
try:
with zipfile.ZipFile(io.BytesIO(data)) as zf:
xml = zf.read("docProps/app.xml").decode("utf-8", "ignore")
m = re.search(r"<Pages>(\d+)</Pages>", xml)
return int(m.group(1)) if m else None
except Exception: # noqa: BLE001
return None
def extract_profile_docx(data: bytes) -> dict[str, Any]:
try:
doc = Document(io.BytesIO(data))
except Exception as exc: # noqa: BLE001
raise FormatCheckError(f"Word 文件解析失败:{exc}") from exc
side_lists: dict[str, list[float]] = {"top": [], "right": [], "bottom": [], "left": []}
for sec in doc.sections:
for k, v in (
("top", sec.top_margin),
("right", sec.right_margin),
("bottom", sec.bottom_margin),
("left", sec.left_margin),
):
if v is not None:
side_lists[k].append(float(v) * _EMU_TO_MM)
margins = None
if all(side_lists.values()):
margins = {k: statistics.median(v) for k, v in side_lists.items()}
try:
normal = doc.styles["Normal"].font
default_name = normal.name
default_size = float(normal.size.pt) if normal.size is not None else None
except Exception: # noqa: BLE001
default_name, default_size = None, None
font_w: dict[str, int] = {}
size_w: dict[float, int] = {}
ratios: list[float] = []
fig_caps = tab_caps = 0
paras = doc.paragraphs
scan_count = min(len(paras), _MAX_PARAS)
for i in range(scan_count):
para = paras[i]
text = para.text
if not text.strip():
continue
ls = para.paragraph_format.line_spacing
if isinstance(ls, Length):
# 精确行距(exactly)以 EMU 存储,Length 是 int 子类——直接当倍数会得出
# 十万级的假值;用正文默认字号换算成倍数
if default_size:
r = float(ls.pt) / default_size
if 0.5 <= r <= 4.0:
ratios.append(r)
elif isinstance(ls, (int, float)) and not isinstance(ls, bool) and ls > 0:
r = float(ls) # multiple 行距本身就是倍数
if 0.5 <= r <= 4.0:
ratios.append(r)
if _FIG_RE.match(text):
fig_caps += 1
elif _TAB_RE.match(text):
tab_caps += 1
for run in para.runs:
if not run.text:
continue
name = run.font.name or default_name or "unknown"
pt = run.font.size.pt if run.font.size is not None else default_size
chars = len(run.text)
font_w[str(name)] = font_w.get(str(name), 0) + chars
if pt:
size_w[round(float(pt), 1)] = size_w.get(round(float(pt), 1), 0) + chars
# 参考文献节:标题段落之后全部文本
refs_text, refs_found = None, False
for i, para in enumerate(paras):
if _HEADING_RE.match((para.text or "").strip()):
tail = "\n".join(p.text for p in paras[i + 1:] if (p.text or "").strip())
if len(tail) >= 50:
refs_text, refs_found = tail, True
break
return {
"source_kind": "docx",
"pages": _docx_app_pages(data), # Word 未记录 → None(检查项降级为 warn)
"margins_mm": margins,
"font_weights": font_w,
"size_weights": size_w,
"line_spacing_median": statistics.median(ratios) if ratios else None,
"figures": max(len(doc.inline_shapes), fig_caps),
"tables": max(len(doc.tables), tab_caps),
"refs_text": refs_text,
"refs_found": refs_found,
}
# ---------------------------------------------------------------- LaTeX 解档(正则+状态机,零新依赖)
#
# 决策前提:只做"解析 + 检查 + 建议代码片段",不自动改写 LaTeX(用户可能有自定义宏,
# 自动改写会引错;LaTeX 应由用户自己编辑)。不引入 pylatexenc/texsoup。
# 会相互覆盖的字体类宏包:≥2 个同时引入 → warn(times + mathptmx 类冲突)
_TEX_FONT_PKGS = {"times", "mathptmx", "newtx", "newpx", "txfonts", "pxfonts", "mathpazo"}
def _strip_tex_comments(src: str) -> str:
r"""去行内 % 注释(r-string,避免 \% 被解释为转义序列)。"""
return re.sub(r"(?<!\\)%.*", "", src)
def _parse_geometry(src: str) -> dict[str, float] | None:
"""\\usepackage[...]{geometry} 或 \\geometry{...} → {边: mm}(可缺边,检查端容缺)。"""
m = re.search(r"\\usepackage\s*\[([^\]]*)\]\s*\{\s*geometry\s*\}", src)
if not m:
m = re.search(r"\\geometry\s*\{([^}]*)\}", src)
if not m:
return None
opts = m.group(1)
def to_mm(val: str) -> float | None:
mm = re.match(r"\s*([\d.]+)\s*(mm|cm|in|pt)?\s*$", val)
if not mm:
return None
x = float(mm.group(1))
unit = mm.group(2) or "pt" # geometry 裸数字按 TeX 惯例为 pt
return {"mm": x, "cm": x * 10, "in": x * 25.4, "pt": x * 25.4 / 72.0}[unit]
kv = {}
for pair in opts.split(","):
if "=" in pair:
k, v = pair.split("=", 1)
kv[k.strip().lower()] = v.strip()
out: dict[str, float] = {}
if "margin" in kv:
v = to_mm(kv["margin"])
if v is not None:
return {"top": v, "right": v, "bottom": v, "left": v}
for k in ("top", "right", "bottom", "left"):
if k in kv:
v = to_mm(kv[k])
if v is not None:
out[k] = v
if "inner" in kv:
v = to_mm(kv["inner"])
if v is not None:
out.setdefault("left", v)
if "outer" in kv:
v = to_mm(kv["outer"])
if v is not None:
out.setdefault("right", v)
return out or None
def extract_tex_profile(data: bytes) -> dict[str, Any]:
"""正则 + 轻量状态机解析 .tex 源文件 → 与 DocProfile 同构(+tex 扩展字段)。"""
try:
text = data.decode("utf-8")
except UnicodeDecodeError:
text = data.decode("latin-1", errors="replace")
src = _strip_tex_comments(text)
# \documentclass[10pt,a4paper]{article}
m_cls = re.search(r"\\documentclass\s*(?:\[[^\]]*\])?\s*\{\s*([\w*]+)\s*\}", src)
cls = m_cls.group(1) if m_cls else None
m_opt = re.search(r"\\documentclass\s*\[([^\]]*)\]", src)
cls_opts = m_opt.group(1) if m_opt else ""
m_pt = re.search(r"(\d+)\s*pt", cls_opts)
# 标准类未写选项时 LaTeX 默认 10pt;无 documentclass → None(检查项降级 warn)
fontsize = int(m_pt.group(1)) if m_pt else (10 if cls else None)
paper = "a4paper" if "a4paper" in cls_opts else ("letterpaper" if "letterpaper" in cls_opts else None)
# \usepackage 列表(支持 {a,b} 逗号多包)
packages: list[str] = []
for m in re.finditer(r"\\usepackage\s*(?:\[[^\]]*\])?\s*\{\s*([^}]+?)\s*\}", src):
packages += [p.strip().lower() for p in m.group(1).split(",") if p.strip()]
# 行距:\linespread{...} 或 \setstretch{...}(值直接按"行距倍数"口径评估——
# 按步骤4 规格:低于 min_line_spacing 即 fail;LaTeX 实际倍数还含 baselinestretch
# 的字号比,本引擎取显式设置值做保守判定)
line_spread: float | None = None
m = re.search(r"\\linespread\s*\{\s*([\d.]+)\s*\}", src)
if not m:
m = re.search(r"\\setstretch\s*\{\s*([\d.]+)\s*\}", src)
if m:
line_spread = float(m.group(1))
# 页边距(geometry)
margins = _parse_geometry(src)
# 计数
figures = len(re.findall(r"\\begin\s*\{\s*figure\*?\s*\}", src))
tables = len(re.findall(r"\\begin\s*\{\s*table\*?\s*\}", src))
cites = len(re.findall(r"\\cite\w*\s*\{", src))
# 参考文献:\bibliography{...}(BibTeX)/ \printbibliography(biblatex)/ thebibliography
has_bibtex = bool(re.search(r"\\bibliography\s*\{", src))
has_printbib = bool(re.search(r"\\printbibliography", src))
has_thebib = bool(re.search(r"\\begin\s*\{\s*thebibliography\s*\}", src))
has_bibstyle = bool(re.search(r"\\bibliographystyle\s*\{", src)) # BibTeX 必需;biblatex 不需要
has_bib = has_bibtex or has_printbib or has_thebib
refs_text: str | None = None
m_bib = re.search(r"\\begin\s*\{\s*thebibliography\s*\}[\s\S]*?\\end\s*\{\s*thebibliography\s*\}", src)
if m_bib:
refs_text = m_bib.group(0)
# 宏表(\newcommand{\xxx} / \renewcommand)——用于解析准确性(识别自定义宏)
macros = re.findall(r"\\(?:new|re)command\s*\{\s*\\(\w+)\s*\}", src)
# 匿名评审命令探测(规则集 anonymous:true 时检查)
m_anon = re.search(r"\\(?:anonymi[sz]e|anon\b|hideauthor|ifdoubleblind)", src, re.IGNORECASE)
return {
"source_kind": "tex",
"pages": None, # LaTeX 源文件页数需编译后才有 → 检查项降级 warn(宽容化)
"margins_mm": margins,
"font_weights": {},
"size_weights": {float(fontsize): 100} if fontsize else {},
"line_spacing_median": line_spread,
"figures": figures,
"tables": tables,
"refs_text": refs_text,
"refs_found": has_bib,
"tex": {
"class": cls,
"fontsize_pt": fontsize,
"paper": paper,
"packages": packages,
"cites": cites,
"macros": macros,
"anon_cmd": m_anon.group(0) if m_anon else None,
"bib_bibtex": has_bibtex,
"bib_style": has_bibstyle,
},
}
# ---------------------------------------------------------------- 参考文献样式检测
def detect_refs_style(text: str | None) -> tuple[str, str]:
"""宽容化检测:返回 (detected, sample_line)。含糊 → unknown(后续只 warn)。"""
if not text:
return "unknown", ""
lines = [l.strip() for l in text.splitlines() if l.strip()][:40]
if not lines:
return "unknown", ""
n = len(lines)
gb = sum(1 for l in lines if _GB_TAG_RE.search(l))
year = sum(1 for l in lines if _PAREN_YEAR_RE.search(l))
num = sum(1 for l in lines if _BRACKET_NUM_RE.match(l))
doi = sum(1 for l in lines if _DOI_RE.search(l))
sample = lines[0][:80]
if gb / n >= 0.4:
return "gb7714", sample
if num / n >= 0.4 and year / n < 0.3:
return "numbered", sample
if year / n >= 0.3 or doi / n >= 0.3:
return "author_year", sample
return "unknown", sample
# ---------------------------------------------------------------- 检查项构建
def _chk(key: str, name: str, status: str, detail: str, suggestion: str = "") -> dict[str, str]:
return {"key": key, "name": name, "status": status, "detail": detail, "suggestion": suggestion}
def _check_pages(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
limit = int(rules.get("max_pages", 10))
pages = profile.get("pages")
kind = profile.get("source_kind") or "doc"
if pages is None:
# 宽容化:Word/LaTeX 页数缺失绝不 fail
if kind == "tex":
return _chk(
"pages", "页数", "warn",
f"LaTeX 源无法直接统计页数(规则上限 {limit} 页),需编译后核对",
"编译后确认(如 xelatex main.tex)总页数 ≤ 上限",
)
return _chk(
"pages", "页数", "warn",
"Word 未记录页数,请自行核对",
f"在 Word 中按 Ctrl+End 查看总页数,或导出 PDF 后重查(规则上限 {limit} 页)",
)
if pages <= limit:
return _chk("pages", "页数", "pass", f"共 {pages} 页,规则上限 {limit} 页")
return _chk(
"pages", "页数", "fail", f"共 {pages} 页,超出规则上限 {limit} 页",
f"压缩至 ≤{limit} 页,或将方法/图表移入补充材料",
)
def _check_fonts(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
min_font = float(rules.get("min_font_pt", 9))
# LaTeX:字号来自 \documentclass 选项(10/11/12pt,粒度粗)
if profile.get("source_kind") == "tex":
tex = profile.get("tex") or {}
cls = tex.get("class") or "documentclass"
fs = tex.get("fontsize_pt")
pkgs = tex.get("packages") or []
font_pkgs = sorted(p for p in pkgs if p in _TEX_FONT_PKGS or p in ("helvet", "courier", "charter", "fourier"))
detail = (
f"\\documentclass 选项字号 {fs}pt(类 {cls});"
f"字体宏包:{'、'.join(font_pkgs) if font_pkgs else '未引入'}"
+ (f";纸张 {tex['paper']}" if tex.get("paper") else "")
)
if fs is None:
return _chk("fonts", "字体字号", "warn", detail + "\n建议:\n \\documentclass[11pt]{article}", "补充 \\documentclass 声明")
if fs < min_font:
# LaTeX 粒度粗 → 按步骤4 规格降为 warn(非 fail)
return _chk(
"fonts", "字体字号", "warn", detail + f"\n建议:\n \\documentclass[11pt]{{{cls}}}",
f"正文字号低于下限 {min_font:.0f}pt,请改用 11pt/12pt 选项",
)
return _chk("fonts", "字体字号", "pass", detail)
size_w: dict[float, int] = profile.get("size_weights") or {}
font_w: dict[str, int] = profile.get("font_weights") or {}
if not size_w:
return _chk("fonts", "字体字号", "warn", "未能提取字体信息", "确认文档含可选中文本(扫描件无法检查字体)")
min_font = float(rules.get("min_font_pt", 9))
max_fonts = int(rules.get("max_fonts", 6))
total = sum(size_w.values())
body_pt = max(size_w, key=size_w.get)
body_ratio = int(size_w[body_pt] / total * 100)
distinct = len(font_w)
top_fonts = sorted(font_w.items(), key=lambda kv: -kv[1])[:3]
top_text = "、".join(f"{k}({v}字)" for k, v in top_fonts)
detail = f"主要字号 {body_pt}pt(约 {body_ratio}%),共 {distinct} 种字体;高频字体:{top_text}"
if body_pt < min_font:
return _chk(
"fonts", "字体字号", "fail", detail,
f"将正文字号提高至 ≥{min_font}pt(当前主字号 {body_pt}pt)",
)
if distinct > max_fonts:
return _chk(
"fonts", "字体字号", "warn", detail,
f"字体种类偏多({distinct} > {max_fonts}),建议全文统一为 1–2 种正文字体",
)
return _chk("fonts", "字体字号", "pass", detail)
def _check_margins(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
limit = float(rules.get("min_margin_mm", 15))
margins = profile.get("margins_mm")
# LaTeX:页边距来自 geometry;低于下限按步骤4 规格给 warn(非 fail),附建议片段
if profile.get("source_kind") == "tex":
if not margins:
return _chk(
"margins", "页边距", "warn",
"未通过 geometry 设置页边距(按类默认评估)"
+ f"\n建议:\n \\usepackage[margin=25mm]{{geometry}}",
"在导言区显式声明页边距",
)
lo = min(margins.values())
detail = f"geometry 页边距最小 {lo:.0f}mm(" + "、".join(f"{k}{v:.0f}" for k, v in margins.items()) + f"),规则下限 {limit:.0f}mm"
if lo < limit:
return _chk(
"margins", "页边距", "warn",
detail + f"\n建议:\n \\usepackage[margin={limit:.0f}mm]{{geometry}}",
f"调大页边距至 ≥{limit:.0f}mm",
)
return _chk("margins", "页边距", "pass", detail)
if not margins:
return _chk("margins", "页边距", "warn", "未能提取页边距信息", "检查文档是否含可定位文本")
vals = [margins.get(k) for k in ("top", "right", "bottom", "left")]
vals = [v for v in vals if v is not None]
if not vals:
return _chk("margins", "页边距", "warn", "未能提取页边距信息", "检查文档是否含可定位文本")
t, r = margins.get("top"), margins.get("right")
b, l = margins.get("bottom"), margins.get("left")
lo = min(vals)
sides = "、".join(
f"{name}{v:.0f}" for name, v in (("上", t), ("右", r), ("下", b), ("左", l)) if v is not None
)
detail = f"四边最小 {lo:.0f}mm({sides}),规则下限 {limit:.0f}mm"
if lo >= limit:
return _chk("margins", "页边距", "pass", detail)
if lo >= limit * 0.8:
return _chk("margins", "页边距", "warn", detail, f"将页边距调大至 ≥{limit:.0f}mm")
return _chk("margins", "页边距", "fail", detail, f"页边距过小,请调至 ≥{limit:.0f}mm(版式设置)")
def _check_spacing(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
lo = float(rules.get("min_line_spacing", 1.15))
ratio = profile.get("line_spacing_median")
# LaTeX:\linespread/\setstretch 显式设置值;低于下限按步骤4 规格直接 fail
if profile.get("source_kind") == "tex":
if ratio is None:
return _chk(
"spacing", "行距", "pass",
"未显式设置 \\linespread/\\setstretch,LaTeX 类默认约 1.2 倍(如 10pt→12pt baselineskip)",
)
detail = f"\\linespread/\\setstretch = {ratio:.2f},规则下限 {lo:.2f}"
if ratio < lo:
return _chk(
"spacing", "行距", "fail",
detail + f"\n建议:\n \\linespread{{{lo:.2f}}} % 或 \\setstretch{{{lo:.2f}}}(导言区)",
"在导言区调大行距设置",
)
return _chk("spacing", "行距", "pass", detail)
if ratio is None:
return _chk("spacing", "行距", "warn", "未能提取行距信息", "确认段落使用固定/多倍行距设置")
lo = float(rules.get("min_line_spacing", 1.15))
detail = f"正文中位行距约 {ratio:.2f} 倍,规则下限 {lo:.2f} 倍"
if ratio >= lo:
return _chk("spacing", "行距", "pass", detail)
if ratio >= lo * 0.9:
return _chk("spacing", "行距", "warn", detail, f"段落行距设为 ≥{lo:.2f} 倍(如 1.5 倍行距)")
return _chk("spacing", "行距", "fail", detail, f"行距过密,请设为 ≥{lo:.2f} 倍行距")
def _check_figures(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
figures = int(profile.get("figures") or 0)
tables = int(profile.get("tables") or 0)
min_f = int(rules.get("min_figures", 1))
detail = f"检测到图 {figures} 张、表 {tables} 个(图注/表注正则 + 内嵌对象启发式)"
if figures >= min_f:
return _chk("figures", "图表数量", "pass", detail)
# 启发式检测不可靠 → 低于阈值只 warn 不 fail
return _chk(
"figures", "图表数量", "warn", detail,
f"常规研究论文建议 ≥{min_f} 图;若为综述/短文可忽略(本项为启发式检测)",
)
# 仅"明确不符"才 fail 的 (expected, detected) 组合(宽容化:其余含糊组合一律 warn)
_REF_FAIL_PAIRS = {
("apa", "gb7714"), # 找到 [J] 标引但规则要求 APA
("gb7714", "author_year"), # 无 [J] 标引的作者-年份制,但规则要求 GB/T 7714
("apa", "numbered"), # 编号制引文但规则要求 APA
}
def _check_refs(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
expected = str(rules.get("refs_style", "apa")).lower()
exp_label = _STYLE_LABEL.get(expected, expected)
if not profile.get("refs_found"):
return _chk(
"refs", "参考文献格式", "warn",
"未找到参考文献节(References/参考文献 / \\bibliography / thebibliography)",
"确认存在参考文献声明后重查",
)
if profile.get("source_kind") == "tex":
tex = profile.get("tex") or {}
# BibTeX:\bibliography 存在但缺 \bibliographystyle → warn(样式引擎未知)
if tex.get("bib_bibtex") and not tex.get("bib_style"):
return _chk(
"refs", "参考文献格式", "warn",
f"规则要求 {exp_label};检测到 \\bibliography 但缺少 \\bibliographystyle(BibTeX 无法确定引用样式)"
+ "\n建议:\n \\bibliographystyle{plain} % 按期刊/模板指定实际样式",
"在 \\bibliography 前补 \\bibliographystyle",
)
if not profile.get("refs_text"):
# 条目在 .bbl/编译产物,源文件无文本 → 宽容化只 warn
return _chk(
"refs", "参考文献格式", "warn",
f"规则要求 {exp_label};检测到参考文献声明(条目由编译生成,源文件无条目文本)",
"编译后核对参考文献条目样式",
)
detected, sample = detect_refs_style(profile.get("refs_text"))
det_label = _STYLE_LABEL[detected]
detail = f"规则要求 {exp_label};检测到 {det_label};样例:{sample}"
# 宽容化:仅"明确不符"(_REF_FAIL_PAIRS)才 fail;含糊/不确定一律 warn
if (expected, detected) in _REF_FAIL_PAIRS:
return _chk("refs", "参考文献格式", "fail", detail, f"按 {exp_label} 调整全部文献条目(可用文献管理工具一键转换)")
if detected == "unknown":
return _chk("refs", "参考文献格式", "warn", detail, f"请人工核对是否符合 {exp_label}")
if expected == "gb7714" and detected == "numbered":
return _chk("refs", "参考文献格式", "warn", detail, "GB/T 7714 顺序编码制也用编号,需人工确认条目含 [J] 等类型标引")
if expected == "author-year-journal" and detected in ("gb7714", "numbered"):
return _chk("refs", "参考文献格式", "warn", detail, f"请对照期刊投稿指南核对是否符合 {exp_label}")
return _chk("refs", "参考文献格式", "pass", detail)
def _check_font_packages(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
"""LaTeX 特有:多个相互覆盖的字体宏包(times + mathptmx 类)→ warn。"""
pkgs = set((profile.get("tex") or {}).get("packages") or [])
clash = sorted(pkgs & _TEX_FONT_PKGS)
if len(clash) >= 2:
detail = (
f"检测到多个相互覆盖的字体宏包:{'、'.join(clash)}(后加载者覆盖前者,易产生冲突)"
+ "\n建议:\n %只保留一套字体方案,例如:\n \\usepackage{newtx} % newtxtext + newtxmath 一体"
)
return _chk("fontpkgs", "字体包兼容", "warn", detail, "移除重复/冲突的字体宏包,只保留一套")
return _chk(
"fontpkgs", "字体包兼容", "pass",
"字体宏包:" + ("、".join(clash) if clash else "未引入冲突类字体包"),
)
def _check_anonymous(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
"""LaTeX 特有且规则集 anonymous:true 时才出:源文件需含匿名处理命令。"""
tex = profile.get("tex") or {}
cmd = tex.get("anon_cmd")
if cmd:
return _chk("anonymous", "匿名评审", "pass", f"检测到匿名处理命令 {cmd}")
return _chk(
"anonymous", "匿名评审", "warn",
"规则要求匿名评审,源文件未检测到匿名处理命令"
+ "\n建议:\n %隐去作者/致谢/自引,或定义匿名宏:\n \\newcommand{\\anon}[1]{[ANON]}",
"按期刊要求对作者信息做匿名处理",
)
def build_checks(profile: dict[str, Any], rules: dict[str, Any]) -> list[dict[str, str]]:
checks = [
_check_pages(profile, rules),
_check_fonts(profile, rules),
_check_margins(profile, rules),
_check_spacing(profile, rules),
]
if profile.get("source_kind") == "tex":
checks.append(_check_font_packages(profile, rules))
checks.append(_check_figures(profile, rules))
checks.append(_check_refs(profile, rules))
if profile.get("source_kind") == "tex" and rules.get("anonymous"):
checks.append(_check_anonymous(profile, rules))
return checks
# ---------------------------------------------------------------- 入口
def check_document(data: bytes, kind: str, target_journal: str) -> dict[str, Any]:
"""kind ∈ pdf|docx|tex。返回 {checks, summary, meta}(filename 由路由器补入 meta)。"""
if kind == "pdf":
profile = extract_profile_pdf(data)
elif kind == "docx":
profile = extract_profile_docx(data)
elif kind == "tex":
profile = extract_tex_profile(data)
else:
raise FormatCheckError(f"不支持的文件类型:{kind}")
rules, ruleset_meta = load_ruleset(target_journal)
checks = build_checks(profile, rules)
summary = {"pass": 0, "warn": 0, "fail": 0}
for c in checks:
summary[c["status"]] = summary.get(c["status"], 0) + 1
detected, _ = detect_refs_style(profile.get("refs_text")) if profile.get("refs_found") else ("unknown", "")
meta = {
**ruleset_meta,
"file_type": kind,
"pages": profile.get("pages"),
"refs_style_expected": str(rules.get("refs_style", "apa")).lower(),
"refs_style_detected": detected,
}
if kind == "tex":
tex = profile.get("tex") or {}
meta["tex"] = {
"class": tex.get("class"),
"fontsize_pt": tex.get("fontsize_pt"),
"paper": tex.get("paper"),
"packages": tex.get("packages"),
"cites": tex.get("cites"),
"macros": (tex.get("macros") or [])[:10],
}
return {"checks": checks, "summary": summary, "meta": meta}
@@ -1,232 +0,0 @@
"""services/format_extractor.py —— 投稿指南 → 格式规则 JSON(LLM 抽取,第一批)。
- 调 services/llm.py 的 complete_json(DeepSeek json_object + 修复重试,零新依赖)
- Prompt = 「格式规则抽取器」模板:目标字段与 journals.json format_rules_default
同构(max_pages / min_font_pt / min_margin_mm / min_line_spacing / min_figures /
max_fonts / refs_style / anonymous),强制每字段带 source_quotes + confidence
- 诚实原则:正文没写 → null,绝不编造(与 format_check 诚实原则一致)
- 解析/校验失败 → {"ok": False, "error": ...}
返回:
成功 {"ok": True, "rules": {...}, "source_quotes": {...},
"confidence": {...}, "notes": str, "raw": LLM原始JSON, "error": None}
失败 {"ok": False, "error": "...", 其余键为 None}
source_url 由调用方(scripts/fetch_format_rules.py)在落盘时补入结果文件。
"""
from __future__ import annotations
import logging
import re
from typing import Any
from .llm import LLMError, complete_json
logger = logging.getLogger(__name__)
# 「格式规则抽取器」模板(system prompt)
_SYSTEM = """你是「格式规则抽取器」:从期刊官方投稿指南的 Markdown 正文中抽取格式规则,供投稿排版检查引擎使用。
# 目标字段(rules 对象,严格用以下字段名)
- max_pages:正文页数上限(整数;原文无明确上限 → null)
- min_font_pt:最小字号(数字,单位 pt;原文若用 cm/mm/inch 给出则换算为 pt)
- min_margin_mm:最小页边距(数字,单位 mm;原文若用 inch 给出则 ×25.4 换算)
- min_line_spacing:最小行距倍数(数字;如 "double-spaced" → 2.0,"1.5 spacing" → 1.5)
- min_figures:研究论文最少图数(整数;无明确要求 → null)
- max_fonts:全文最多字体种类(整数;无明确要求 → null)
- refs_style:参考文献样式,枚举 apa | gb7714 | author_year | numbered | vancouver | mla | chicago | other(无法判断 → null)
- anonymous:是否要求匿名/双盲评审(true / false / null)
# 凭证(强制)
- 每个非 null 字段必须给 source_quotes[字段名]:从正文【逐字】摘录的 1-3 句原文,
不改写、不翻译、不丢掉含关键数字的句子。
- source_quotes["_general"]:整体最有凭证价值的 1-3 条原文摘录。
- confidence:{"overall": 0-1, "per_field": {"字段名": 0-1}}。
直接原文命中 ≥0.9;单位换算 0.6-0.9;靠上下文推断 <0.6。
# 诚实原则
- 正文没写到的字段一律 null,绝不用常识或训练记忆补值。
- 单位换算允许做,但该字段 confidence 不得超过 0.9。
- 换算过程、拿不准的点、页面疑似不完整,写进 notes。
# 输出
只输出一个 JSON 对象,不要代码块、不要解释文字:
{"rules": {"max_pages": null, "min_font_pt": null, "min_margin_mm": null,
"min_line_spacing": null, "min_figures": null, "max_fonts": null,
"refs_style": null, "anonymous": null},
"source_quotes": {"字段名": ["原文1", "..."], "_general": ["原文1", "..."]},
"confidence": {"overall": 0.0, "per_field": {"字段名": 0.0}},
"notes": "一句话说明抽取依据与不确定点"}"""
_USER_TMPL = """期刊:{journal_name}
=== 官方投稿指南正文(Markdown,已截断) ===
{markdown}"""
_RULE_INT_KEYS = ("max_pages", "min_figures", "max_fonts")
_REFS_ALLOWED = {"apa", "gb7714", "author_year", "numbered", "vancouver",
"mla", "chicago", "other"}
_REFS_ALIAS = {
"apa-7": "apa", "apa7": "apa", "apa 7th": "apa", "apa_7th": "apa",
"harvard": "author_year", "author-year": "author_year", "authoryear": "author_year",
"gb/t 7714": "gb7714", "gb/t-7714": "gb7714", "gb7714-2015": "gb7714",
"gb/t7714": "gb7714", "chicago author-date": "chicago",
}
# 字符串值带显式单位时的目标单位换算表(LLM 若回传原文单位的兜底;无单位=数值即目标单位)
_UNIT_TO_MM = {"mm": 1.0, "cm": 10.0, "in": 25.4, "inch": 25.4, "inches": 25.4, "pt": 25.4 / 72}
_UNIT_TO_PT = {"pt": 1.0, "mm": 72 / 25.4, "cm": 72 / 2.54, "in": 72.0, "inch": 72.0, "inches": 72.0}
def _to_int(v: Any) -> int | None:
if isinstance(v, bool) or v is None:
return None
if isinstance(v, int):
return v
if isinstance(v, float):
return int(round(v))
if isinstance(v, str):
m = re.search(r"-?\d+", v)
if m:
return int(m.group(0))
return None
def _to_float(v: Any, units: dict[str, float] | None = None) -> float | None:
if isinstance(v, bool) or v is None:
return None
if isinstance(v, (int, float)):
return float(v)
if isinstance(v, str):
m = re.search(r"(-?\d+(?:\.\d+)?)\s*([a-zA-Z]+)?", v)
if not m:
return None
x = float(m.group(1))
unit = (m.group(2) or "").lower()
if units and unit:
factor = units.get(unit)
if factor is None:
return None # 未知单位不猜
x *= factor
return x
return None
def _to_bool(v: Any) -> bool | None:
if isinstance(v, bool):
return v
if isinstance(v, (int, float)) and not isinstance(v, bool):
if v in (0, 1):
return bool(v)
if isinstance(v, str):
low = v.strip().lower()
if low in ("true", "yes", "required"):
return True
if low in ("false", "no", "not required"):
return False
return None
def _norm_refs(v: Any) -> str | None:
if not isinstance(v, str) or not v.strip():
return None
low = v.strip().lower()
low = _REFS_ALIAS.get(low, low)
return low if low in _REFS_ALLOWED else "other"
def _norm_conf(v: Any) -> float | None:
if isinstance(v, bool) or not isinstance(v, (int, float)):
return None
return max(0.0, min(1.0, float(v)))
def _normalize(raw: dict[str, Any]) -> dict[str, Any]:
"""白名单归一:类型错的值置 null(宁缺勿错,raw 全量保留在结果里)。"""
src = raw.get("rules")
src = src if isinstance(src, dict) else {}
rules: dict[str, Any] = {}
for k in _RULE_INT_KEYS:
rules[k] = _to_int(src.get(k))
rules["min_font_pt"] = _to_float(src.get("min_font_pt"), _UNIT_TO_PT)
rules["min_margin_mm"] = _to_float(src.get("min_margin_mm"), _UNIT_TO_MM)
rules["min_line_spacing"] = _to_float(src.get("min_line_spacing"))
rules["refs_style"] = _norm_refs(src.get("refs_style"))
rules["anonymous"] = _to_bool(src.get("anonymous"))
sq_src = raw.get("source_quotes")
sq_src = sq_src if isinstance(sq_src, dict) else {}
quotes: dict[str, list[str]] = {}
for key, val in sq_src.items():
if key != "_general" and key not in rules:
continue
items = val if isinstance(val, list) else [val]
cleaned = [str(x).strip() for x in items if isinstance(x, (str, int, float))
and str(x).strip()]
if cleaned:
quotes[key] = cleaned
conf_src = raw.get("confidence")
conf_src = conf_src if isinstance(conf_src, dict) else {}
pf_src = conf_src.get("per_field")
pf_src = pf_src if isinstance(pf_src, dict) else {}
per_field: dict[str, float] = {}
for k, v in pf_src.items():
c = _norm_conf(v)
if c is not None and (k in rules or k == "_general"):
per_field[k] = c
overall = _norm_conf(conf_src.get("overall"))
if overall is None and per_field:
overall = sum(per_field.values()) / len(per_field)
confidence = {"overall": overall, "per_field": per_field}
notes = raw.get("notes")
return {
"rules": rules,
"source_quotes": quotes,
"confidence": confidence,
"notes": str(notes) if notes else "",
}
def _validation_error(norm: dict[str, Any]) -> str | None:
"""强制凭证:任一非 null 字段缺 source_quotes → 拒收(ok=False)。"""
missing = [k for k, v in norm["rules"].items()
if v is not None and k not in norm["source_quotes"]]
if missing:
return f"LLM 输出缺少非 null 字段的 source_quotes:{', '.join(missing)}"
if not norm["source_quotes"].get("_general"):
return "LLM 输出缺少 source_quotes._general 整体引证"
if any(v is not None for v in norm["rules"].values()):
return None
return "LLM 未抽出任何非 null 字段(rules 全空)"
async def extract_format_rules(markdown: str, journal_name: str) -> dict:
"""markdown(≤30000 字符)+ 期刊名 → 抽取结果(schema 见模块 docstring)。"""
if not markdown or not markdown.strip():
return {"ok": False, "error": "markdown 为空", "rules": None,
"source_quotes": None, "confidence": None, "notes": None, "raw": None}
user = _USER_TMPL.format(journal_name=journal_name, markdown=markdown)
try:
raw = await complete_json(_SYSTEM, user, temperature=0.1, max_tokens=8000)
except LLMError as exc:
return {"ok": False, "error": f"LLM 调用失败:{exc}", "rules": None,
"source_quotes": None, "confidence": None, "notes": None, "raw": None}
if not isinstance(raw, dict):
return {"ok": False, "error": f"LLM 输出非 JSON 对象:{type(raw).__name__}",
"rules": None, "source_quotes": None, "confidence": None,
"notes": None, "raw": None}
norm = _normalize(raw)
err = _validation_error(norm)
if err:
logger.warning("[extract] %s 校验失败:%s", journal_name, err)
return {"ok": False, "error": err, "rules": None,
"source_quotes": norm["source_quotes"], "confidence": norm["confidence"],
"notes": norm["notes"], "raw": raw}
filled = sum(1 for v in norm["rules"].values() if v is not None)
logger.info("[extract] %s → 8 字段中 %d 个非 null", journal_name, filled)
return {"ok": True, "error": None, "rules": norm["rules"],
"source_quotes": norm["source_quotes"],
"confidence": norm["confidence"], "notes": norm["notes"], "raw": raw}
@@ -1,356 +0,0 @@
"""services/format_fetcher.py —— 投稿指南页面抓取(格式规则自动获取·第一批)。
约束(用户拍板 2026-10-10):
- 零新依赖:httpx + beautifulsoup4(venv 已装 4.15.0,用户明确指定使用;
lxml 6.1.2 在 requirements.lock 内,作 bs4 解析器)
- 同域限速 ≥ 2s(robots.txt 与正文请求共用计时器)
- 抓正文前先 GET /robots.txt,命中 Disallow → ok=False 跳过
- UA: "PlantAI-RAG-Bot/1.0 (+contact: admin@example.com)"
- 只提取正文(去导航/页脚/脚本),截断至 30000 字符
返回固定四键:{"ok": bool, "markdown": str, "error": str | None, "fetched_at": str}
"""
from __future__ import annotations
import asyncio
import logging
import re
import time
from datetime import datetime, timezone
from urllib.parse import urlparse
import httpx
from bs4 import BeautifulSoup, NavigableString
logger = logging.getLogger(__name__)
USER_AGENT = "PlantAI-RAG-Bot/1.0 (+contact: admin@example.com)"
TIMEOUT = 30.0
MAX_CHARS = 30_000
MIN_INTERVAL_PER_HOST = 2.0 # 同域请求间隔 ≥ 2s(robots 与正文共用)
_MIN_TEXT_CHARS = 200 # 正文短于此 → 疑似 JS 渲染/反爬页
_HEADERS = {
"User-Agent": USER_AGENT,
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
}
# 同域限速:host → 上次请求完成时刻(monotonic)
_last_hit: dict[str, float] = {}
_throttle_lock = asyncio.Lock()
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat(timespec="seconds")
def _fail(error: str) -> dict:
return {"ok": False, "markdown": "", "error": error, "fetched_at": _now_iso()}
async def _throttle(host: str) -> None:
"""同域串行 + ≥2s 间隔(全局锁:本批抓取量小,简单可靠优先)。"""
async with _throttle_lock:
now = time.monotonic()
wait = _last_hit.get(host, now) + MIN_INTERVAL_PER_HOST - now
if wait > 0:
await asyncio.sleep(wait)
_last_hit[host] = time.monotonic()
# ---------------------------------------------------------------- robots.txt
def _parse_robots(text: str) -> list[tuple[list[str], list[tuple[str, str]]]]:
"""→ [(agents, [(kind, pattern)...]), ...](连续 UA 行同组;规则行后出现 UA=新组)。"""
groups: list[tuple[list[str], list[tuple[str, str]]]] = []
agents: list[str] = []
rules: list[tuple[str, str]] = []
for raw in text.splitlines():
line = raw.split("#", 1)[0].strip()
if not line or ":" not in line:
continue
key, _, value = line.partition(":")
key, value = key.strip().lower(), value.strip()
if key == "user-agent":
if agents and rules: # 上一组已出规则 → 当前 UA 行开新组
groups.append((agents, rules))
agents, rules = [], []
agents.append(value)
elif key in ("allow", "disallow"):
if not agents: # 无前置 UA 的规则按 * 组处理
agents = ["*"]
rules.append((key, value))
if agents:
groups.append((agents, rules))
return groups
def _robots_path_allowed(groups, path: str) -> tuple[bool, str]:
"""最长匹配决胜(同长 Allow 优先;`$` 结尾=精确匹配;空 Disallow 跳过)。"""
def matched_by(pat: str) -> bool:
if pat.endswith("$"):
return path == pat[:-1]
return path.startswith(pat)
token = "plantai-rag-bot"
matched = [rules for agents, rules in groups
if any(token in a.lower() for a in agents)]
if not matched:
matched = [rules for agents, rules in groups
if any(a.lower() == "*" for a in agents)]
if not matched:
return True, "robots.txt 无匹配组(本 bot 不受限)"
best_len, best_allow, best_pat = -1, False, ""
for rules in matched:
for kind, pat in rules:
if not pat: # "Disallow:" 空值 = 放行一切(该组无需处理)
continue
if not matched_by(pat):
continue
length = len(pat)
allow = kind == "allow"
if length > best_len or (length == best_len and allow and not best_allow):
best_len, best_allow, best_pat = length, allow, pat
if best_len < 0:
return True, "robots.txt 允许(无路径匹配)"
if best_allow:
return True, f"robots.txt Allow 命中({best_pat})"
return False, f"robots.txt Disallow 命中({best_pat})"
async def _robots_allows(client: httpx.AsyncClient, url: str) -> tuple[bool, str]:
p = urlparse(url)
robots_url = f"{p.scheme}://{p.netloc}/robots.txt"
await _throttle(p.netloc.lower())
try:
resp = await client.get(robots_url, headers=_HEADERS)
except httpx.HTTPError as exc:
return True, f"robots.txt 获取失败({type(exc).__name__}),按允许处理"
if resp.status_code != 200:
return True, f"robots.txt HTTP {resp.status_code},按允许处理"
groups = _parse_robots(resp.text)
return _robots_path_allowed(groups, p.path or "/")
# ---------------------------------------------------------------- 正文提取
_STRIP_TAGS = ("script", "style", "noscript", "iframe", "form", "nav",
"footer", "aside", "button", "select", "textarea", "input", "svg")
_STRIP_ROLES = ("navigation", "contentinfo", "complementary", "search")
# 正文容器候选(按优先级;首个文本 ≥500 字符者胜出)
_CONTENT_SELECTORS = (
"main", "article", '[role="main"]', "#content", "#main", "#main-content",
".article-body", "#article-body", ".article__body", ".hub-article-body",
".page-content", ".main-content", ".content-area",
)
_CHALLENGE_MARKERS = ("just a moment", "attention required", "captcha",
"access denied", "ddos protection", "verify you are human")
def _decode_html(resp: httpx.Response) -> str:
"""header charset → <meta charset> → utf-8(errors=replace),防 OUP/Wiley 无 header 乱码。"""
enc = None
m = re.search(r"charset=([\w.-]+)", resp.headers.get("content-type", ""), re.I)
if m:
enc = m.group(1)
if not enc:
m = re.search(rb"""charset\s*=\s*["']?([\w.-]+)""", resp.content[:4096], re.I)
if m:
enc = m.group(1).decode("ascii", "ignore")
try:
return resp.content.decode(enc or "utf-8", errors="replace")
except LookupError:
return resp.content.decode("utf-8", errors="replace")
def _squeeze(s: str) -> str:
"""行内空白压缩 + 去空行(保留单换行)。"""
lines = [re.sub(r"[ \t\r\f\v]+", " ", line).strip() for line in s.split("\n")]
return "\n".join(line for line in lines if line)
def _inline(el) -> str:
"""行内渲染:a→md链接 / strong→** / em→* / code→` / br→换行;跳过列表与图片。"""
parts: list[str] = []
for child in el.children:
name = getattr(child, "name", None)
if name is None: # NavigableString
parts.append(str(child))
continue
if name in ("ul", "ol", "img", "picture", "figure"):
continue
if name == "a":
href = (child.get("href") or "").strip()
txt = _inline(child).strip()
if not txt:
continue
parts.append(f"[{txt}]({href})" if href.startswith(("http://", "https://")) else txt)
elif name in ("strong", "b"):
parts.append(f"**{_inline(child)}**")
elif name in ("em", "i"):
parts.append(f"*{_inline(child)}*")
elif name == "code":
parts.append(f"`{_inline(child).strip()}`")
elif name == "br":
parts.append("\n")
else:
parts.append(_inline(child))
return "".join(parts)
def _list_md(el, ordered: bool) -> str:
lines: list[str] = []
idx = 1
for li in el.find_all("li", recursive=False):
# li 文本不含嵌套列表(嵌套列表随后缩进输出)
buf = []
for child in li.children:
if getattr(child, "name", None) in ("ul", "ol"):
continue
buf.append(_inline(child) if getattr(child, "name", None) else str(child))
txt = " ".join("".join(buf).split())
if txt:
lines.append(f"{idx}. {txt}" if ordered else f"- {txt}")
idx += 1
for nested in li.find_all(("ul", "ol"), recursive=False):
sub = _list_md(nested, nested.name == "ol")
if sub:
lines.extend(" " + line for line in sub.split("\n"))
return "\n".join(lines)
def _table_md(table) -> str | None:
rows: list[list[str]] = []
for tr in table.find_all("tr"):
cells = [" ".join(_squeeze(_inline(c)).split())
for c in tr.find_all(("td", "th"))]
if cells:
rows.append(cells)
if not rows:
return None
out = ["| " + " | ".join(rows[0]) + " |"]
if table.find("th"): # 首行含表头 → 补分隔行(合法 GFM 表)
out.append("| " + " | ".join(["---"] * len(rows[0])) + " |")
out += ["| " + " | ".join(r) + " |" for r in rows[1:]]
return "\n".join(out)
def _blocks(el, out: list[str]) -> None:
for child in el.children:
name = getattr(child, "name", None)
if name is None: # 裸文本
txt = _squeeze(str(child))
if txt:
out.append(txt)
continue
if name in _STRIP_TAGS:
continue
if name in ("h1", "h2", "h3", "h4", "h5", "h6"):
txt = " ".join(_squeeze(_inline(child)).split())
if txt:
out.append("#" * int(name[1]) + " " + txt)
elif name == "p":
txt = _squeeze(_inline(child))
if txt:
out.append(txt)
elif name in ("ul", "ol"):
md = _list_md(child, name == "ol")
if md:
out.append(md)
elif name == "table":
md = _table_md(child)
if md:
out.append(md)
elif name == "blockquote":
txt = _squeeze(_inline(child))
if txt:
out.append("\n".join("> " + line for line in txt.split("\n")))
elif name == "pre":
code = child.get_text().strip("\n")
if code.strip():
out.append("```\n" + code + "\n```")
elif name == "hr":
out.append("---")
else: # div/section/article/main/figure/dl/details/… 容器递归
_blocks(child, out)
def _html_to_markdown(html: str) -> tuple[str, str | None]:
"""→ (markdown, error)。选正文容器 → 去壳 → DOM 递归转 Markdown。"""
soup = BeautifulSoup(html, "lxml") # lxml 在 lock 内;缺时 bs4 自动回退 html.parser
title = (soup.title.get_text() if soup.title else "") or ""
for tag in soup.find_all(_STRIP_TAGS):
tag.decompose()
for tag in soup.find_all(attrs={"role": _STRIP_ROLES}):
tag.decompose()
container = None
fallback_hits = []
for sel in _CONTENT_SELECTORS:
for node in soup.select(sel):
text_len = len(node.get_text(" ", strip=True))
fallback_hits.append((text_len, node))
if text_len >= 500:
container = node
break
if container is not None:
break
if container is None:
if fallback_hits:
container = max(fallback_hits, key=lambda x: x[0])[1]
else:
container = soup.body or soup
blocks: list[str] = []
_blocks(container, blocks)
md = "\n\n".join(blocks)
probe = (title + " " + md[:500]).lower()
marker = next((m for m in _CHALLENGE_MARKERS if m in probe), None)
if marker:
return "", f"疑似反爬/挑战页(命中 {marker!r},title={title[:80]!r})"
if len(md) < _MIN_TEXT_CHARS:
return "", f"正文提取过短({len(md)} < {_MIN_TEXT_CHARS} 字符),疑似 JS 渲染或空壳页"
if len(md) > MAX_CHARS:
md = md[:MAX_CHARS - 1] + "…"
return md, None
# ---------------------------------------------------------------- 入口
async def fetch_submission_guidelines(url: str) -> dict:
"""抓投稿指南页面 → 正文 Markdown。
流程:robots.txt 检查(≥2s 限速)→ 禁止即跳过 → GET 正文(≥2s 限速)
→ 状态/类型校验 → 正文提取 → 截断 30000 字符。
"""
parsed = urlparse(url)
if parsed.scheme not in ("http", "https") or not parsed.netloc:
return _fail(f"非法 URL:{url!r}")
host = parsed.netloc.lower()
async with httpx.AsyncClient(follow_redirects=True, timeout=TIMEOUT) as client:
allowed, robots_reason = await _robots_allows(client, url)
if not allowed:
logger.info("[fetch] robots 禁止,跳过 %s(%s)", url, robots_reason)
return _fail(robots_reason)
await _throttle(host)
try:
resp = await client.get(url, headers=_HEADERS)
except httpx.HTTPError as exc:
return _fail(f"请求失败({type(exc).__name__}):{exc}")
if resp.status_code != 200:
return _fail(f"HTTP {resp.status_code}")
ctype = resp.headers.get("content-type", "")
if "html" not in ctype and "xhtml" not in ctype:
return _fail(f"非 HTML 响应:{ctype or '无 Content-Type'}")
markdown, err = _html_to_markdown(_decode_html(resp))
if err:
return _fail(f"{err}(robots: {robots_reason})")
logger.info("[fetch] %s → %d 字符(robots: %s)", url, len(markdown), robots_reason)
return {"ok": True, "markdown": markdown, "error": None, "fetched_at": _now_iso()}
-223
View File
@@ -1,223 +0,0 @@
"""services/journal_kb.py —— 期刊 scope_desc → vstore 入库 + 向量匹配(只读消费 8094)。
链路(全部复用 vstore 既有 API,零改动):
1. ensure_scope_docs:各期刊 scope_desc 以 .txt 入库(category=journal_scope,
name=期刊名),document_id 回写 journals.vstore_doc_id(幂等:已有 id 不重灌)。
2. match:GET /api/vstore/search-chunks?q=<摘要截断>&doc_ids=<id 白名单>&top_k=N
→ 按 doc_id 映射回期刊,取该期刊最高 chunk 分数为 similarity。
3. 自愈:白名单命中 0 条 → GET /documents/{id} 探活 → 失效的重灌(vstore 去重
语义下同内容重灌会换 doc_id,需回写)→ 重试一次 → 仍 0 视为语义低相似。
错误语义:连通性/上游异常抛 MatchError(路由降级为全量候选);
返回 ([], False) 表示检索成功但无有效匹配(路由同样降级)。
"""
from __future__ import annotations
import logging
from typing import Any
import httpx
from ..config import get_settings
from ..db import set_vstore_doc_id
logger = logging.getLogger(__name__)
SCOPE_CATEGORY = "journal_scope"
class MatchError(Exception):
"""vstore 向量匹配失败(不可达/响应异常)。"""
def _timeout() -> httpx.Timeout:
t = get_settings().vstore_match_timeout
return httpx.Timeout(connect=5.0, read=t, write=t, pool=5.0)
def _err_body(resp: httpx.Response) -> str:
try:
payload = resp.json()
except ValueError:
return (resp.text or "").strip()[:500] or "(空响应体)"
if isinstance(payload, dict):
err = payload.get("error")
if isinstance(err, dict) and err.get("message"):
return str(err["message"])
if isinstance(err, str):
return err
return str(payload)[:500]
async def _ingest_scope(client: httpx.AsyncClient, base: str, journal: dict[str, Any]) -> str:
"""单本期刊 scope_desc 入库,返回 document_id。"""
content = str(journal.get("scope_desc") or "").strip()
if not content:
raise MatchError(f"期刊 scope_desc 为空:{journal.get('id')}")
jid = str(journal.get("id") or "")
try:
resp = await client.post(
f"{base}/api/vstore/ingest",
files={"file": (f"{jid}.txt", content.encode("utf-8"), "text/plain")},
data={
"name": str(journal.get("name") or jid),
"category": SCOPE_CATEGORY,
"description": "review_service 期刊范围种子(scope_desc 向量匹配)",
},
)
except httpx.HTTPError as exc:
raise MatchError(f"vstore 不可达({base}):{exc}") from exc
if resp.status_code >= 400:
raise MatchError(f"vstore ingest 失败(HTTP {resp.status_code}):{_err_body(resp)}")
try:
payload = resp.json()
except ValueError as exc:
raise MatchError(f"vstore ingest 响应非 JSON:{exc}") from exc
if not isinstance(payload, dict) or not payload.get("ok"):
raise MatchError(
str((payload.get("error") or {}).get("message") if isinstance(payload, dict) else "") or "vstore ingest 失败"
)
doc_id = str((payload.get("item") or {}).get("document_id") or "")
if not doc_id:
raise MatchError("vstore ingest 响应缺少 document_id")
return doc_id
async def ensure_scope_docs(journals: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""补齐缺失的 scope 入库映射(就地回写 vstore_doc_id 并持久到 DB)。"""
settings = get_settings()
missing = [j for j in journals if not j.get("vstore_doc_id")]
if not missing:
return journals
async with httpx.AsyncClient(timeout=_timeout()) as client:
for j in missing:
try:
doc_id = await _ingest_scope(client, settings.vstore_base, j)
except MatchError as exc:
# 单本失败不阻断其余(该期刊将不进白名单,由路由降级兜底)
logger.warning("journal_kb: scope 入库失败(%s):%s", j.get("id"), exc)
continue
j["vstore_doc_id"] = doc_id
await set_vstore_doc_id(str(j["id"]), doc_id)
return journals
async def _search(
client: httpx.AsyncClient, base: str, query: str, doc_ids: list[str]
) -> list[dict[str, Any]]:
"""chunk 级白名单检索。连通/上游错误抛 MatchError。"""
if not doc_ids:
# doc_ids 为空时 vstore 侧等于不过滤 → 会命中用户私有库,必须拦死
raise MatchError("白名单为空,拒绝全库检索")
try:
resp = await client.get(
f"{base}/api/vstore/search-chunks",
params={
"q": query,
"top_k": get_settings().recommend_scope_top_k,
"doc_ids": ",".join(doc_ids),
},
)
except httpx.HTTPError as exc:
raise MatchError(f"vstore search-chunks 不可达:{exc}") from exc
if resp.status_code >= 400:
raise MatchError(f"vstore search-chunks 失败(HTTP {resp.status_code}):{_err_body(resp)}")
try:
payload = resp.json()
except ValueError as exc:
raise MatchError(f"vstore search-chunks 响应非 JSON:{exc}") from exc
if not isinstance(payload, dict) or not payload.get("ok"):
raise MatchError(
str((payload.get("error") or {}).get("message") if isinstance(payload, dict) else "") or "vstore search-chunks 失败"
)
items = payload.get("items")
return items if isinstance(items, list) else []
async def _doc_alive(client: httpx.AsyncClient, base: str, doc_id: str) -> bool:
try:
resp = await client.get(f"{base}/api/vstore/documents/{doc_id}")
except httpx.HTTPError:
return False
if resp.status_code != 200:
return False
try:
payload = resp.json()
except ValueError:
return False
return bool(isinstance(payload, dict) and payload.get("ok"))
async def _refresh_dead_docs(
client: httpx.AsyncClient, base: str, journals: list[dict[str, Any]]
) -> None:
"""白名单 0 命中时探活:失效文档重灌并回写新 id。"""
for j in journals:
doc_id = str(j.get("vstore_doc_id") or "")
if not doc_id or await _doc_alive(client, base, doc_id):
continue
try:
new_id = await _ingest_scope(client, base, j)
except MatchError as exc:
logger.warning("journal_kb: scope 重灌失败(%s):%s", j.get("id"), exc)
continue
j["vstore_doc_id"] = new_id
await set_vstore_doc_id(str(j["id"]), new_id)
async def match(
abstract: str, journals: list[dict[str, Any]]
) -> tuple[list[dict[str, Any]], bool]:
"""摘要 vs scope_desc 向量匹配。
返回 (candidates, vector_ok);candidates 元素:
{"journal": <journals 行>, "similarity": float}
按 similarity 降序。MatchError → 路由降级;([], False) → 语义低相似。
"""
settings = get_settings()
query = abstract.strip()[: settings.recommend_query_chars]
if not query:
return [], False
await ensure_scope_docs(journals)
whitelisted = [j for j in journals if j.get("vstore_doc_id")]
if not whitelisted:
raise MatchError("无任何期刊 scope 入库成功(vstore_doc_id 为空)")
doc_map = {str(j["vstore_doc_id"]): j for j in whitelisted}
timeout = _timeout()
async with httpx.AsyncClient(timeout=timeout) as client:
items = await _search(client, settings.vstore_base, query, list(doc_map))
if not items:
await _refresh_dead_docs(client, settings.vstore_base, whitelisted)
doc_map = {str(j["vstore_doc_id"]): j for j in whitelisted if j.get("vstore_doc_id")}
if not doc_map:
return [], False
items = await _search(client, settings.vstore_base, query, list(doc_map))
# 聚合:每本期刊取最高 chunk 分
best: dict[str, float] = {}
for item in items:
if not isinstance(item, dict):
continue
j = doc_map.get(str(item.get("doc_id") or ""))
if j is None:
continue
try:
score = float(item.get("score"))
except (TypeError, ValueError):
continue
jid = str(j["id"])
if jid not in best or score > best[jid]:
best[jid] = score
if not best:
return [], False
by_id = {str(j["id"]): j for j in journals}
candidates = [
{"journal": by_id[jid], "similarity": score}
for jid, score in best.items()
if jid in by_id
]
candidates.sort(key=lambda c: c["similarity"], reverse=True)
return candidates, bool(candidates)
-210
View File
@@ -1,210 +0,0 @@
"""services/journal_rank.py —— LLM 重排 + 幻觉防护 + 三档归类。
职责:
1. rerank:把带语义分的候选期刊(来自 journal_kb)+ 用户偏好交给 LLM,
要求按 id 白名单返回 {rankings:[{id, tier, match_reason, risk_note}]}。
2. validate:硬校验——白名单外 id 丢弃(幻觉拦截);tier 非法/LLM 缺漏的
期刊按规则补位(绝不因 LLM 输出丢失候选期刊);期刊名称/指标一律不采信
LLM,最终展示全部用 DB 字段(防止改名型幻觉)。
3. fallback_rankings:LLM 不可用时的确定性规则分档(按 impact_factor)。
三档语义:challenge 挑战型(高影响/慢审稿)、fit 适配型(均衡)、
safe 稳妥型(快速/门槛低)。
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from typing import Any
from ..config import get_settings
from . import llm
TIERS = ("challenge", "fit", "safe")
TIER_LABELS = {
"challenge": "挑战型(高影响因子/慢审稿)",
"fit": "适配型(中等影响/周期适中)",
"safe": "稳妥型(快速发表/门槛较低)",
}
# 规则分档阈值(近似值,可后续配置化)
_CHALLENGE_MIN_IF = 9.0
_SAFE_MAX_IF = 4.0
# tier 同义词归一(LLM 偶发中文标签)
_TIER_ALIASES = {
"挑战": "challenge",
"挑战型": "challenge",
"高风险": "challenge",
"适配": "fit",
"适配型": "fit",
"稳妥": "safe",
"稳妥型": "safe",
"低风险": "safe",
}
SYSTEM_PROMPT = """你是学术投稿推荐专家(植物基因组学方向)。根据给定候选期刊、论文摘要与用户偏好,为**每一个**候选期刊输出推荐分档与理由。
硬规则(违反即视为无效输出):
1. id 白名单:只能使用输入 candidates 中出现的 id;绝对禁止添加、替换或臆造列表之外的任何期刊。
2. 每个输入 id 恰好输出一次,不得遗漏、不得重复;期刊名称等元数据由系统填充,你只输出 id。
3. tier 只能取 challenge | fit | safe:
- challenge(挑战型):高影响因子、审稿较慢或竞争激烈的期刊
- fit(适配型):影响力与发表周期均衡、与论文契合的期刊
- safe(稳妥型):审稿较快、门槛较低、命中概率较高的期刊
4. match_reason:中文,1-2 句,结合摘要的研究主题与该刊 scope_desc 及指标(如有 similarity 一并参考),说明为何匹配/不匹配。
5. risk_note:中文,1-2 句,指出投稿风险(超出用户偏好、IF 门槛偏高、范围匹配偏弱、审稿周期长等);无明显风险写「无」。
6. 用户 preferences 为软约束:不满足偏好的期刊仍必须输出,并在 risk_note 中明确说明。
7. 不确定时宁可保守陈述,禁止编造统计数据(录用率、审稿天数等)。
输出:仅一个 JSON 对象,格式:
{"rankings":[{"id":"...","tier":"challenge","match_reason":"...","risk_note":"..."}]}
不要输出任何 JSON 之外的内容。"""
class RankError(Exception):
"""LLM 输出形态异常(无法按 JSON 解析/结构非法)。"""
@dataclass
class Ranking:
journal_id: str
tier: str
match_reason: str
risk_note: str
def rule_tier(journal: dict[str, Any]) -> str:
"""确定性分档规则(LLM 缺漏/不可用时兜底)。"""
inf = journal.get("impact_factor")
weeks = journal.get("review_time_weeks")
if isinstance(inf, (int, float)):
if inf >= _CHALLENGE_MIN_IF:
return "challenge"
if inf <= _SAFE_MAX_IF:
return "safe"
elif isinstance(weeks, (int, float)) and weeks <= 4:
return "safe"
return "fit"
def build_user_prompt(
abstract: str,
candidates: list[dict[str, Any]],
preferences: dict[str, Any],
) -> str:
"""候选 + 偏好 + 摘要 → 用户消息(JSON)。"""
payload = {
"abstract": abstract.strip()[:4000],
"preferences": preferences or {},
"candidates": [
{
"id": str(c["journal"]["id"]),
"name": str(c["journal"].get("name") or ""),
"impact_factor": c["journal"].get("impact_factor"),
"review_time_weeks": c["journal"].get("review_time_weeks"),
"acceptance_rate": c["journal"].get("acceptance_rate"),
"similarity": c.get("similarity"),
"scope_desc": str(c["journal"].get("scope_desc") or "")[:400],
}
for c in candidates
],
}
return json.dumps(payload, ensure_ascii=False)
def _normalize_tier(raw: Any) -> str | None:
tier = str(raw or "").strip().lower()
tier = _TIER_ALIASES.get(str(raw or "").strip(), tier)
return tier if tier in TIERS else None
def validate_rankings(
payload: Any, candidates: list[dict[str, Any]]
) -> tuple[list[Ranking], int, int]:
"""校验 LLM 输出。
返回 (rankings, dropped_hallucinated, repaired):
- 白名单外 id / 重复 id → 丢弃并计数(repaired 中重复不计);
- tier 非法、LLM 缺漏的 id → 按规则补位(repaired 计数)。
"""
if not isinstance(payload, dict) or not isinstance(payload.get("rankings"), list):
raise RankError("LLM 输出缺少 rankings 数组")
by_id = {str(c["journal"]["id"]): c["journal"] for c in candidates}
input_ids = list(by_id)
seen: set[str] = set()
rankings: list[Ranking] = []
dropped = 0
repaired = 0
for entry in payload["rankings"]:
if not isinstance(entry, dict):
dropped += 1
continue
jid = str(entry.get("id") or "").strip()
if jid in seen:
continue
if jid not in by_id:
dropped += 1 # 幻觉期刊:绝不进输出
continue
seen.add(jid)
tier = _normalize_tier(entry.get("tier"))
if tier is None:
tier = rule_tier(by_id[jid])
repaired += 1
reason = str(entry.get("match_reason") or "").strip() or "基于期刊范围与指标的匹配"
risk = str(entry.get("risk_note") or "").strip() or "无"
rankings.append(Ranking(journal_id=jid, tier=tier, match_reason=reason, risk_note=risk))
for jid in input_ids:
if jid in seen:
continue
rankings.append(
Ranking(
journal_id=jid,
tier=rule_tier(by_id[jid]),
match_reason="LLM 未给出该期刊条目,已按影响因子/周期规则补位",
risk_note="未提供(LLM 输出缺漏,规则兜底)",
)
)
repaired += 1
return rankings, dropped, repaired
async def rerank(
abstract: str,
candidates: list[dict[str, Any]],
preferences: dict[str, Any],
) -> tuple[list[Ranking], int, int]:
"""LLM 重排 + 校验。LLMError / RankError 上抛(路由降级为规则分档)。"""
settings = get_settings()
payload = await llm.complete_json(
SYSTEM_PROMPT,
build_user_prompt(abstract, candidates, preferences),
temperature=0.2,
max_tokens=3000,
timeout=settings.recommend_read_timeout,
)
return validate_rankings(payload, candidates)
def fallback_rankings(candidates: list[dict[str, Any]]) -> list[Ranking]:
"""LLM 不可用时的确定性分档(不丢任何候选期刊)。"""
rankings: list[Ranking] = []
for c in candidates:
j = c["journal"]
inf = j.get("impact_factor")
weeks = j.get("review_time_weeks")
rankings.append(
Ranking(
journal_id=str(j["id"]),
tier=rule_tier(j),
match_reason=(
f"规则分档:影响因子 {inf if inf is not None else '未收录'}、"
f"审稿约 {weeks if weeks is not None else '未收录'} 周"
),
risk_note="LLM 不可用,本条按影响因子/周期规则估算",
)
)
return rankings

Some files were not shown because too many files have changed in this diff Show More