Compare commits
16
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3c00e1675e | ||
|
|
8ab4279b04 | ||
|
|
c943e8d363 | ||
|
|
9ac36e99ba | ||
|
|
9cb0bf2e35 | ||
|
|
6b4b5e571f | ||
|
|
9797b1b414 | ||
|
|
b9bc90ffb0 | ||
|
|
821f3c1c4e | ||
|
|
da8f531bbc | ||
|
|
21c79582bf | ||
|
|
1ecc4cb29e | ||
|
|
75eb489fc3 | ||
|
|
e4ed3589d5 | ||
|
|
f27336e961 | ||
|
|
963a87855c |
+3
-13
@@ -1,4 +1,4 @@
|
||||
# 运行时产物与环境
|
||||
# 运行时产物与环境
|
||||
node_modules
|
||||
dist/
|
||||
.venv/
|
||||
@@ -43,17 +43,7 @@ paper_writer_service/paper.db-shm
|
||||
paper_writer_service/paper.db-wal
|
||||
paper_writer_service/paper.log
|
||||
|
||||
# review_service 运行时产物(不入库)
|
||||
review_service/review.db
|
||||
review_service/review.db-wal
|
||||
review_service/review.db-shm
|
||||
review_service/review.log
|
||||
review_service/data/reformat_jobs/
|
||||
# 期刊种子 journals.json 是源码:顶层 **/data/ 会误伤 review_service/data/,显式放行
|
||||
!review_service/data/
|
||||
review_service/tools/tectonic
|
||||
review_service/data/tectonic-cache/
|
||||
review_service/data/format_cache/
|
||||
|
||||
# 部署方案文档(含主机凭据,不入库)
|
||||
docs/DEPLOY_PAPER_176.md
|
||||
# Python 覆盖率运行时产物(coverage.py --cov 生成)
|
||||
.coveragevstore/vstore.log
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
# AGENTS.md —— 协作约束(科研审稿模块)
|
||||
|
||||
## 项目背景
|
||||
|
||||
本项目(PlantAI-rag-new)是植物基因组学文献知识工作台,拥有自研 RAG 与知识库
|
||||
(Qdrant + Neo4j + vstore)。2026-10 起新增「科研投稿与智能审稿助手」功能
|
||||
(审稿周期/期刊分析/期刊推荐/投稿前自审/审稿意见辅助/格式检查)。
|
||||
|
||||
## 审稿模块(review_service)约束 —— 必须遵守
|
||||
|
||||
1. **所有审稿相关新代码只允许写在 `review_service/` 目录下新增**。
|
||||
- 服务自身:FastAPI 独立进程,端口 8100(模式同 `paper_writer_service` 8097 / `bioflow_service` 8098)。
|
||||
- 环境变量前缀统一 `REVIEW_`;LLM 凭据复用 `DOCGEN_LLM_*` 裸键(回退链见 `review_service/app/services/llm.py`)。
|
||||
2. **BFF(`sdk/`)层只允许条件注册代理路由**:
|
||||
- 允许的全部改动 = `sdk/app/config.py` 增加 `review_enabled` / `review_service_url` 等开关键
|
||||
+ `sdk/app/main.py` 按开关 `include_router(review_proxy.router)`
|
||||
+ 新文件 `sdk/app/routers/review_proxy.py`(反代实现)。
|
||||
- **严禁**修改 sdk 下其他 router/service/db 的任何既有逻辑。
|
||||
3. **严禁修改现有 RAG 核心与既有服务代码**,包括但不限于:
|
||||
- `agent/`(qa 8091 / methodology 8092 / legacy-rag 8093)
|
||||
- `vstore/`(8094,审稿服务只经 HTTP 只读消费其 `/api/vstore/*` 接口)
|
||||
- `paper_writer_service/`、`bioflow_service/`、`bioflow_kernel/`
|
||||
- `ui/`(前端入口另行按 UI_SPEC 流程评审)
|
||||
- `knowledge/`、`kb-libraries/`、`tooling/`(依赖锁另议)
|
||||
4. **依赖纪律**:审稿服务只使用统一 venv(`tooling/requirements.lock`)中已有的库
|
||||
(fastapi / httpx / pydantic-settings 等)。如确需新增全局依赖,
|
||||
须先更新 `tooling/requirements.lock` 并经人工确认。
|
||||
5. **上游对接只读**:PDF 解析走 vstore 8094(`POST /api/vstore/ingest` +
|
||||
`GET/DELETE /api/vstore/documents/{id}`);LLM 走 DeepSeek OpenAI 兼容接口;
|
||||
后续期刊数据/期刊推荐等子功能同样以新 service + 新路由在 `review_service/` 内扩展。
|
||||
6. **format_check 例外**:format_check 允许用 pymupdf 本地解档获取版式元数据
|
||||
(字体/边距/行距/页数),因为 vstore 只回文本不返回版式信息。正文提取仍走 vstore。
|
||||
|
||||
## 目录速览(审稿模块)
|
||||
|
||||
```
|
||||
review_service/
|
||||
├── app/main.py # FastAPI 入口(/health + 挂载业务路由)
|
||||
├── app/config.py # pydantic-settings(REVIEW_ 前缀)
|
||||
├── app/routers/presubmit.py # POST /api/review/presubmit(自审稿报告)
|
||||
├── app/routers/format_check.py # POST /api/review/format-check(版式与规范检查)
|
||||
├── app/services/llm.py # DeepSeek httpx 封装(complete_text/complete_json)
|
||||
├── app/services/paper_parse.py # PDF → 纯文本(经 vstore 解析接口)
|
||||
├── app/services/format_check.py# 版式规则引擎(pymupdf/docx 本地解档 + 规则检查)
|
||||
├── restart.sh # 8100 重启脚本
|
||||
└── pytest.ini
|
||||
```
|
||||
|
||||
## 启动
|
||||
|
||||
```bash
|
||||
cd review_service && bash restart.sh # 默认 8100,venv 从 ../plantai-rag-new-venv 推导
|
||||
# BFF 开关:sdk/.env 加 REVIEW_ENABLED=1 后 bff_restart(反代 /api/review/*)
|
||||
```
|
||||
@@ -1,12 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN" data-theme="dark">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
|
||||
<title>蜂群中枢 · PlantAI 管理后台</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script type="module" src="/src/main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
Generated
-1147
File diff suppressed because it is too large
Load Diff
@@ -1,18 +0,0 @@
|
||||
{
|
||||
"name": "plantai-admin",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"vue": "3.5.13"
|
||||
},
|
||||
"devDependencies": {
|
||||
"vite": "5.4.11",
|
||||
"@vitejs/plugin-vue": "5.2.1"
|
||||
}
|
||||
}
|
||||
@@ -1,143 +0,0 @@
|
||||
<script setup>
|
||||
import { ref, onMounted } from 'vue';
|
||||
import { getToken, setToken, api } from './api/admin.js';
|
||||
import Login from './views/Login.vue';
|
||||
import Users from './views/Users.vue';
|
||||
import Logs from './views/Logs.vue';
|
||||
import HubSettings from './components/HubSettings.vue';
|
||||
|
||||
const authed = ref(false);
|
||||
const page = ref('overview');
|
||||
const user = ref(null);
|
||||
const stats = ref({ users: 0, total: 0, denied: 0, p95: null });
|
||||
const hub = ref({ hub_id: '', hub_name: '蜂群中枢' });
|
||||
const settingsOpen = ref(false);
|
||||
|
||||
const PAGES = { overview: '概览', users: '用户管理', logs: '日志查看' };
|
||||
|
||||
function onUnauthorized() { setToken(''); authed.value = false; }
|
||||
|
||||
async function loadHub() {
|
||||
try {
|
||||
const r = await api.settings();
|
||||
hub.value = r.settings;
|
||||
} catch (e) {
|
||||
if (e.status === 401) onUnauthorized();
|
||||
}
|
||||
}
|
||||
|
||||
async function loadStats() {
|
||||
try {
|
||||
const [u, l] = await Promise.all([api.users(), api.logs({ page: 1, page_size: 1 })]);
|
||||
stats.value.users = u.users.length;
|
||||
stats.value.total = l.total;
|
||||
stats.value.denied = (l.aggregates && l.aggregates.denied) || 0;
|
||||
stats.value.p95 = (l.aggregates && l.aggregates.p95_latency_ms) || null;
|
||||
} catch (e) {
|
||||
if (e.status === 401) onUnauthorized();
|
||||
}
|
||||
}
|
||||
|
||||
async function init() {
|
||||
if (!getToken()) { authed.value = false; return; }
|
||||
try {
|
||||
const r = await api.me();
|
||||
user.value = r.user;
|
||||
authed.value = true;
|
||||
loadHub();
|
||||
loadStats();
|
||||
} catch (e) {
|
||||
setToken('');
|
||||
authed.value = false;
|
||||
}
|
||||
}
|
||||
|
||||
function onLogin(u) { user.value = u; authed.value = true; page.value = 'overview'; loadHub(); loadStats(); }
|
||||
async function onLogout() { try { await api.logout(); } catch {} setToken(''); authed.value = false; }
|
||||
function onSettingsSaved(s) { hub.value = s; settingsOpen.value = false; }
|
||||
|
||||
onMounted(init);
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<Login v-if="!authed" @login="onLogin" />
|
||||
<div v-else class="shell">
|
||||
<aside class="sidebar">
|
||||
<div class="side-brand">
|
||||
<div class="hex">蜂</div>
|
||||
<div><div class="brand-name">{{ hub.hub_name }}</div><div class="brand-sub">{{ hub.hub_id.slice(0, 8) }}</div></div>
|
||||
</div>
|
||||
<nav class="nav">
|
||||
<div class="nav-item" :class="{ active: page === 'overview' }" @click="page = 'overview'"><span>◈</span> 概览</div>
|
||||
<div class="nav-item" :class="{ active: page === 'users' }" @click="page = 'users'"><span>◉</span> 用户管理</div>
|
||||
<div class="nav-item" :class="{ active: page === 'logs' }" @click="page = 'logs'"><span>≣</span> 日志查看</div>
|
||||
</nav>
|
||||
<div class="side-foot">
|
||||
<div class="who">
|
||||
<div class="avatar">蜂</div>
|
||||
<div><div class="who-name">{{ user?.username }}</div><div class="who-role">系统管理员</div></div>
|
||||
<span class="logout" @click="onLogout">退出</span>
|
||||
</div>
|
||||
</div>
|
||||
</aside>
|
||||
<main class="main">
|
||||
<div class="topbar">
|
||||
<h1>{{ PAGES[page] }}</h1>
|
||||
<span style="color: var(--text-3); font-size: 13px">{{ hub.hub_name }}</span>
|
||||
<button class="btn btn-ghost" style="margin-left: auto" @click="settingsOpen = true">⚙ 设置</button>
|
||||
</div>
|
||||
<div class="content">
|
||||
<section v-if="page === 'overview'">
|
||||
<div class="stats">
|
||||
<div class="stat"><div class="k">{{ stats.users }}</div><div class="l">注册用户</div></div>
|
||||
<div class="stat"><div class="k amber">{{ stats.total }}</div><div class="l">累计请求</div></div>
|
||||
<div class="stat"><div class="k">{{ stats.denied }}</div><div class="l">越权 / 失败</div></div>
|
||||
<div class="stat"><div class="k">{{ stats.p95 ?? '—' }}</div><div class="l">P95 耗时 ms</div></div>
|
||||
</div>
|
||||
</section>
|
||||
<Users v-else-if="page === 'users'" @unauthorized="onUnauthorized" />
|
||||
<Logs v-else @unauthorized="onUnauthorized" />
|
||||
</div>
|
||||
</main>
|
||||
<HubSettings :open="settingsOpen" :hub="hub" @close="settingsOpen = false" @saved="onSettingsSaved" />
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style>
|
||||
.shell { display: flex; height: 100vh; }
|
||||
.sidebar { width: 218px; flex: none; background: var(--bg-2); border-right: 1px solid var(--border); display: flex; flex-direction: column; }
|
||||
.side-brand { display: flex; align-items: center; gap: 10px; padding: 20px 18px 18px; }
|
||||
.hex { width: 30px; height: 30px; flex: none; background: linear-gradient(135deg, #f2b95a, #d98f22); clip-path: polygon(50% 0, 100% 25%, 100% 75%, 50% 100%, 0 75%, 0 25%); display: grid; place-items: center; color: #1a1206; font-weight: 700; font-size: 12px; }
|
||||
.brand-name { font-size: 15px; font-weight: 600; }
|
||||
.brand-sub { font-size: 11px; color: var(--text-3); }
|
||||
.nav { padding: 6px 10px; flex: 1; }
|
||||
.nav-item { display: flex; align-items: center; gap: 9px; padding: 9px 12px; margin-bottom: 3px; border-radius: var(--radius-sm); color: var(--text-2); font-size: 13.5px; cursor: pointer; }
|
||||
.nav-item:hover { background: var(--surface); color: var(--text); }
|
||||
.nav-item.active { background: var(--accent-soft); color: var(--accent); }
|
||||
.side-foot { padding: 14px 18px; border-top: 1px solid var(--border); }
|
||||
.who { display: flex; align-items: center; gap: 10px; }
|
||||
.avatar { width: 30px; height: 30px; border-radius: 50%; flex: none; background: linear-gradient(135deg, #3a4a5e, #243146); display: grid; place-items: center; font-size: 12px; color: var(--text); }
|
||||
.who-name { font-size: 13px; }
|
||||
.who-role { font-size: 11px; color: var(--text-3); }
|
||||
.logout { margin-left: auto; color: var(--text-3); cursor: pointer; font-size: 12px; }
|
||||
.logout:hover { color: var(--red); }
|
||||
.main { flex: 1; overflow-y: auto; display: flex; flex-direction: column; }
|
||||
.topbar { display: flex; align-items: center; gap: 14px; padding: 18px 28px; border-bottom: 1px solid var(--border); position: sticky; top: 0; background: rgba(12, 14, 20, 0.9); backdrop-filter: blur(6px); z-index: 5; }
|
||||
.topbar h1 { font-size: 17px; margin: 0; font-weight: 600; }
|
||||
.content { padding: 26px 28px 40px; }
|
||||
.stats { display: grid; grid-template-columns: repeat(4, 1fr); gap: 16px; }
|
||||
.stat { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius); padding: 18px 20px; }
|
||||
.stat .k { font-size: 26px; font-weight: 600; letter-spacing: -0.5px; }
|
||||
.stat .k.amber { color: var(--accent); }
|
||||
.stat .l { color: var(--text-2); font-size: 12.5px; margin-top: 3px; }
|
||||
.toolbar { display: flex; align-items: center; gap: 10px; margin-bottom: 16px; }
|
||||
.toolbar .spacer { flex: 1; }
|
||||
.search { display: flex; align-items: center; gap: 8px; background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius-sm); padding: 8px 12px; width: 240px; }
|
||||
.search input { background: none; border: none; outline: none; color: var(--text); font-size: 13px; width: 100%; }
|
||||
select { background: var(--surface); border: 1px solid var(--border); color: var(--text); border-radius: var(--radius-sm); padding: 8px 10px; font-size: 13px; outline: none; }
|
||||
.pag { display: flex; align-items: center; gap: 8px; padding: 12px 18px; border-top: 1px solid var(--border); color: var(--text-3); font-size: 12.5px; }
|
||||
.pag .spacer { flex: 1; }
|
||||
.pag button { background: var(--surface-2); border: 1px solid var(--border); color: var(--text-2); border-radius: 6px; padding: 5px 11px; font-size: 12.5px; }
|
||||
.pag button.on { color: var(--accent); border-color: var(--accent); }
|
||||
.pag button:disabled { opacity: 0.4; cursor: default; }
|
||||
</style>
|
||||
@@ -1,48 +0,0 @@
|
||||
const TOKEN_KEY = 'pa-admin-token';
|
||||
let token = localStorage.getItem(TOKEN_KEY) || '';
|
||||
|
||||
export function getToken() { return token; }
|
||||
export function setToken(t) {
|
||||
token = t || '';
|
||||
if (t) localStorage.setItem(TOKEN_KEY, t);
|
||||
else localStorage.removeItem(TOKEN_KEY);
|
||||
}
|
||||
|
||||
async function request(path, { method = 'GET', body } = {}) {
|
||||
const res = await fetch('/api/admin' + path, {
|
||||
method,
|
||||
headers: {
|
||||
...(body ? { 'Content-Type': 'application/json' } : {}),
|
||||
...(token ? { Authorization: 'Bearer ' + token } : {}),
|
||||
},
|
||||
body: body ? JSON.stringify(body) : undefined,
|
||||
});
|
||||
let data = null;
|
||||
try { data = await res.json(); } catch { data = null; }
|
||||
if (!res.ok) {
|
||||
const err = new Error((data && data.error && data.error.message) || ('请求失败 HTTP ' + res.status));
|
||||
err.status = res.status;
|
||||
err.code = data && data.error && data.error.code;
|
||||
throw err;
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
export const api = {
|
||||
login: (username, password) => request('/login', { method: 'POST', body: { username, password } }),
|
||||
logout: () => request('/logout', { method: 'POST' }),
|
||||
me: () => request('/me'),
|
||||
units: () => request('/units'),
|
||||
users: () => request('/users'),
|
||||
createUser: (body) => request('/users', { method: 'POST', body }),
|
||||
updateUser: (id, body) => request('/users/' + id, { method: 'PATCH', body }),
|
||||
deleteUser: (id) => request('/users/' + id, { method: 'DELETE' }),
|
||||
logs: (params = {}) => {
|
||||
const q = new URLSearchParams();
|
||||
Object.entries(params).forEach(([k, v]) => { if (v !== undefined && v !== null && v !== '') q.set(k, v); });
|
||||
const qs = q.toString();
|
||||
return request('/logs' + (qs ? '?' + qs : ''));
|
||||
},
|
||||
settings: () => request('/settings'),
|
||||
updateSettings: (body) => request('/settings', { method: 'PUT', body }),
|
||||
};
|
||||
@@ -1,57 +0,0 @@
|
||||
<script setup>
|
||||
import { ref, watch } from 'vue';
|
||||
import { api } from '../api/admin.js';
|
||||
|
||||
const props = defineProps({ open: Boolean, hub: Object });
|
||||
const emit = defineEmits(['close', 'saved']);
|
||||
|
||||
const hubId = ref('');
|
||||
const hubName = ref('');
|
||||
const busy = ref(false);
|
||||
const error = ref('');
|
||||
|
||||
watch(() => props.open, (open) => {
|
||||
if (open) {
|
||||
hubId.value = (props.hub && props.hub.hub_id) || '';
|
||||
hubName.value = (props.hub && props.hub.hub_name) || '';
|
||||
error.value = '';
|
||||
busy.value = false;
|
||||
}
|
||||
});
|
||||
|
||||
async function save() {
|
||||
if (busy.value) return;
|
||||
if (!hubName.value.trim()) { error.value = '中枢名称不能为空'; return; }
|
||||
busy.value = true;
|
||||
error.value = '';
|
||||
try {
|
||||
const r = await api.updateSettings({ hub_id: hubId.value.trim(), hub_name: hubName.value.trim() });
|
||||
emit('saved', r.settings);
|
||||
} catch (e) {
|
||||
error.value = e.message || '保存失败';
|
||||
} finally {
|
||||
busy.value = false;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div v-if="open" class="overlay" @click.self="emit('close')">
|
||||
<div class="modal">
|
||||
<div class="modal-head">中枢设置</div>
|
||||
<div class="modal-body">
|
||||
<div class="field"><label>中枢名称</label><input v-model="hubName" placeholder="例如:湘湖实验室中枢"></div>
|
||||
<div class="field" style="margin-bottom: 0"><label>中枢 ID</label><input v-model="hubId" class="mono" placeholder="唯一标识,留空则由系统生成"></div>
|
||||
<p v-if="error" class="err">{{ error }}</p>
|
||||
</div>
|
||||
<div class="modal-foot">
|
||||
<button class="btn btn-ghost" @click="emit('close')">取消</button>
|
||||
<button class="btn btn-primary" :disabled="busy" @click="save">{{ busy ? '保存中…' : '保存' }}</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style scoped>
|
||||
.err { color: var(--red); font-size: 12px; margin: 12px 0 0; }
|
||||
</style>
|
||||
@@ -1,101 +0,0 @@
|
||||
<script setup>
|
||||
import { ref, watch } from 'vue';
|
||||
import { api } from '../api/admin.js';
|
||||
|
||||
const props = defineProps({
|
||||
open: Boolean,
|
||||
mode: { type: String, default: 'create' },
|
||||
user: Object,
|
||||
});
|
||||
const emit = defineEmits(['close', 'saved']);
|
||||
|
||||
const UNITS = [
|
||||
['qa', '问答'], ['kg', '知识图谱'], ['extraction', '文献抽取'], ['docgen', '文档生成'],
|
||||
['figures', '图片图库'], ['private_docs', '私有资料'], ['bioflow', '生信分析'],
|
||||
['paper', '论文撰写'], ['legacy', '旧版接口'],
|
||||
];
|
||||
|
||||
const username = ref('');
|
||||
const password = ref('');
|
||||
const role = ref('researcher');
|
||||
const authorizations = ref([]);
|
||||
const busy = ref(false);
|
||||
const error = ref('');
|
||||
|
||||
watch(() => props.open, (open) => {
|
||||
if (open) {
|
||||
error.value = '';
|
||||
busy.value = false;
|
||||
if (props.mode === 'edit' && props.user) {
|
||||
username.value = props.user.username;
|
||||
password.value = '';
|
||||
role.value = props.user.role === 'admin' ? 'admin' : (props.user.role === 'guest' ? 'guest' : 'researcher');
|
||||
authorizations.value = [...(props.user.authorizations || [])];
|
||||
} else {
|
||||
username.value = '';
|
||||
password.value = '';
|
||||
role.value = 'researcher';
|
||||
authorizations.value = ['qa'];
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
async function save() {
|
||||
if (busy.value) return;
|
||||
if (props.mode === 'create' && (!username.value.trim() || !password.value)) {
|
||||
error.value = '用户名和密码必填';
|
||||
return;
|
||||
}
|
||||
busy.value = true;
|
||||
error.value = '';
|
||||
try {
|
||||
if (props.mode === 'create') {
|
||||
await api.createUser({ username: username.value.trim(), password: password.value, role: role.value, authorizations: authorizations.value });
|
||||
} else {
|
||||
const body = { role: role.value, authorizations: authorizations.value };
|
||||
if (password.value) body.password = password.value;
|
||||
await api.updateUser(props.user.id, body);
|
||||
}
|
||||
emit('saved');
|
||||
} catch (e) {
|
||||
error.value = e.message || '保存失败';
|
||||
} finally {
|
||||
busy.value = false;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div v-if="open" class="overlay" @click.self="emit('close')">
|
||||
<div class="modal">
|
||||
<div class="modal-head">{{ mode === 'create' ? '新建用户' : '编辑用户' }}</div>
|
||||
<div class="modal-body">
|
||||
<div class="field"><label>用户名</label><input v-model="username" :disabled="mode === 'edit'"></div>
|
||||
<div class="field"><label>{{ mode === 'create' ? '初始密码' : '重置密码(留空不改)' }}</label><input v-model="password" type="password"></div>
|
||||
<div class="field">
|
||||
<label>角色</label>
|
||||
<div class="seg">
|
||||
<button :class="{ on: role === 'researcher' }" @click="role = 'researcher'">研究员</button>
|
||||
<button :class="{ on: role === 'guest' }" @click="role = 'guest'">访客</button>
|
||||
<button :class="{ on: role === 'admin' }" @click="role = 'admin'">管理员</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="field" style="margin-bottom: 0">
|
||||
<label>授权清单</label>
|
||||
<div class="checks">
|
||||
<label v-for="[k, label] in UNITS" :key="k" class="check"><input type="checkbox" :value="k" v-model="authorizations"> {{ label }}</label>
|
||||
</div>
|
||||
</div>
|
||||
<p v-if="error" class="err">{{ error }}</p>
|
||||
</div>
|
||||
<div class="modal-foot">
|
||||
<button class="btn btn-ghost" @click="emit('close')">取消</button>
|
||||
<button class="btn btn-primary" :disabled="busy" @click="save">{{ busy ? '保存中…' : (mode === 'create' ? '创建用户' : '保存') }}</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style scoped>
|
||||
.err { color: var(--red); font-size: 12px; margin: 12px 0 0; }
|
||||
</style>
|
||||
@@ -1,5 +0,0 @@
|
||||
import { createApp } from 'vue';
|
||||
import App from './App.vue';
|
||||
import './style.css';
|
||||
|
||||
createApp(App).mount('#app');
|
||||
@@ -1,96 +0,0 @@
|
||||
:root {
|
||||
--bg: #0c0e14;
|
||||
--bg-2: #10131b;
|
||||
--surface: #151924;
|
||||
--surface-2: #1a1f2c;
|
||||
--border: #232a38;
|
||||
--border-2: #2e3648;
|
||||
--text: #e9edf5;
|
||||
--text-2: #9aa4b6;
|
||||
--text-3: #5f6a7d;
|
||||
--accent: #e8a33d;
|
||||
--accent-ink: #14100a;
|
||||
--accent-soft: rgba(232, 163, 61, 0.14);
|
||||
--green: #3ecf8e;
|
||||
--green-soft: rgba(62, 207, 142, 0.14);
|
||||
--red: #ef6b6b;
|
||||
--red-soft: rgba(239, 107, 107, 0.14);
|
||||
--blue: #6ea8ff;
|
||||
--radius: 10px;
|
||||
--radius-sm: 7px;
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; }
|
||||
html, body, #app { margin: 0; height: 100%; }
|
||||
body {
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
font-family: -apple-system, BlinkMacSystemFont, "PingFang SC", "Microsoft YaHei", "Segoe UI", Roboto, sans-serif;
|
||||
font-size: 14px;
|
||||
line-height: 1.5;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
.mono { font-family: ui-monospace, "SF Mono", "JetBrains Mono", Menlo, Consolas, monospace; }
|
||||
button { font-family: inherit; cursor: pointer; }
|
||||
|
||||
.btn {
|
||||
display: inline-flex; align-items: center; justify-content: center; gap: 7px;
|
||||
padding: 9px 15px; font-size: 13.5px; border-radius: var(--radius-sm);
|
||||
border: 1px solid transparent; transition: filter .15s; background: transparent;
|
||||
}
|
||||
.btn:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
|
||||
.btn-primary { background: var(--accent); color: var(--accent-ink); font-weight: 600; }
|
||||
.btn-primary:hover { filter: brightness(1.06); }
|
||||
.btn-ghost { border-color: var(--border-2); color: var(--text-2); }
|
||||
.btn-ghost:hover { color: var(--text); border-color: var(--text-3); }
|
||||
.btn-danger { background: var(--red-soft); color: var(--red); }
|
||||
.btn-block { width: 100%; }
|
||||
|
||||
.field { margin-bottom: 15px; }
|
||||
.field label { display: block; font-size: 12px; color: var(--text-2); margin-bottom: 7px; }
|
||||
.field input, .field select {
|
||||
width: 100%; padding: 10px 12px; font-size: 14px; color: var(--text);
|
||||
background: var(--bg-2); border: 1px solid var(--border); border-radius: var(--radius-sm); outline: none;
|
||||
}
|
||||
.field input:focus, .field select:focus { border-color: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
|
||||
|
||||
.chip {
|
||||
display: inline-block; padding: 3px 9px; border-radius: 20px; font-size: 11.5px;
|
||||
border: 1px solid var(--border-2); color: var(--text-2); margin: 1.5px 3px 1.5px 0;
|
||||
}
|
||||
.chip.amber { border-color: transparent; background: var(--accent-soft); color: var(--accent); }
|
||||
.badge { display: inline-block; padding: 2px 8px; border-radius: 5px; font-size: 11.5px; font-family: ui-monospace, Menlo, monospace; }
|
||||
.badge.green { background: var(--green-soft); color: var(--green); }
|
||||
.badge.red { background: var(--red-soft); color: var(--red); }
|
||||
.badge.amber { background: var(--accent-soft); color: var(--accent); }
|
||||
.badge.muted { background: var(--surface-2); color: var(--text-3); }
|
||||
|
||||
.panel { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius); }
|
||||
table { width: 100%; border-collapse: collapse; }
|
||||
th { text-align: left; font-size: 11.5px; font-weight: 500; color: var(--text-3); padding: 10px 18px; border-bottom: 1px solid var(--border); }
|
||||
td { padding: 13px 18px; border-bottom: 1px solid var(--border); font-size: 13.5px; }
|
||||
tr:last-child td { border-bottom: none; }
|
||||
tbody tr:hover { background: rgba(255,255,255,0.015); }
|
||||
.cell-id { color: var(--text-2); }
|
||||
.row-actions { display: flex; gap: 6px; }
|
||||
.row-actions .act { color: var(--blue); cursor: pointer; font-size: 13px; padding: 3px 6px; border-radius: 5px; }
|
||||
.row-actions .act:hover { background: rgba(110,168,255,0.1); }
|
||||
.row-actions .act.danger { color: var(--red); }
|
||||
.row-actions .act.danger:hover { background: var(--red-soft); }
|
||||
|
||||
.role { font-weight: 600; }
|
||||
.role.admin { color: var(--accent); }
|
||||
.role.researcher { color: var(--blue); }
|
||||
.role.guest { color: var(--text-2); }
|
||||
|
||||
.overlay { position: fixed; inset: 0; background: rgba(0,0,0,0.55); display: flex; align-items: center; justify-content: center; z-index: 50; }
|
||||
.modal { width: 460px; max-width: calc(100vw - 32px); background: var(--surface); border: 1px solid var(--border-2); border-radius: 14px; }
|
||||
.modal-head { padding: 18px 22px 0; font-size: 15px; font-weight: 600; }
|
||||
.modal-body { padding: 18px 22px; }
|
||||
.modal-foot { padding: 14px 22px 20px; display: flex; justify-content: flex-end; gap: 10px; border-top: 1px solid var(--border); }
|
||||
.checks { display: grid; grid-template-columns: 1fr 1fr; gap: 9px 14px; margin-top: 8px; }
|
||||
.check { display: flex; align-items: center; gap: 8px; font-size: 13px; color: var(--text-2); }
|
||||
.check input { accent-color: var(--accent); width: 15px; height: 15px; }
|
||||
.seg { display: flex; gap: 6px; }
|
||||
.seg button { background: transparent; border: 1px solid var(--border-2); color: var(--text-2); border-radius: 7px; padding: 7px 14px; font-size: 13px; }
|
||||
.seg button.on { background: var(--accent-soft); border-color: transparent; color: var(--accent); }
|
||||
@@ -1,53 +0,0 @@
|
||||
<script setup>
|
||||
import { ref } from 'vue';
|
||||
import { api, setToken } from '../api/admin.js';
|
||||
|
||||
const emit = defineEmits(['login']);
|
||||
const username = ref('');
|
||||
const password = ref('');
|
||||
const error = ref('');
|
||||
const busy = ref(false);
|
||||
|
||||
async function submit() {
|
||||
if (busy.value) return;
|
||||
if (!username.value || !password.value) { error.value = '请输入用户名和密码'; return; }
|
||||
busy.value = true;
|
||||
error.value = '';
|
||||
try {
|
||||
const r = await api.login(username.value, password.value);
|
||||
setToken(r.token);
|
||||
emit('login', r.user);
|
||||
} catch (e) {
|
||||
error.value = e.message || '登录失败';
|
||||
} finally {
|
||||
busy.value = false;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div class="login">
|
||||
<div class="card">
|
||||
<div class="brand">
|
||||
<div class="hex">蜂</div>
|
||||
<div><div class="brand-name">蜂群中枢</div><div class="brand-sub">PlantAI 管理后台</div></div>
|
||||
</div>
|
||||
<div class="field"><label>用户名</label><input v-model="username" autocomplete="username" @keyup.enter="submit"></div>
|
||||
<div class="field"><label>密码</label><input v-model="password" type="password" autocomplete="current-password" @keyup.enter="submit"></div>
|
||||
<button class="btn btn-primary btn-block" :disabled="busy" @click="submit">{{ busy ? '登录中…' : '登录' }}</button>
|
||||
<p v-if="error" class="err">{{ error }}</p>
|
||||
<div class="hint">仅限中枢管理员。登录与所有操作均会留痕。</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style scoped>
|
||||
.login { height: 100vh; display: flex; align-items: center; justify-content: center; background: radial-gradient(1200px 600px at 70% -10%, rgba(232, 163, 61, 0.08), transparent 60%), var(--bg); }
|
||||
.card { width: 360px; background: var(--surface); border: 1px solid var(--border); border-radius: 14px; padding: 34px 32px 30px; }
|
||||
.brand { display: flex; align-items: center; gap: 11px; margin-bottom: 26px; }
|
||||
.hex { width: 34px; height: 34px; flex: none; background: linear-gradient(135deg, #f2b95a, #d98f22); clip-path: polygon(50% 0, 100% 25%, 100% 75%, 50% 100%, 0 75%, 0 25%); display: grid; place-items: center; color: #1a1206; font-weight: 700; font-size: 14px; }
|
||||
.brand-name { font-size: 16px; font-weight: 600; }
|
||||
.brand-sub { color: var(--text-3); font-size: 12px; }
|
||||
.err { color: var(--red); font-size: 12px; margin: 10px 0 0; }
|
||||
.hint { margin-top: 20px; padding-top: 16px; border-top: 1px solid var(--border); font-size: 12px; color: var(--text-3); }
|
||||
</style>
|
||||
@@ -1,102 +0,0 @@
|
||||
<script setup>
|
||||
import { ref, onMounted } from 'vue';
|
||||
import { api } from '../api/admin.js';
|
||||
|
||||
const emit = defineEmits(['unauthorized']);
|
||||
|
||||
const from = ref('');
|
||||
const event = ref('');
|
||||
const q = ref('');
|
||||
const page = ref(1);
|
||||
const pageSize = 30;
|
||||
const items = ref([]);
|
||||
const total = ref(0);
|
||||
const aggregates = ref({});
|
||||
const loading = ref(false);
|
||||
|
||||
const EVENT_LABEL = {
|
||||
http_request: '调用', admin_login_ok: '登录成功', admin_login_fail: '登录失败',
|
||||
call_denied: '越权', user_created: '建用户', user_updated: '改用户', user_deleted: '删用户',
|
||||
};
|
||||
|
||||
function statusBadge(s) {
|
||||
if (!s) return 'muted';
|
||||
if (s >= 200 && s < 300) return 'green';
|
||||
if (s >= 500) return 'red';
|
||||
return 'amber';
|
||||
}
|
||||
|
||||
async function load() {
|
||||
loading.value = true;
|
||||
try {
|
||||
const r = await api.logs({
|
||||
from_date: from.value || undefined,
|
||||
event: event.value || undefined,
|
||||
q: q.value || undefined,
|
||||
page: page.value,
|
||||
page_size: pageSize,
|
||||
});
|
||||
items.value = r.items;
|
||||
total.value = r.total;
|
||||
aggregates.value = r.aggregates || {};
|
||||
} catch (e) {
|
||||
if (e.status === 401) emit('unauthorized');
|
||||
} finally {
|
||||
loading.value = false;
|
||||
}
|
||||
}
|
||||
|
||||
function search() { page.value = 1; load(); }
|
||||
function prev() { if (page.value > 1) { page.value -= 1; load(); } }
|
||||
function next() { if (page.value * pageSize < total.value) { page.value += 1; load(); } }
|
||||
|
||||
onMounted(load);
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div>
|
||||
<div class="stats" style="margin-bottom: 16px">
|
||||
<div class="stat"><div class="k">{{ total }}</div><div class="l">匹配日志</div></div>
|
||||
<div class="stat"><div class="k amber">{{ aggregates.denied ?? 0 }}</div><div class="l">越权 / 失败</div></div>
|
||||
<div class="stat"><div class="k">{{ aggregates.avg_latency_ms ?? '—' }}</div><div class="l">平均耗时 ms</div></div>
|
||||
<div class="stat"><div class="k">{{ aggregates.p95_latency_ms ?? '—' }}</div><div class="l">P95 耗时 ms</div></div>
|
||||
</div>
|
||||
<div class="toolbar">
|
||||
<input class="date" v-model="from" type="date">
|
||||
<select v-model="event">
|
||||
<option value="">全部事件</option>
|
||||
<option v-for="(l, k) in EVENT_LABEL" :key="k" :value="k">{{ l }}</option>
|
||||
</select>
|
||||
<div class="search"><input v-model="q" placeholder="按用户 / 路径过滤…" @keyup.enter="search"></div>
|
||||
<button class="btn btn-ghost" @click="search">查询</button>
|
||||
</div>
|
||||
<div class="panel">
|
||||
<table>
|
||||
<thead><tr><th>时间</th><th>用户</th><th>事件</th><th>方法</th><th>路径</th><th>状态</th><th>耗时</th></tr></thead>
|
||||
<tbody>
|
||||
<tr v-if="loading"><td colspan="7" class="cell-id">加载中…</td></tr>
|
||||
<tr v-else-if="!items.length"><td colspan="7" class="cell-id">暂无日志</td></tr>
|
||||
<tr v-for="(e, i) in items" :key="i">
|
||||
<td class="cell-id mono">{{ (e.ts || '').replace('T', ' ').slice(0, 19) }}</td>
|
||||
<td class="mono cell-id">{{ e.user_id ? e.user_id.slice(0, 8) : '—' }}</td>
|
||||
<td>{{ EVENT_LABEL[e.event] || e.event }}</td>
|
||||
<td class="cell-id mono">{{ e.method || '—' }}</td>
|
||||
<td class="mono">{{ e.path || '—' }}</td>
|
||||
<td><span v-if="e.status" class="badge" :class="statusBadge(e.status)">{{ e.status }}</span><span v-else class="badge muted">—</span></td>
|
||||
<td class="cell-id mono">{{ e.latency_ms != null ? e.latency_ms + 'ms' : '—' }}</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<div class="pag">
|
||||
<span>共 {{ total }} 条</span><span class="spacer"></span>
|
||||
<button :disabled="page <= 1" @click="prev">上一页</button>
|
||||
<button class="on">{{ page }}</button>
|
||||
<button :disabled="page * pageSize >= total" @click="next">下一页</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style scoped>
|
||||
input.date { background: var(--surface); border: 1px solid var(--border); color: var(--text); border-radius: var(--radius-sm); padding: 8px 10px; font-size: 13px; outline: none; }
|
||||
</style>
|
||||
@@ -1,72 +0,0 @@
|
||||
<script setup>
|
||||
import { ref, onMounted } from 'vue';
|
||||
import { api } from '../api/admin.js';
|
||||
import UserForm from '../components/UserForm.vue';
|
||||
|
||||
const emit = defineEmits(['unauthorized']);
|
||||
|
||||
const users = ref([]);
|
||||
const loading = ref(true);
|
||||
const formOpen = ref(false);
|
||||
const formMode = ref('create');
|
||||
const editing = ref(null);
|
||||
|
||||
const ROLE_LABEL = { admin: '管理员', researcher: '研究员', guest: '访客' };
|
||||
const UNIT_LABEL = { qa: '问答', kg: '知识图谱', extraction: '文献抽取', docgen: '文档生成', figures: '图片图库', private_docs: '私有资料', bioflow: '生信分析', paper: '论文撰写', legacy: '旧版接口' };
|
||||
|
||||
async function load() {
|
||||
loading.value = true;
|
||||
try {
|
||||
const r = await api.users();
|
||||
users.value = r.users;
|
||||
} catch (e) {
|
||||
if (e.status === 401) emit('unauthorized');
|
||||
} finally {
|
||||
loading.value = false;
|
||||
}
|
||||
}
|
||||
|
||||
function create() { formMode.value = 'create'; editing.value = null; formOpen.value = true; }
|
||||
function edit(u) { formMode.value = 'edit'; editing.value = u; formOpen.value = true; }
|
||||
async function del(u) {
|
||||
if (!confirm(`删除用户「${u.username}」?此操作不可撤销。`)) return;
|
||||
try { await api.deleteUser(u.id); load(); } catch (e) { alert(e.message || '删除失败'); }
|
||||
}
|
||||
function onSaved() { formOpen.value = false; load(); }
|
||||
|
||||
onMounted(load);
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div>
|
||||
<div class="toolbar">
|
||||
<div class="spacer"></div>
|
||||
<button class="btn btn-primary" @click="create">+ 新建用户</button>
|
||||
</div>
|
||||
<div class="panel">
|
||||
<table>
|
||||
<thead><tr><th>用户</th><th>角色</th><th>授权范围</th><th>创建时间</th><th style="width: 120px"></th></tr></thead>
|
||||
<tbody>
|
||||
<tr v-if="loading"><td colspan="5" class="cell-id">加载中…</td></tr>
|
||||
<tr v-else-if="!users.length"><td colspan="5" class="cell-id">暂无用户</td></tr>
|
||||
<tr v-for="u in users" :key="u.id">
|
||||
<td>
|
||||
<div style="display: flex; align-items: center; gap: 10px">
|
||||
<div class="avatar">{{ u.username[0] }}</div>
|
||||
<div><div>{{ u.username }}</div><div class="cell-id" style="font-size: 12px">{{ u.id.slice(0, 8) }}</div></div>
|
||||
</div>
|
||||
</td>
|
||||
<td><span class="role" :class="u.role">{{ ROLE_LABEL[u.role] || u.role }}</span></td>
|
||||
<td>
|
||||
<span v-if="u.role === 'admin'" class="chip amber">全部权限</span>
|
||||
<template v-else><span v-for="a in u.authorizations" :key="a" class="chip">{{ UNIT_LABEL[a] || a }}</span></template>
|
||||
</td>
|
||||
<td class="cell-id mono">{{ (u.created_at || '').slice(0, 10) }}</td>
|
||||
<td><div class="row-actions"><span class="act" @click="edit(u)">编辑</span><span class="act danger" @click="del(u)">删除</span></div></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<UserForm :open="formOpen" :mode="formMode" :user="editing" @close="formOpen = false" @saved="onSaved" />
|
||||
</div>
|
||||
</template>
|
||||
@@ -1,15 +0,0 @@
|
||||
import { defineConfig } from 'vite';
|
||||
import vue from '@vitejs/plugin-vue';
|
||||
|
||||
export default defineConfig({
|
||||
base: '/admin/',
|
||||
plugins: [vue()],
|
||||
server: {
|
||||
host: '127.0.0.1',
|
||||
port: 5181,
|
||||
proxy: {
|
||||
'/api': { target: 'http://127.0.0.1:8090', changeOrigin: false },
|
||||
},
|
||||
},
|
||||
build: { outDir: 'dist', target: 'es2020' },
|
||||
});
|
||||
@@ -16,3 +16,70 @@ capabilities:
|
||||
endpoint: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi
|
||||
method: GET
|
||||
timeout: 8
|
||||
|
||||
# ---- 生信能力单元(S1):两个 MCP 端点,各自独立装卸 ----
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 基因家族分析(路径规划/建任务/参数确认/文件获取与预处理/状态查询),自然语言触发
|
||||
session_default: true
|
||||
nl_triggers:
|
||||
- 基因家族
|
||||
- 家族成员
|
||||
- 亚家族
|
||||
- 系统发育
|
||||
- 建树
|
||||
- 进化树
|
||||
- 保守基序
|
||||
- motif
|
||||
- 结构域
|
||||
- 基因结构
|
||||
- 共线性
|
||||
- Ka/Ks
|
||||
- 选择压力
|
||||
- 组合证据
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/gene-family/
|
||||
prompt_pack:
|
||||
file: bioflow_gene_family_agent_rules.md
|
||||
|
||||
- name: bioflow-rnaseq
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 转录组分析(实验设计确认/样本配置/路径规划/建任务/差异表达/富集),自然语言触发
|
||||
session_default: true
|
||||
nl_triggers:
|
||||
- 转录组
|
||||
- RNA-seq
|
||||
- RNAseq
|
||||
- rnaseq
|
||||
- 差异表达
|
||||
- 表达定量
|
||||
- 富集
|
||||
- WGCNA
|
||||
- GSEA
|
||||
- 实验设计
|
||||
- fastq
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/rnaseq/
|
||||
prompt_pack:
|
||||
file: bioflow_rnaseq_agent_rules.md
|
||||
|
||||
# ---- 生信能力单元(S4):第三个 MCP 端点,用户自定义扩展分析 ----
|
||||
- name: bioflow-extension
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 用户自定义扩展分析(创建/转换/试运行/校验/发布/运行),自然语言触发
|
||||
session_default: true
|
||||
nl_triggers:
|
||||
- 扩展
|
||||
- 拓展分析
|
||||
- 扩展分析
|
||||
- 自定义分析
|
||||
- 分析模板
|
||||
- 分析脚本
|
||||
- 分析工具
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/extension/
|
||||
prompt_pack:
|
||||
file: bioflow_extension_agent_rules.md
|
||||
|
||||
Vendored
+3
@@ -31,3 +31,6 @@ QA_V1_WEB_SEARCH_MAX_RESULTS=4
|
||||
QA_V1_WEB_SEARCH_TIMEOUT=8
|
||||
QA_V1_WEB_SEARCH_MAX_CHARS=1200
|
||||
QA_V1_ENABLE_SUPERPOWERS=1
|
||||
|
||||
# 生信对话式能力熔断开关(S1):默认关(=0)。置 1 才启用 QA 前门 bioflow_agent。
|
||||
QA_V1_ENABLE_BIOFLOW_AGENT=0
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# 用户自定义扩展分析对话规范(bioflow_extension_agent_rules)
|
||||
|
||||
你是用户自定义扩展分析助手。引导用户把一段可运行的分析脚本/工具,通过「创建 → AI 转换 → 试运行 → 校验 → 确认 → 发布 → 运行」流程,做成 PlantAI 可自然语言触发的自定义扩展。
|
||||
|
||||
## 对话节奏(自然语言主导 + 卡片辅助)
|
||||
每次需要用户输入/选择时,遵循固定节奏:
|
||||
1. 先用自然语言说明并列表展示需要什么;
|
||||
2. 提示用户可用卡片快捷操作,也可直接打字(两条通道语义一致);
|
||||
3. 用户回答后,用自然语言复述,确认无误再继续。
|
||||
|
||||
## 创建(create_extension)
|
||||
- 需要用户提供:显示名、运行命令、代码文件(可上传/粘贴文本内容)、运行时镜像、可选测试数据/金标。
|
||||
- 代码文件以「路径 + 文本内容」提交;引导用户说明脚本是做什么的、依赖什么。
|
||||
|
||||
## AI 转换与校验(build → trial_run → validate)
|
||||
- 创建后调用 build 做 AI 转换(原始命令 → command_template + 参数 + 依赖),失败时规则版兜底。
|
||||
- 转换结果先向用户说明(识别到的参数/依赖/输入角色),请用户确认参数名与默认值。
|
||||
- 试运行/校验的结果要转述给用户,失败时说明原因并引导修复(repair 循环)。
|
||||
|
||||
## 确认与发布(confirm_extension → publish_extension)
|
||||
- 确认前请用户确认触发词(nl_triggers)与参数——触发词决定用户以后说什么话能唤起这个扩展。
|
||||
- 发布后扩展才可被自然语言触发运行;发布是不可逆的对外可用状态,须经用户确认。
|
||||
|
||||
## 运行(run_extension)
|
||||
- 仅已发布扩展可运行;运行参数须在白名单内(参数 schema),未知参数明确追问。
|
||||
- 运行结果(run_result)转述给用户。
|
||||
|
||||
## 其它
|
||||
- 工具名/参数名/内部实现一律不外泄;
|
||||
- 状态不确定时主动查 extension_status 并向用户转述当前阶段与下一步;
|
||||
- 人工确认边界(确认触发词、发布)不可跳过、不可代用户拍板。
|
||||
@@ -0,0 +1,32 @@
|
||||
# 基因家族分析对话规范(bioflow_gene_family_agent_rules)
|
||||
|
||||
你是基因家族分析助手。用自然语言引导用户从需求到分析完成,全程对话式交互。
|
||||
|
||||
## 人工确认边界(不可跳过、不可代确认)
|
||||
- **subfamily_confirm(亚家族预测人工确认)是硬性人工边界**:系统预测的亚家族划分结果,
|
||||
必须停下来请用户查看并确认或修改。你绝不能代用户确认,也不能在用户未确认前继续下游分析。
|
||||
- 用户提出修改(如"把某成员从亚家族 A 移到亚家族 B")时,先复述修改内容,再请用户做最终确认。
|
||||
|
||||
## 对话节奏(自然语言主导 + 卡片辅助)
|
||||
每次需要用户输入/选择时,遵循固定节奏:
|
||||
1. 先用自然语言说明并**列表展示**选项及含义;
|
||||
2. 提示用户可用卡片快捷操作,也可直接打字回复(两条通道语义一致);
|
||||
3. 用户选择后,用自然语言**复述**其选择,确认无误再继续。
|
||||
|
||||
## 路径规划与建任务
|
||||
- 规划后列出"核心流程(必选)+ 可选分析"候选路径,说明每个可选分析的生物学意义;
|
||||
- **建任务前必须经用户确认阶段链**,收到确认后再创建任务,绝不擅自建任务。
|
||||
|
||||
## 文件获取与预处理
|
||||
- 源文件支持上传、http/https 链接、OSS 地址、局域网机器路径;
|
||||
- 文件到达后先探测格式,**格式异常(如 Excel/文本)时先向用户说明列结构并提问确认**
|
||||
(哪列是序列、是否去重、是否去表头等),再由模型整理成 fasta/gff 等标准格式;
|
||||
- 转换结果过格式验证,失败时说明问题并追问修正(循环不超过 3 次后建议人工整理)。
|
||||
|
||||
## 参数确认
|
||||
- 列"参数名(默认值,含义)"请用户确认或修改;未识别参数时明确指出并给出允许范围。
|
||||
|
||||
## 其它
|
||||
- 工具名、参数名、内部实现一律不外泄;
|
||||
- 任务运行中用户提问(流程/参数/生物学意义)随时回答,不打断任务;
|
||||
- 卡住或不确定时主动查询任务状态,向用户转述当前进度与下一步。
|
||||
@@ -0,0 +1,29 @@
|
||||
# 转录组分析对话规范(bioflow_rnaseq_agent_rules)
|
||||
|
||||
你是转录组分析助手。用自然语言引导用户从实验设计确认到差异表达/富集分析完成。
|
||||
|
||||
## 人工确认边界(不可跳过、不可代确认)
|
||||
- **实验设计确认(rnaseq_de 之前的必确认信息)是硬性边界**:开始分析前必须与用户确认——
|
||||
① 物种与参考基因组版本;② 每个样本的 FASTQ(上传/链接/局域网路径)及单端双端;
|
||||
③ 对照组/处理组各几个重复、有无时间点/批次;④ 建库是否链特异性。
|
||||
- 统计模型正确性依赖上述信息,未确认前不得创建差异表达任务,也不能代用户拍板基线/比较方向。
|
||||
|
||||
## 对话节奏(自然语言主导 + 卡片辅助)
|
||||
每次需要用户输入/选择时,遵循固定节奏:
|
||||
1. 先用自然语言说明并**列表展示**必确认信息与可选分析;
|
||||
2. 提示用户可用卡片快捷操作,也可直接打字回复(两条通道语义一致);
|
||||
3. 用户回答后,用自然语言**复述**其样本设计与选择(如"干旱 3 vs 对照 3,双端,非链特异,对照为基线"),
|
||||
确认无误再继续。
|
||||
|
||||
## 样本配置与文件预处理
|
||||
- 用户用自然语言描述样本时,可解析生成 samples.yaml(样本名/对照处理组/重复/单双端);
|
||||
- FASTQ 支持上传、http/https 链接、OSS 地址、局域网路径;核对成对完整;
|
||||
- 非 FASTQ 的输入文件(表格/文本等)先探测格式,**异常时先说明列结构并提问确认**,再整理为可用格式。
|
||||
|
||||
## 参数确认
|
||||
- 列"参数名(默认值,含义)"请用户确认或修改;未识别参数时明确指出并给出允许范围。
|
||||
|
||||
## 其它
|
||||
- 工具名、参数名、内部实现一律不外泄;
|
||||
- 任务运行中用户提问(流程/参数/实验设计意义)随时回答,不打断任务;
|
||||
- 卡住或不确定时主动查询任务状态,向用户转述当前进度与下一步(含样本质控 PCA、差异基因数目等结果转述)。
|
||||
@@ -13,6 +13,7 @@ from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from . import __version__
|
||||
from .bioflow_agent import try_bioflow_agent
|
||||
from .config import QAConfig
|
||||
from .pipeline import QAAnswerError, QAV1Pipeline
|
||||
|
||||
@@ -47,6 +48,9 @@ class QARequest(BaseModel):
|
||||
# ---- Capability Gateway(S1):纯数据字段,无 validator(Cython 编译安全)----
|
||||
history: List[Dict[str, str]] = Field(default_factory=list)
|
||||
capabilities: Dict[str, Any] = Field(default_factory=dict)
|
||||
# ---- Bioflow 对话式编排(S2):卡片回填协议 + session↔job 绑定 ----
|
||||
card_result: Optional[Dict[str, Any]] = None
|
||||
bioflow: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
@@ -150,6 +154,18 @@ async def chat(request: QARequest) -> Dict[str, Any]:
|
||||
detail={"error_code": "llm_unavailable", "answer": "", "message": "Final-answer LLM is not configured"},
|
||||
)
|
||||
try:
|
||||
# ---- Bioflow Agent 前门(S1):命中则短路返回,否则走原 pipeline ----
|
||||
bioflow_result = await asyncio.to_thread(
|
||||
try_bioflow_agent,
|
||||
request.question,
|
||||
llm_client=pipeline.llm,
|
||||
history=request.history,
|
||||
capabilities=request.capabilities,
|
||||
card_result=request.card_result,
|
||||
bioflow=request.bioflow,
|
||||
)
|
||||
if bioflow_result is not None:
|
||||
return bioflow_result
|
||||
ablation = dict(request.ablation or {})
|
||||
ablation["use_final_llm"] = True
|
||||
return await asyncio.to_thread(
|
||||
@@ -216,6 +232,19 @@ async def chat_stream(request: QARequest) -> StreamingResponse:
|
||||
|
||||
def run() -> None:
|
||||
try:
|
||||
# ---- Bioflow Agent 前门(S1):命中则短路,否则走原 pipeline ----
|
||||
bioflow_result = try_bioflow_agent(
|
||||
request.question,
|
||||
llm_client=pipeline.llm,
|
||||
history=request.history,
|
||||
capabilities=request.capabilities,
|
||||
card_result=request.card_result,
|
||||
bioflow=request.bioflow,
|
||||
progress_callback=emit,
|
||||
)
|
||||
if bioflow_result is not None:
|
||||
events.put(("complete", bioflow_result))
|
||||
return
|
||||
ablation = dict(request.ablation or {})
|
||||
ablation["use_final_llm"] = True
|
||||
result = pipeline.answer(
|
||||
|
||||
@@ -0,0 +1,764 @@
|
||||
from __future__ import annotations
|
||||
|
||||
"""bioflow_agent.py —— 生信 MCP 能力前门路由(S1 + S2)。
|
||||
|
||||
S1:三重门(会话开关/注册表/意图预筛)→ ToolRunner 工具循环 → 自然语言转述。
|
||||
S2:对话式编排闭环——card_result 卡片回填协议 + session↔job 绑定 + 多轮记忆
|
||||
+ 参数抽取闭环 + job_status 主动查询转述。
|
||||
|
||||
card_result 协议(前端卡片操作 → 结构化用户消息 → QA 流):
|
||||
{card: path_selection|form_confirm|file_upload|archive_choice, payload: {...}}
|
||||
BioflowAgent 读取后经工具执行,再自然语言复述;卡片与自然语言两条通道在
|
||||
服务端语义一致(QA 对话状态机是唯一决策点)。
|
||||
|
||||
session↔job 绑定:BFF sessions.meta.bioflow = {active_job_id, chain},经请求
|
||||
body["bioflow"] 注入(含 session_id);job_status 工具是任务真相唯一权威。
|
||||
|
||||
金标路径零改动:任何不命中/异常都返回 None 回退原 pipeline。
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from .capabilities.manifests import CapabilityManifest
|
||||
from .capabilities.providers import PromptPackProvider, get_dispatcher
|
||||
from .capabilities.registry import get_registry
|
||||
from .tool_runner import ToolRunner, make_llm_complete
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
# 三个生信能力单元(与 bioflow_service 三个 MCP 端点一一对应)
|
||||
BIOFLOW_CAPABILITY_NAMES = ("bioflow-gene-family", "bioflow-rnaseq", "bioflow-extension")
|
||||
ENV_FUSE = "QA_V1_ENABLE_BIOFLOW_AGENT"
|
||||
MAX_ROUNDS = 5
|
||||
|
||||
# card_result 卡片类型(S2 前端卡片回填协议)
|
||||
CARD_PATH_SELECTION = "path_selection"
|
||||
CARD_FORM_CONFIRM = "form_confirm"
|
||||
CARD_FILE_UPLOAD = "file_upload"
|
||||
CARD_ARCHIVE_CHOICE = "archive_choice"
|
||||
CARD_EXTENSION_ACTION = "extension_action" # S4:扩展能力单元操作(build/trial/validate/confirm/publish/run)
|
||||
CARD_TYPES = (CARD_PATH_SELECTION, CARD_FORM_CONFIRM, CARD_FILE_UPLOAD, CARD_ARCHIVE_CHOICE, CARD_EXTENSION_ACTION)
|
||||
|
||||
# chain 名 → 能力名(建任务时选择哪条链的 create 工具)
|
||||
_CHAIN_TO_CAPABILITY = {
|
||||
"gene_family": "bioflow-gene-family",
|
||||
"rnaseq": "bioflow-rnaseq",
|
||||
}
|
||||
|
||||
# 能力名 → chain 名(反向映射,供规划阶段从命中能力回填 chain,续接多轮对话)
|
||||
_CAPABILITY_TO_CHAIN = {v: k for k, v in _CHAIN_TO_CAPABILITY.items()}
|
||||
|
||||
|
||||
def _chain_to_capability(chain: str) -> Optional[str]:
|
||||
"""把 chain 名(gene_family/rnaseq)映射为能力名;未知返回 None。"""
|
||||
return _CHAIN_TO_CAPABILITY.get(str(chain or "").strip().lower())
|
||||
|
||||
_SYSTEM_BASE = (
|
||||
"你是 PlantAI 生信分析助手,用自然语言引导用户完成基因家族或转录组分析。\n"
|
||||
"你可以调用生信工具完成规划、建任务、确认参数、登记文件、查状态等操作。\n"
|
||||
"硬性规则:\n"
|
||||
"1. 回复只使用自然语言,绝不向用户泄露工具名、参数名或内部实现细节。\n"
|
||||
"2. 需要用户提供文件/参数/做选择时,先自然语言说明并列表展示选项与含义,再请用户回答,"
|
||||
"绝不擅自替用户做不可逆决策。\n"
|
||||
"3. 人工确认边界(基因家族 subfamily_confirm、转录组实验设计确认)不可跳过、不可代用户确认。\n"
|
||||
"4. 任务运行中可随时回答流程/参数/生物学意义类问题;卡住或不确定时主动查询任务状态,"
|
||||
"再向用户转述当前进度与下一步(等待文件、等待参数确认、失败可重跑等)。\n"
|
||||
"5. 文件格式异常时先向用户说明并提问确认,再由用户决定如何处理。\n"
|
||||
"6. 用户用自然语言给出参数(如「用 8 核、padj 用 0.01」)时,抽取为结构化参数后经确认步骤提交;"
|
||||
"只提交白名单内的参数名,未知参数明确指出并追问(最多 3 次)。\n"
|
||||
"7. 若用户消息是对上一轮卡片/提问的确认或选择,先复述用户的选择,确认无误再执行对应动作。\n"
|
||||
)
|
||||
|
||||
|
||||
def _env_enabled() -> bool:
|
||||
"""门 1:QA_V1_ENABLE_BIOFLOW_AGENT 熔断开关。默认关(未设置 = 关)。"""
|
||||
raw = os.environ.get(ENV_FUSE)
|
||||
return raw is not None and str(raw).strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _intent_match(question: str, triggers: List[str]) -> bool:
|
||||
"""门 3:意图预筛——问句是否命中任一触发词(大小写不敏感子串匹配)。"""
|
||||
if not triggers:
|
||||
return False
|
||||
q = (question or "").lower()
|
||||
return any(t and t.lower() in q for t in triggers)
|
||||
|
||||
|
||||
def _registered_capabilities(capabilities: Optional[Dict[str, Any]]) -> List[CapabilityManifest]:
|
||||
"""注册表 + 会话开关:返回所有已注册且会话开启的生信能力 manifest(不判意图)。"""
|
||||
try:
|
||||
registry = get_registry()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: registry 访问失败,回退原 pipeline: %s", exc)
|
||||
return []
|
||||
session_caps = capabilities or {}
|
||||
out: List[CapabilityManifest] = []
|
||||
for name in BIOFLOW_CAPABILITY_NAMES:
|
||||
try:
|
||||
spec = registry.get(name)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: registry.get(%s) 失败: %s", name, exc)
|
||||
continue
|
||||
if spec is None:
|
||||
continue # 未注册
|
||||
manifest = spec.manifest
|
||||
if not bool(session_caps.get(name, manifest.session_default)):
|
||||
continue # 会话开关关
|
||||
out.append(manifest)
|
||||
return out
|
||||
|
||||
|
||||
def _candidate_capabilities(
|
||||
question: str,
|
||||
capabilities: Optional[Dict[str, Any]],
|
||||
) -> List[CapabilityManifest]:
|
||||
"""门 2 + 门 3:注册表 + 会话开关 + 意图预筛,返回命中的候选能力清单。"""
|
||||
return [
|
||||
m for m in _registered_capabilities(capabilities)
|
||||
if _intent_match(question, m.nl_triggers)
|
||||
]
|
||||
|
||||
|
||||
def _build_tools(
|
||||
manifests: List[CapabilityManifest],
|
||||
) -> Tuple[List[Dict[str, Any]], Dict[str, CapabilityManifest]]:
|
||||
"""经 MCP list_tools 发现各能力工具,构造 LLM 工具表 + ToolRunner manifests。
|
||||
|
||||
返回 (tools, tool_manifests):
|
||||
- tools: OpenAI function schema 列表(真实 server 工具名/描述/入参 schema)
|
||||
- tool_manifests: {真实工具名: manifest},manifest.name=能力名(共享连接),
|
||||
mcp.tool=真实工具名
|
||||
|
||||
多能力同时命中时:专有工具名本就不同,各自保留;共享工具(两端点语义一致)
|
||||
按名去重,只保留首个能力的一份,避免 LLM tools 重名 + 同名覆盖。
|
||||
"""
|
||||
dispatcher = get_dispatcher()
|
||||
tools: List[Dict[str, Any]] = []
|
||||
tool_manifests: Dict[str, CapabilityManifest] = {}
|
||||
for cap_manifest in manifests:
|
||||
url = str((cap_manifest.config.get("mcp") or {}).get("url") or "").strip()
|
||||
if not url:
|
||||
LOGGER.warning("bioflow agent: 能力 %s 缺 mcp.url,跳过", cap_manifest.name)
|
||||
continue
|
||||
try:
|
||||
discovered = dispatcher.list_mcp_tools(cap_manifest)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: 能力 %s MCP 工具发现失败: %s", cap_manifest.name, exc)
|
||||
continue
|
||||
for item in discovered:
|
||||
tool_name = str(item.get("name") or "").strip()
|
||||
if not tool_name:
|
||||
continue
|
||||
if tool_name in tool_manifests:
|
||||
# 共享工具去重(见 S1 修复说明)
|
||||
continue
|
||||
input_schema = item.get("input_schema") or {}
|
||||
if not isinstance(input_schema, dict):
|
||||
input_schema = {"type": "object", "properties": {}}
|
||||
tools.append(
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": tool_name,
|
||||
"description": str(item.get("description") or "")[:1000],
|
||||
"parameters": input_schema,
|
||||
},
|
||||
}
|
||||
)
|
||||
tool_manifests[tool_name] = CapabilityManifest(
|
||||
name=cap_manifest.name,
|
||||
provider="mcp",
|
||||
permission="read_only",
|
||||
description=str(item.get("description") or ""),
|
||||
config={"mcp": {"url": url, "tool": tool_name}},
|
||||
)
|
||||
return tools, tool_manifests
|
||||
|
||||
|
||||
def _build_system(manifests: List[CapabilityManifest]) -> str:
|
||||
"""基础系统提示 + 各能力对话规范(prompt_pack 注入)。"""
|
||||
qa_root = Path(os.environ.get("GENE_FAMILY_ROOT", "/data/gene-family-mas-v2"))
|
||||
parts = [_SYSTEM_BASE]
|
||||
seen: set[str] = set()
|
||||
for manifest in manifests:
|
||||
file_name = str((manifest.config.get("prompt_pack") or {}).get("file") or "").strip()
|
||||
if not file_name or file_name in seen:
|
||||
continue
|
||||
seen.add(file_name)
|
||||
try:
|
||||
text = PromptPackProvider.load(manifest, qa_root)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: prompt_pack 加载失败 %s: %s", file_name, exc)
|
||||
text = ""
|
||||
if text:
|
||||
parts.append(text)
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# S2 工具执行 / 转述 / 参数抽取 辅助
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _execute_tool(
|
||||
tool_manifests: Dict[str, CapabilityManifest], tool_name: str, arguments: Dict[str, Any]
|
||||
) -> str:
|
||||
"""执行单个工具,返回结果文本(异常转文本,不炸主流程)。"""
|
||||
manifest = tool_manifests.get(tool_name)
|
||||
if manifest is None:
|
||||
return f"工具 {tool_name} 未注册"
|
||||
try:
|
||||
return get_dispatcher().execute(manifest, arguments)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return f"工具执行失败: {type(exc).__name__}: {exc}"
|
||||
|
||||
|
||||
def _paraphrase(llm_client: Any, user_text: str, system: Optional[str] = None) -> str:
|
||||
"""把工具执行结果转述为自然语言(LLM 优先,失败回退原文,绝不抛异常)。"""
|
||||
system = system or (
|
||||
"你是 PlantAI 生信分析助手。把下面的工具执行结果转述为对用户的自然语言回复,"
|
||||
"绝不提及工具名/参数名/内部实现;需要用户下一步操作时明确说明。"
|
||||
)
|
||||
try:
|
||||
text = llm_client.complete(system, user_text, temperature=0.2, max_tokens=600)
|
||||
if text and str(text).strip():
|
||||
return str(text).strip()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: paraphrase LLM 失败,回退原文: %s", exc)
|
||||
return str(user_text or "").strip()
|
||||
|
||||
|
||||
def _extract_params(
|
||||
llm_client: Any, user_text: str, form_defaults: Dict[str, Any]
|
||||
) -> Tuple[Dict[str, Any], List[str]]:
|
||||
"""LLM 抽取用户自然语言中的阶段确认参数 → (已识别参数, 未识别参数列表)。
|
||||
|
||||
白名单 = form_defaults 的键;未识别参数(不在白名单)单独返回供追问(≤3 次)。
|
||||
参照 extension_llm 的「LLM 出 JSON → 白名单净化」模式,LLM 失败/坏 JSON 返回空。
|
||||
"""
|
||||
whitelist = {str(k) for k in (form_defaults or {}).keys()}
|
||||
if not whitelist:
|
||||
return {}, []
|
||||
system = (
|
||||
"你是参数抽取助手。从用户的话中抽取阶段确认参数,只输出 JSON。\n"
|
||||
'输出形态:{"params": {"<参数名>": <值>}, "unknown": ["<未识别参数名>"]}。\n'
|
||||
f"允许的参数名(白名单):{', '.join(sorted(whitelist))}。\n"
|
||||
"只抽取用户明确提到的参数;不在白名单里的参数名放进 unknown。"
|
||||
)
|
||||
try:
|
||||
raw = llm_client.complete_json(system, user_text, max_tokens=500)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: 参数抽取 LLM 失败: %s", exc)
|
||||
return {}, []
|
||||
if not isinstance(raw, dict):
|
||||
return {}, []
|
||||
params = raw.get("params") if isinstance(raw.get("params"), dict) else {}
|
||||
known = {str(k): v for k, v in params.items() if str(k) in whitelist}
|
||||
unknown_raw = raw.get("unknown") or []
|
||||
if isinstance(unknown_raw, str):
|
||||
unknown_raw = [unknown_raw]
|
||||
unknown = [str(x).strip() for x in unknown_raw if isinstance(x, str) and str(x).strip()]
|
||||
return known, unknown
|
||||
|
||||
|
||||
def _try_parse_json(text: str) -> Optional[Any]:
|
||||
if not text:
|
||||
return None
|
||||
try:
|
||||
return json.loads(text)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _extract_job_id(result_text: str) -> Optional[str]:
|
||||
"""从 create_job 工具结果文本里提取 job_id(结果形如 {"ok":true,"data":{"id":...}})。"""
|
||||
data = _try_parse_json(result_text)
|
||||
if isinstance(data, dict):
|
||||
inner = data.get("data") if isinstance(data.get("data"), dict) else data
|
||||
jid = inner.get("id") or inner.get("job_id")
|
||||
if jid:
|
||||
return str(jid)
|
||||
return None
|
||||
|
||||
|
||||
def _query_job_status(job_id: str, tool_manifests: Dict[str, CapabilityManifest], llm_client: Any) -> str:
|
||||
"""主动查 job_status 并转述(任务真相唯一权威)。"""
|
||||
result = _execute_tool(tool_manifests, "job_status", {"job_id": job_id})
|
||||
return _paraphrase(llm_client, f"查询任务状态的结果:\n{result}")
|
||||
|
||||
|
||||
def _empty_audit() -> Dict[str, Any]:
|
||||
return {"rounds": 0, "tool_calls": 0, "used_json_fallback": False, "tool_names": [], "errors": []}
|
||||
|
||||
|
||||
def _map_readiness(data: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""check_readiness 结果 → 就绪卡字段 {ready, quadrants:[{name,ok}]}。"""
|
||||
inner = data.get("result", data) if isinstance(data, dict) else {}
|
||||
quadrants = []
|
||||
for key in ("environment", "reference", "space", "budget"):
|
||||
q = inner.get(key) or {}
|
||||
quadrants.append({"name": key, "ok": bool(q.get("ok"))})
|
||||
return {"ready": bool(inner.get("ready")), "quadrants": quadrants}
|
||||
|
||||
|
||||
def _map_versioning(data: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""propose_change 结果 → 版本化卡字段 {plan_revision, run_id, affected_stages, based_on_revision}。"""
|
||||
inner = data.get("result", data) if isinstance(data, dict) else {}
|
||||
affected = inner.get("affected") or []
|
||||
return {
|
||||
"plan_revision": "",
|
||||
"run_id": "",
|
||||
"affected_stages": list(affected),
|
||||
"based_on_revision": None,
|
||||
}
|
||||
|
||||
|
||||
def _make_payload(
|
||||
question: str,
|
||||
answer: str,
|
||||
*,
|
||||
audit: Optional[Dict[str, Any]] = None,
|
||||
capabilities: Optional[List[str]] = None,
|
||||
active_job_id: Optional[str] = None,
|
||||
chain: Optional[str] = None,
|
||||
readiness: Optional[Dict[str, Any]] = None,
|
||||
versioning: Optional[Dict[str, Any]] = None,
|
||||
plan: Optional[Dict[str, Any]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
payload: Dict[str, Any] = {
|
||||
"success": True,
|
||||
"version": "bioflow-agent-s2",
|
||||
"question": question,
|
||||
"answer": answer,
|
||||
"sources": [],
|
||||
"source_count": 0,
|
||||
"citation_validation": {"valid": True, "cited_pmids": [], "unknown_pmids": []},
|
||||
"bioflow": {
|
||||
"handled": True,
|
||||
"audit": audit or _empty_audit(),
|
||||
"capabilities": capabilities or [],
|
||||
},
|
||||
}
|
||||
if active_job_id:
|
||||
payload["bioflow"]["active_job_id"] = str(active_job_id)
|
||||
if chain:
|
||||
payload["bioflow"]["chain"] = str(chain)
|
||||
if readiness:
|
||||
payload["bioflow"]["readiness"] = readiness
|
||||
if versioning:
|
||||
payload["bioflow"]["versioning"] = versioning
|
||||
if plan:
|
||||
payload["bioflow"]["plan"] = plan
|
||||
return payload
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# card_result 回填处理(S2)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _card_path_selection(
|
||||
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""路径卡确认 → create_*_job → 复述;带 reuse_archive 时先复用存档的阶段链/参数再建任务。"""
|
||||
chain = str(payload.get("chain") or (bioflow or {}).get("chain") or "").strip()
|
||||
stages = payload.get("stages")
|
||||
reuse_name = str(payload.get("reuse_archive") or payload.get("reuse_name") or "").strip()
|
||||
has_stages = isinstance(stages, list) and len(stages) > 0
|
||||
if not has_stages and not reuse_name:
|
||||
# 既无显式阶段链也无存档复用来源 → 无法建任务,回退原 pipeline
|
||||
return None
|
||||
cap_name = _chain_to_capability(chain)
|
||||
if cap_name is None:
|
||||
# 无法确定能力(chain 空/未知):不能默认走某条链,回退原 pipeline
|
||||
return None
|
||||
tool = "create_gene_family_job" if cap_name == "bioflow-gene-family" else "create_rnaseq_job"
|
||||
if tool not in tool_manifests:
|
||||
return None
|
||||
args: Dict[str, Any] = {
|
||||
"session_id": str((bioflow or {}).get("session_id") or ""),
|
||||
"stages": [str(s) for s in stages] if isinstance(stages, list) and stages else [],
|
||||
"title": str(payload.get("title") or ""),
|
||||
"auto_continue": bool(payload.get("auto_continue", False)),
|
||||
}
|
||||
tool_names: List[str] = []
|
||||
# 存档复用(FR-BIO-023):先 reuse_archive 取阶段链/参数/archive_id,再以其发起
|
||||
if reuse_name and "reuse_archive" in tool_manifests:
|
||||
tool_names.append("reuse_archive") # 实际先执行 reuse_archive(audit 顺序与实际一致)
|
||||
reuse_text = _execute_tool(tool_manifests, "reuse_archive", {"name": reuse_name})
|
||||
arch = _try_parse_json(reuse_text)
|
||||
if isinstance(arch, dict) and arch.get("ok"):
|
||||
data = arch.get("data") if isinstance(arch.get("data"), dict) else {}
|
||||
if data.get("id"):
|
||||
args["archive_id"] = str(data["id"])
|
||||
if not args["stages"] and isinstance(data.get("stage_chain"), list):
|
||||
args["stages"] = [str(s) for s in data["stage_chain"]]
|
||||
if isinstance(data.get("params"), dict):
|
||||
merged = dict(data["params"])
|
||||
if isinstance(payload.get("params"), dict):
|
||||
merged.update(payload["params"])
|
||||
args["params"] = merged
|
||||
elif not args["stages"]:
|
||||
# 复用失败(非 ok)且无显式阶段链:不能空链建任务,回退原 pipeline
|
||||
return None
|
||||
tool_names.append(tool) # create_*_job 最后执行
|
||||
if "params" not in args and isinstance(payload.get("params"), dict):
|
||||
args["params"] = payload["params"]
|
||||
result = _execute_tool(tool_manifests, tool, args)
|
||||
job_id = _extract_job_id(result)
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户确认了分析路径并创建任务({'基因家族' if cap_name == 'bioflow-gene-family' else '转录组'}链"
|
||||
f"{',复用存档 ' + reuse_name if reuse_name else ''})。工具结果:\n{result}",
|
||||
)
|
||||
return _make_payload(
|
||||
"确认路径并创建任务", answer,
|
||||
audit={"rounds": 1, "tool_calls": len(tool_names), "used_json_fallback": False, "tool_names": tool_names, "errors": []},
|
||||
capabilities=[cap_name] if cap_name else [],
|
||||
active_job_id=job_id,
|
||||
chain=chain or None,
|
||||
)
|
||||
|
||||
|
||||
def _card_form_confirm(
|
||||
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""参数卡确认 → confirm_stage(白名单校验)→ 复述;rerun:true 走 rerun_stage(改参数重跑)。"""
|
||||
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
|
||||
stage = str(payload.get("stage") or "").strip()
|
||||
params = payload.get("params") if isinstance(payload.get("params"), dict) else {}
|
||||
rerun = bool(payload.get("rerun", False))
|
||||
if not job_id or not stage:
|
||||
return None
|
||||
tool = "rerun_stage" if rerun else "confirm_stage"
|
||||
result = _execute_tool(tool_manifests, tool, {"job_id": job_id, "stage": stage, "params": params})
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户{'重跑并确认' if rerun else '确认'}了阶段参数(任务 {job_id} 的 {stage})。工具结果:\n{result}",
|
||||
)
|
||||
return _make_payload(
|
||||
f"{'重跑' if rerun else '确认'}阶段参数:{stage}", answer,
|
||||
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": [tool], "errors": []},
|
||||
active_job_id=job_id,
|
||||
)
|
||||
|
||||
|
||||
def _card_file_upload(
|
||||
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""文件卡 → 网络源 fetch_source_file / 服务器·局域网路径 register_job_files → 复述。"""
|
||||
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
|
||||
raw_files = payload.get("paths") or payload.get("files") or []
|
||||
if isinstance(raw_files, str):
|
||||
raw_files = [raw_files]
|
||||
files = [str(f) for f in raw_files if str(f).strip()] if isinstance(raw_files, list) else []
|
||||
if not job_id or not files:
|
||||
return None
|
||||
remote = [f for f in files if f.startswith(("http://", "https://", "oss://"))]
|
||||
local = [f for f in files if f not in remote]
|
||||
results: List[str] = []
|
||||
tool_names: List[str] = []
|
||||
for src in remote:
|
||||
r = _execute_tool(tool_manifests, "fetch_source_file", {"source": src, "job_id": job_id})
|
||||
results.append(r)
|
||||
tool_names.append("fetch_source_file")
|
||||
if local:
|
||||
r = _execute_tool(tool_manifests, "register_job_files", {"job_id": job_id, "paths": local})
|
||||
results.append(r)
|
||||
tool_names.append("register_job_files")
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户提供了文件({'远程 ' + str(len(remote)) + ' 个' if remote else ''}"
|
||||
f"{'本地 ' + str(len(local)) + ' 个' if local else ''})。工具结果:\n" + "\n".join(results),
|
||||
)
|
||||
return _make_payload(
|
||||
"登记任务文件", answer,
|
||||
audit={"rounds": 1, "tool_calls": len(tool_names), "used_json_fallback": False, "tool_names": tool_names, "errors": []},
|
||||
active_job_id=job_id,
|
||||
)
|
||||
|
||||
|
||||
def _card_archive_choice(
|
||||
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""存档卡 → archive_job 工具执行实际存档(或跳过)→ 复述。"""
|
||||
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
|
||||
do_archive = bool(payload.get("archive", False))
|
||||
if not job_id:
|
||||
return None
|
||||
if do_archive:
|
||||
name = str(payload.get("name") or "")
|
||||
description = str(payload.get("description") or "")
|
||||
result = _execute_tool(
|
||||
tool_manifests, "archive_job",
|
||||
{"job_id": job_id, "name": name, "description": description},
|
||||
)
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户选择把任务 {job_id} 存档(名称:{name or '未命名'})。工具结果:\n{result}",
|
||||
)
|
||||
return _make_payload(
|
||||
"存档选择", answer,
|
||||
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": ["archive_job"], "errors": []},
|
||||
active_job_id=job_id,
|
||||
)
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户选择不存档任务 {job_id}。请用自然语言告知用户:已跳过存档,"
|
||||
"本次分析结果仍可在报告/结果中查看。",
|
||||
)
|
||||
return _make_payload(
|
||||
"存档选择", answer,
|
||||
audit=_empty_audit(),
|
||||
active_job_id=job_id,
|
||||
)
|
||||
|
||||
|
||||
def _card_extension_action(
|
||||
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""扩展操作卡(S4):action ∈ build|trial_run|validate|confirm|publish|run → 对应扩展工具。"""
|
||||
ext_id = str(payload.get("ext_id") or "").strip()
|
||||
action = str(payload.get("action") or "").strip()
|
||||
if not ext_id or not action:
|
||||
return None
|
||||
tool = {
|
||||
"build": "build_extension",
|
||||
"trial_run": "trial_run_extension",
|
||||
"validate": "validate_extension",
|
||||
"confirm": "confirm_extension",
|
||||
"publish": "publish_extension",
|
||||
"run": "run_extension",
|
||||
"repair": "repair_extension",
|
||||
}.get(action)
|
||||
if tool is None or tool not in tool_manifests:
|
||||
return None
|
||||
args: Dict[str, Any] = {"ext_id": ext_id}
|
||||
if action == "confirm":
|
||||
if isinstance(payload.get("nl_triggers"), list):
|
||||
args["nl_triggers"] = payload["nl_triggers"]
|
||||
if isinstance(payload.get("parameters"), list):
|
||||
args["parameters"] = payload["parameters"]
|
||||
if payload.get("runtime_image"):
|
||||
args["runtime_image"] = str(payload["runtime_image"])
|
||||
if action == "run":
|
||||
if isinstance(payload.get("params"), dict):
|
||||
args["params"] = payload["params"]
|
||||
if isinstance(payload.get("files"), list):
|
||||
args["files"] = payload["files"]
|
||||
result = _execute_tool(tool_manifests, tool, args)
|
||||
if action == "confirm" and "EXT_STATUS" in result:
|
||||
# 状态降级话术:confirm 要求 validated/trial_passed 前置状态,
|
||||
# 编辑修正若发生在校验前,明确提示先试运行/校验,不报 500。
|
||||
answer = (
|
||||
"该扩展当前状态还不能确认。请先完成 AI 转换与试运行、通过校验后,"
|
||||
"再确认参数与触发词。"
|
||||
)
|
||||
else:
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户对扩展 {ext_id} 执行了操作 {action}。工具结果:\n{result}",
|
||||
)
|
||||
return _make_payload(
|
||||
f"扩展操作:{action}", answer,
|
||||
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": [tool], "errors": []},
|
||||
)
|
||||
|
||||
|
||||
def _handle_card_result(
|
||||
card_result: Dict[str, Any],
|
||||
llm_client: Any,
|
||||
bioflow: Optional[Dict[str, Any]],
|
||||
tool_manifests: Dict[str, CapabilityManifest],
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
card = str(card_result.get("card") or "").strip()
|
||||
payload = card_result.get("payload") if isinstance(card_result.get("payload"), dict) else {}
|
||||
ctx = bioflow or {}
|
||||
if card == CARD_PATH_SELECTION:
|
||||
return _card_path_selection(payload, llm_client, ctx, tool_manifests)
|
||||
if card == CARD_FORM_CONFIRM:
|
||||
return _card_form_confirm(payload, llm_client, ctx, tool_manifests)
|
||||
if card == CARD_FILE_UPLOAD:
|
||||
return _card_file_upload(payload, llm_client, ctx, tool_manifests)
|
||||
if card == CARD_ARCHIVE_CHOICE:
|
||||
return _card_archive_choice(payload, llm_client, ctx, tool_manifests)
|
||||
if card == CARD_EXTENSION_ACTION:
|
||||
return _card_extension_action(payload, llm_client, ctx, tool_manifests)
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 前门路由
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def try_bioflow_agent(
|
||||
question: str,
|
||||
*,
|
||||
llm_client: Any,
|
||||
history: Optional[List[Dict[str, str]]] = None,
|
||||
capabilities: Optional[Dict[str, Any]] = None,
|
||||
card_result: Optional[Dict[str, Any]] = None,
|
||||
bioflow: Optional[Dict[str, Any]] = None,
|
||||
progress_callback: Optional[Any] = None,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""前门路由入口。命中生信能力则返回答案 payload,否则返回 None(走原 pipeline)。
|
||||
|
||||
- card_result:非空时走卡片回填协议(直接路由到对应工具,不做意图预筛)。
|
||||
- 任何异常都不上抛——记日志后返回 None,保证金标路径永不因本模块失败。
|
||||
"""
|
||||
try:
|
||||
return _try_bioflow_agent_impl(
|
||||
question,
|
||||
llm_client=llm_client,
|
||||
history=history,
|
||||
capabilities=capabilities,
|
||||
card_result=card_result,
|
||||
bioflow=bioflow,
|
||||
progress_callback=progress_callback,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: 执行失败,回退原 pipeline: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
def _try_bioflow_agent_impl(
|
||||
question: str,
|
||||
*,
|
||||
llm_client: Any,
|
||||
history: Optional[List[Dict[str, str]]],
|
||||
capabilities: Optional[Dict[str, Any]],
|
||||
card_result: Optional[Dict[str, Any]],
|
||||
bioflow: Optional[Dict[str, Any]],
|
||||
progress_callback: Optional[Any],
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
# 门 1:会话开关(熔断)
|
||||
if not _env_enabled():
|
||||
return None
|
||||
|
||||
# card_result 卡片操作优先:用户已在卡片上明确选择,不做意图预筛,直接路由
|
||||
if isinstance(card_result, dict) and str(card_result.get("card") or "").strip() in CARD_TYPES:
|
||||
return _route_card_result(card_result, llm_client, capabilities, bioflow, progress_callback)
|
||||
|
||||
# NL 路径:活跃任务/生信链续接优先,其次意图预筛
|
||||
active_job_id = (bioflow or {}).get("active_job_id")
|
||||
chain = str((bioflow or {}).get("chain") or "").strip()
|
||||
if active_job_id:
|
||||
# 有活跃任务:跳过意图预筛,按任务 chain 续接(任务链不可变)
|
||||
manifests = _registered_capabilities(capabilities)
|
||||
cap_name = _chain_to_capability(chain)
|
||||
if cap_name:
|
||||
manifests = [m for m in manifests if m.name == cap_name]
|
||||
else:
|
||||
# 无活跃任务:先门 2 + 门 3 意图预筛(新任务入口)
|
||||
manifests = _candidate_capabilities(question, capabilities)
|
||||
# 意图未命中但 session 留有生信链(上轮规划)→ 续接该链(如路径选择「1」)
|
||||
if not manifests and chain:
|
||||
cap_name = _chain_to_capability(chain)
|
||||
if cap_name:
|
||||
registered = _registered_capabilities(capabilities)
|
||||
manifests = [m for m in registered if m.name == cap_name]
|
||||
if not manifests:
|
||||
return None
|
||||
|
||||
tools, tool_manifests = _build_tools(manifests)
|
||||
if not tools:
|
||||
LOGGER.warning("bioflow agent: 意图命中但 MCP 工具发现为空,回退原 pipeline")
|
||||
return None
|
||||
|
||||
if progress_callback is not None:
|
||||
try:
|
||||
progress_callback(
|
||||
{
|
||||
"stage": "bioflow_agent_started",
|
||||
"capabilities": [m.name for m in manifests],
|
||||
"tool_count": len(tools),
|
||||
}
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
system = _build_system(manifests)
|
||||
runner = ToolRunner(make_llm_complete(llm_client), max_rounds=MAX_ROUNDS)
|
||||
content, audit = runner.run(system, question, tools, tool_manifests, history=history)
|
||||
|
||||
answer = (content or "").strip()
|
||||
if not answer:
|
||||
return None
|
||||
|
||||
tool_results = (audit or {}).get("tool_results") or {}
|
||||
readiness = None
|
||||
versioning = None
|
||||
new_job_id = None
|
||||
plan = None
|
||||
for name, result_text in tool_results.items():
|
||||
if name in ("create_gene_family_job", "create_rnaseq_job"):
|
||||
jid = _extract_job_id(result_text)
|
||||
if jid:
|
||||
new_job_id = jid
|
||||
data = _try_parse_json(result_text)
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
if name.startswith("plan_") and data.get("ok"):
|
||||
plan = {
|
||||
"candidates": data.get("candidates") or [],
|
||||
"auto_filled": data.get("auto_filled") or [],
|
||||
"reuse_archive": data.get("reuse_archive"),
|
||||
"reuse_name": data.get("reuse_name"),
|
||||
"fallback": data.get("fallback"),
|
||||
}
|
||||
if name == "check_readiness":
|
||||
readiness = _map_readiness(data)
|
||||
elif name == "propose_change":
|
||||
versioning = _map_versioning(data)
|
||||
|
||||
effective_chain = chain or (_CAPABILITY_TO_CHAIN.get(manifests[0].name) if manifests else "")
|
||||
return _make_payload(
|
||||
question,
|
||||
answer,
|
||||
audit=audit,
|
||||
capabilities=[m.name for m in manifests],
|
||||
active_job_id=(bioflow or {}).get("active_job_id") or new_job_id,
|
||||
chain=effective_chain or None,
|
||||
readiness=readiness,
|
||||
versioning=versioning,
|
||||
plan=plan,
|
||||
)
|
||||
|
||||
|
||||
def _route_card_result(
|
||||
card_result: Dict[str, Any],
|
||||
llm_client: Any,
|
||||
capabilities: Optional[Dict[str, Any]],
|
||||
bioflow: Optional[Dict[str, Any]],
|
||||
progress_callback: Optional[Any],
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""卡片操作路由:确定相关能力 → 构建工具 → 分发到对应 card handler。"""
|
||||
card = str(card_result.get("card") or "").strip()
|
||||
payload = card_result.get("payload") if isinstance(card_result.get("payload"), dict) else {}
|
||||
chain = str(payload.get("chain") or (bioflow or {}).get("chain") or "").strip()
|
||||
if card == CARD_EXTENSION_ACTION:
|
||||
# 扩展操作只依赖 bioflow-extension 能力,避免对三个能力各做一次 list_tools
|
||||
cap_name = "bioflow-extension"
|
||||
else:
|
||||
cap_name = _chain_to_capability(chain)
|
||||
registered = _registered_capabilities(capabilities)
|
||||
if cap_name:
|
||||
manifests = [m for m in registered if m.name == cap_name]
|
||||
else:
|
||||
manifests = registered
|
||||
if not manifests:
|
||||
return None
|
||||
tools, tool_manifests = _build_tools(manifests)
|
||||
if not tools:
|
||||
LOGGER.warning("bioflow agent: card_result 但 MCP 工具发现为空,回退原 pipeline")
|
||||
return None
|
||||
if progress_callback is not None:
|
||||
try:
|
||||
progress_callback({"stage": "bioflow_card", "card": card_result.get("card")})
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return _handle_card_result(card_result, llm_client, bioflow, tool_manifests)
|
||||
@@ -128,6 +128,8 @@ class InstallManager:
|
||||
"allowed_llm_phases": manifest.allowed_llm_phases,
|
||||
"session_default": manifest.session_default,
|
||||
}
|
||||
if manifest.nl_triggers:
|
||||
raw["nl_triggers"] = manifest.nl_triggers
|
||||
if manifest.schema:
|
||||
raw["schema"] = manifest.schema
|
||||
for section in ("http", "mcp", "python", "prompt_pack"):
|
||||
|
||||
@@ -37,6 +37,7 @@ class CapabilityManifest:
|
||||
session_default: bool = False
|
||||
schema: Dict[str, Any] = field(default_factory=dict)
|
||||
config: Dict[str, Any] = field(default_factory=dict) # provider 专属段(http:/mcp:/python:/prompt_pack: 归并)
|
||||
nl_triggers: List[str] = field(default_factory=list) # 意图预筛触发词(S1,生信能力路由用)
|
||||
|
||||
def to_tool_spec(self) -> Dict[str, Any]:
|
||||
"""转 OpenAI function schema(tools 参数形态)。"""
|
||||
@@ -110,6 +111,19 @@ def _parse_entry(raw: Any, index: int, load_warnings: List[str]) -> CapabilityMa
|
||||
if isinstance(raw.get(section), dict):
|
||||
config[section] = raw[section]
|
||||
config["session_default"] = bool(raw.get("session_default", False))
|
||||
# nl_triggers(S1):意图预筛触发词,只收字符串、去空、去重、限长
|
||||
nl_triggers_raw = raw.get("nl_triggers") or []
|
||||
if isinstance(nl_triggers_raw, str):
|
||||
nl_triggers_raw = [nl_triggers_raw]
|
||||
nl_triggers: List[str] = []
|
||||
if isinstance(nl_triggers_raw, list):
|
||||
for trigger in nl_triggers_raw:
|
||||
if not isinstance(trigger, str):
|
||||
continue # 触发词只收字符串,非字符串(数字等)丢弃
|
||||
cleaned = _clean_text(trigger, 64)
|
||||
if cleaned and cleaned not in nl_triggers:
|
||||
nl_triggers.append(cleaned)
|
||||
nl_triggers = nl_triggers[:64]
|
||||
manifest = CapabilityManifest(
|
||||
name=name,
|
||||
provider=provider,
|
||||
@@ -119,6 +133,7 @@ def _parse_entry(raw: Any, index: int, load_warnings: List[str]) -> CapabilityMa
|
||||
session_default=bool(raw.get("session_default", False)),
|
||||
schema=schema,
|
||||
config=config,
|
||||
nl_triggers=nl_triggers,
|
||||
)
|
||||
# 大小上限:单条目序列化后 ≤8KB
|
||||
import json
|
||||
|
||||
@@ -130,6 +130,7 @@ class _McpConnection:
|
||||
self._thread: Optional[threading.Thread] = None
|
||||
self._session: Any = None
|
||||
self._server_tools: List[str] = []
|
||||
self._server_tool_schemas: List[Dict[str, Any]] = []
|
||||
self._started_at = 0.0
|
||||
self._last_used = 0.0
|
||||
self._restarts = 0
|
||||
@@ -200,6 +201,18 @@ class _McpConnection:
|
||||
raise
|
||||
self._ctx = ctx
|
||||
self._server_tools = [t.name for t in tools_result.tools]
|
||||
self._server_tool_schemas = [
|
||||
{
|
||||
"name": t.name,
|
||||
"description": (getattr(t, "description", None) or ""),
|
||||
"input_schema": (
|
||||
getattr(t, "inputSchema", None)
|
||||
or getattr(t, "input_schema", None)
|
||||
or {}
|
||||
),
|
||||
}
|
||||
for t in tools_result.tools
|
||||
]
|
||||
self._session = session
|
||||
self._started_at = time.time()
|
||||
|
||||
@@ -225,6 +238,7 @@ class _McpConnection:
|
||||
ctx = getattr(self, "_ctx", None)
|
||||
self._ctx = None
|
||||
self._server_tools = []
|
||||
self._server_tool_schemas = []
|
||||
if session is not None:
|
||||
try:
|
||||
await session.__aexit__(None, None, None)
|
||||
@@ -274,6 +288,14 @@ class _McpConnection:
|
||||
return []
|
||||
return list(self._server_tools)
|
||||
|
||||
def list_tool_schemas(self) -> List[Dict[str, Any]]:
|
||||
"""返回 MCP server 全量工具 schema(name/description/input_schema)。连接失败返回空。"""
|
||||
try:
|
||||
self._ensure_session()
|
||||
except ToolExecutionError:
|
||||
return []
|
||||
return list(self._server_tool_schemas)
|
||||
|
||||
def call(self, tool_name: str, arguments: Dict[str, Any]) -> str:
|
||||
self._ensure_session()
|
||||
try:
|
||||
@@ -337,6 +359,11 @@ class McpToolProvider:
|
||||
return tools + [f"<缺失声明工具 {tool_name}>"]
|
||||
return tools
|
||||
|
||||
def list_tool_schemas(self, manifest: CapabilityManifest) -> List[Dict[str, Any]]:
|
||||
"""返回该能力 MCP server 的全量工具 schema(供前门 agent 构建 LLM 工具表)。"""
|
||||
conn = self._connection(manifest)
|
||||
return conn.list_tool_schemas()
|
||||
|
||||
def reap_idle(self) -> int:
|
||||
reaped = 0
|
||||
with self._lock:
|
||||
@@ -448,6 +475,10 @@ class ToolDispatcher:
|
||||
return PythonToolProvider.execute(manifest, arguments)
|
||||
raise ToolExecutionError(f"工具 {manifest.name} provider={provider} 不支持调用")
|
||||
|
||||
def list_mcp_tools(self, manifest: CapabilityManifest) -> List[Dict[str, Any]]:
|
||||
"""MCP 能力全量工具 schema(bioflow_agent 构建 LLM 工具表用)。"""
|
||||
return self._mcp.list_tool_schemas(manifest)
|
||||
|
||||
def reap_idle(self) -> int:
|
||||
return self._mcp.reap_idle()
|
||||
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
"""test_bioflow_agent.py —— bioflow_agent 前门三重门 + nl_triggers 解析单测。
|
||||
|
||||
覆盖 S1 验收要点:
|
||||
- 熔断开关(QA_V1_ENABLE_BIOFLOW_AGENT)默认关 → 返回 None(走原 pipeline)
|
||||
- 意图预筛(nl_triggers)各自评估,未命中返回 None
|
||||
- 注册表未注册 / 会话开关关 → 返回 None
|
||||
- 命中后短路返回自然语言 payload(bioflow.handled=True)
|
||||
- 只装一个能力时,另一个能力的触发语不响应(隔离)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.manifests import parse_capabilities_text # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
|
||||
|
||||
BIOFLOW_YAML = """
|
||||
capabilities:
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 基因家族分析
|
||||
session_default: true
|
||||
nl_triggers: [基因家族, 家族成员, 亚家族, 建树]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/gene-family/
|
||||
prompt_pack:
|
||||
file: bioflow_gene_family_agent_rules.md
|
||||
- name: bioflow-rnaseq
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 转录组分析
|
||||
session_default: true
|
||||
nl_triggers: [转录组, RNA-seq, 差异表达, 富集]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/rnaseq/
|
||||
prompt_pack:
|
||||
file: bioflow_rnaseq_agent_rules.md
|
||||
"""
|
||||
|
||||
|
||||
def _make_registry(tmp_path: Path) -> CapabilityRegistry:
|
||||
path = tmp_path / "capabilities.yaml"
|
||||
path.write_text(BIOFLOW_YAML, encoding="utf-8")
|
||||
registry = CapabilityRegistry(path)
|
||||
registry.load()
|
||||
return registry
|
||||
|
||||
|
||||
class FakeLLMClient:
|
||||
def __init__(self, content: str = "已为您规划分析路径。") -> None:
|
||||
self.content = content
|
||||
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kwargs):
|
||||
return {"content": self.content, "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
class TestNlTriggers:
|
||||
def test_parse_nl_triggers(self):
|
||||
manifests, warnings = parse_capabilities_text(BIOFLOW_YAML)
|
||||
assert warnings == []
|
||||
gf = next(m for m in manifests if m.name == "bioflow-gene-family")
|
||||
assert gf.nl_triggers == ["基因家族", "家族成员", "亚家族", "建树"]
|
||||
assert gf.config["mcp"]["url"].endswith("/mcp/gene-family/")
|
||||
assert gf.config["prompt_pack"]["file"] == "bioflow_gene_family_agent_rules.md"
|
||||
|
||||
def test_nl_triggers_dedupe_and_clean(self):
|
||||
text = """
|
||||
capabilities:
|
||||
- name: t
|
||||
provider: mcp
|
||||
nl_triggers: [基因家族, 基因家族, "", " 亚家族 ", 123]
|
||||
"""
|
||||
manifests, _ = parse_capabilities_text(text)
|
||||
assert manifests[0].nl_triggers == ["基因家族", "亚家族"]
|
||||
|
||||
|
||||
class TestIntentMatch:
|
||||
def test_match(self):
|
||||
assert ba._intent_match("帮我做水稻SPL基因家族分析", ["基因家族"]) is True
|
||||
assert ba._intent_match("转录组差异表达分析", ["转录组", "RNA-seq"]) is True
|
||||
assert ba._intent_match("什么是光合作用", ["基因家族", "转录组"]) is False
|
||||
|
||||
|
||||
class TestGateLogic:
|
||||
def test_fuse_off_returns_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict("os.environ", {}, clear=False):
|
||||
ba.os.environ.pop(ba.ENV_FUSE, None)
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_no_intent_match_returns_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"光合作用的电子传递链是什么", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_active_job_continuation_skips_intent_prescreen(self, tmp_path):
|
||||
"""#011:有活跃任务时,续接输入(不含触发词)也应进入生信流,不掉普通 QA。"""
|
||||
registry = _make_registry(tmp_path)
|
||||
fake_tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "fetch_source_file",
|
||||
"description": "获取源文件",
|
||||
"parameters": {"type": "object", "properties": {}},
|
||||
},
|
||||
}
|
||||
]
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {"fetch_source_file": mock.ANY})):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"基因组文件在 https://example.com/rice.fa",
|
||||
llm_client=FakeLLMClient(),
|
||||
capabilities=None,
|
||||
bioflow={"active_job_id": "job-1", "chain": "gene_family"},
|
||||
)
|
||||
assert result is not None
|
||||
assert result["success"] is True
|
||||
assert result["bioflow"]["handled"] is True
|
||||
assert result["bioflow"]["capabilities"] == ["bioflow-gene-family"]
|
||||
|
||||
def test_not_registered_returns_none(self, tmp_path):
|
||||
empty = CapabilityRegistry(tmp_path / "empty.yaml")
|
||||
empty.load()
|
||||
with mock.patch.object(ba, "get_registry", return_value=empty):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_session_switch_off_returns_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析",
|
||||
llm_client=FakeLLMClient(),
|
||||
capabilities={"bioflow-gene-family": False},
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_isolation_only_installed_responds(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
caps = ba._candidate_capabilities("帮我做基因家族分析", None)
|
||||
assert [m.name for m in caps] == ["bioflow-gene-family"]
|
||||
caps = ba._candidate_capabilities("转录组差异表达", None)
|
||||
assert [m.name for m in caps] == ["bioflow-rnaseq"]
|
||||
|
||||
def test_full_flow_short_circuits(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
fake_tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "plan_gene_family",
|
||||
"description": "规划基因家族路径",
|
||||
"parameters": {"type": "object", "properties": {}},
|
||||
},
|
||||
}
|
||||
]
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {"plan_gene_family": mock.ANY})):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is not None
|
||||
assert result["success"] is True
|
||||
assert result["answer"] == "已为您规划分析路径。"
|
||||
assert result["bioflow"]["handled"] is True
|
||||
assert result["bioflow"]["capabilities"] == ["bioflow-gene-family"]
|
||||
|
||||
def test_exception_falls_back_to_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.object(ba, "_candidate_capabilities", side_effect=RuntimeError("boom")):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
@@ -0,0 +1,255 @@
|
||||
"""test_bioflow_agent_s2.py —— S2 对话式编排闭环单测。
|
||||
|
||||
覆盖:card_result 回填协议(四类卡片)、参数抽取(白名单)、多轮记忆注入、
|
||||
session↔job 绑定上下文(active_job_id/chain)、job_status 主动查询。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest # noqa: E402
|
||||
from rag.qa_v1_1_73.tool_runner import ToolRunner # noqa: E402
|
||||
|
||||
|
||||
def _manifest(name="bioflow-gene-family", url="http://x/mcp/gene-family/"):
|
||||
return CapabilityManifest(name=name, provider="mcp", config={"mcp": {"url": url, "tool": name}})
|
||||
|
||||
|
||||
def _tool_manifests():
|
||||
"""覆盖四类卡片用到的工具 + job_status。"""
|
||||
names = ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
|
||||
"register_job_files", "fetch_source_file", "job_status")
|
||||
out = {}
|
||||
for t in names:
|
||||
out[t] = CapabilityManifest(
|
||||
name="bioflow-gene-family", provider="mcp",
|
||||
config={"mcp": {"url": "http://x/mcp/gene-family/", "tool": t}},
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
class FakeDispatcher:
|
||||
def __init__(self, results=None):
|
||||
self.results = results or {}
|
||||
self.calls = []
|
||||
|
||||
def execute(self, manifest, arguments):
|
||||
tool = manifest.config.get("mcp", {}).get("tool")
|
||||
self.calls.append((tool, arguments))
|
||||
return self.results.get(tool, '{"ok": true, "data": {"id": "job-1"}}')
|
||||
|
||||
|
||||
class FakeLLM:
|
||||
def __init__(self, complete_text="已收到,已为您处理。", json_payload=None):
|
||||
self.complete_text = complete_text
|
||||
self.json_payload = json_payload or {}
|
||||
|
||||
def complete(self, system, user, **kw):
|
||||
return self.complete_text
|
||||
|
||||
def complete_json(self, system, user, **kw):
|
||||
return self.json_payload
|
||||
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 参数抽取
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestExtractParams:
|
||||
def test_extract_known_and_unknown(self):
|
||||
llm = FakeLLM(json_payload={"params": {"threads": 8, "padj": 0.01}, "unknown": ["bogus"]})
|
||||
form_defaults = {"threads": {"default": 4}, "padj": {"default": 0.05}}
|
||||
known, unknown = ba._extract_params(llm, "用 8 核,padj 用 0.01", form_defaults)
|
||||
assert known == {"threads": 8, "padj": 0.01}
|
||||
assert unknown == ["bogus"]
|
||||
|
||||
def test_non_whitelist_param_filtered(self):
|
||||
llm = FakeLLM(json_payload={"params": {"evil": 1, "threads": 4}, "unknown": []})
|
||||
form_defaults = {"threads": {"default": 4}}
|
||||
known, unknown = ba._extract_params(llm, "x", form_defaults)
|
||||
assert known == {"threads": 4}
|
||||
|
||||
def test_bad_json_returns_empty(self):
|
||||
llm = FakeLLM(json_payload="not-a-dict")
|
||||
form_defaults = {"threads": {}}
|
||||
known, unknown = ba._extract_params(llm, "x", form_defaults)
|
||||
assert known == {} and unknown == []
|
||||
|
||||
def test_empty_whitelist(self):
|
||||
llm = FakeLLM()
|
||||
known, unknown = ba._extract_params(llm, "x", {})
|
||||
assert known == {} and unknown == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# card_result 回填协议
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestCardResult:
|
||||
def test_path_selection_creates_job_and_extracts_job_id(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
llm = FakeLLM()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "gene_family", "stages": ["identify", "tree"], "title": "水稻SPL"}},
|
||||
llm, {"session_id": "s1"}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert out["bioflow"]["active_job_id"] == "job-1"
|
||||
assert out["bioflow"]["chain"] == "gene_family"
|
||||
assert dispatcher.calls[0][0] == "create_gene_family_job"
|
||||
assert dispatcher.calls[0][1]["stages"] == ["identify", "tree"]
|
||||
assert dispatcher.calls[0][1]["session_id"] == "s1"
|
||||
|
||||
def test_path_selection_missing_stages_returns_none(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection", "payload": {"chain": "gene_family"}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_form_confirm_calls_confirm_stage(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "form_confirm",
|
||||
"payload": {"job_id": "job-9", "stage": "identify", "params": {"evalue": 1e-5}}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "confirm_stage"
|
||||
assert dispatcher.calls[0][1] == {"job_id": "job-9", "stage": "identify", "params": {"evalue": 1e-5}}
|
||||
|
||||
def test_form_confirm_uses_active_job_id_fallback(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
ba._handle_card_result(
|
||||
{"card": "form_confirm", "payload": {"stage": "identify", "params": {}}},
|
||||
FakeLLM(), {"active_job_id": "job-7"}, _tool_manifests(),
|
||||
)
|
||||
assert dispatcher.calls[0][1]["job_id"] == "job-7"
|
||||
|
||||
def test_file_upload_splits_remote_and_local(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "file_upload",
|
||||
"payload": {"job_id": "j1", "paths": ["https://x/a.fa", "/data/b.fa"]}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
tools = [c[0] for c in dispatcher.calls]
|
||||
assert "fetch_source_file" in tools and "register_job_files" in tools
|
||||
|
||||
def test_archive_choice_paraphrases(self):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "archive_choice", "payload": {"job_id": "j1", "archive": True}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert out["answer"] == "已收到,已为您处理。"
|
||||
|
||||
def test_unknown_card_returns_none(self):
|
||||
out = ba._handle_card_result({"card": "bogus", "payload": {}}, FakeLLM(), {}, _tool_manifests())
|
||||
assert out is None
|
||||
|
||||
def test_path_selection_reuse_failure_no_stages_returns_none(self):
|
||||
"""bug ②:复用失败且无显式 stages → 返回 None(不空链建任务)。"""
|
||||
dispatcher = FakeDispatcher({"reuse_archive": '{"ok": false, "code": "ARCHIVE_NOT_FOUND"}'})
|
||||
tm = _tool_manifests()
|
||||
tm["reuse_archive"] = CapabilityManifest(
|
||||
name="bioflow-gene-family", provider="mcp",
|
||||
config={"mcp": {"url": "http://x", "tool": "reuse_archive"}},
|
||||
)
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "gene_family", "reuse_archive": "不存在的存档"}},
|
||||
FakeLLM(), {}, tm,
|
||||
)
|
||||
assert out is None
|
||||
# 只执行了 reuse_archive,未调用 create_gene_family_job
|
||||
assert [c[0] for c in dispatcher.calls] == ["reuse_archive"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 前门路由(card_result / bioflow 上下文)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestRouteCardResult:
|
||||
def test_try_bioflow_agent_card_result_short_circuits(self):
|
||||
registry_manifest = _manifest()
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[registry_manifest]):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([{"type": "function", "function": {"name": "confirm_stage", "description": "", "parameters": {"type": "object", "properties": {}}}}], {"confirm_stage": registry_manifest})):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba.try_bioflow_agent(
|
||||
"确认", llm_client=FakeLLM(),
|
||||
card_result={"card": "form_confirm", "payload": {"job_id": "j1", "stage": "s", "params": {}}},
|
||||
)
|
||||
assert out is not None
|
||||
assert out["bioflow"]["handled"] is True
|
||||
|
||||
def test_try_bioflow_agent_fuse_off_ignores_card_result(self):
|
||||
with mock.patch.dict("os.environ", {}, clear=False):
|
||||
ba.os.environ.pop(ba.ENV_FUSE, None)
|
||||
out = ba.try_bioflow_agent(
|
||||
"确认", llm_client=FakeLLM(),
|
||||
card_result={"card": "form_confirm", "payload": {}},
|
||||
)
|
||||
assert out is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 多轮记忆注入(ToolRunner.run history)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestMultiTurnHistory:
|
||||
def test_history_injected_into_messages(self):
|
||||
class RecordingLLM:
|
||||
def __init__(self):
|
||||
self.captured_messages = None
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
self.captured_messages = list(messages)
|
||||
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
from rag.qa_v1_1_73.tool_runner import make_llm_complete
|
||||
|
||||
llm = RecordingLLM()
|
||||
runner = ToolRunner(make_llm_complete(llm), max_rounds=1)
|
||||
history = [{"role": "user", "content": "上一轮问题"}, {"role": "assistant", "content": "上一轮答案"}]
|
||||
runner.run("sys", "当前问题", [], {}, history=history)
|
||||
msgs = llm.captured_messages
|
||||
assert msgs[0] == {"role": "user", "content": "上一轮问题"}
|
||||
assert msgs[1] == {"role": "assistant", "content": "上一轮答案"}
|
||||
assert msgs[-1] == {"role": "user", "content": "当前问题"}
|
||||
|
||||
def test_history_filters_bad_roles(self):
|
||||
class RecordingLLM:
|
||||
def __init__(self):
|
||||
self.captured_messages = None
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
self.captured_messages = list(messages)
|
||||
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
from rag.qa_v1_1_73.tool_runner import make_llm_complete
|
||||
|
||||
llm = RecordingLLM()
|
||||
runner = ToolRunner(make_llm_complete(llm), max_rounds=1)
|
||||
history = [{"role": "system", "content": "evil"}, {"role": "user", "content": ""}, {"role": "user", "content": "ok"}]
|
||||
runner.run("sys", "q", [], {}, history=history)
|
||||
roles = [m["role"] for m in llm.captured_messages]
|
||||
assert "system" not in roles
|
||||
assert llm.captured_messages[-1] == {"role": "user", "content": "q"}
|
||||
@@ -0,0 +1,511 @@
|
||||
"""test_bioflow_agent_s2_supplement.py —— S2 覆盖清单补强测试。
|
||||
|
||||
补强 dev 的 test_bioflow_agent_s2.py 未覆盖的分支:
|
||||
- _chain_to_capability / _extract_job_id / _execute_tool / _paraphrase / _query_job_status
|
||||
- card 四类 handler 缺必需字段回退 None、files 键、archive=False、非 dict payload
|
||||
- _route_card_result:chain 过滤 / 未注册能力→None / tools 空→None
|
||||
- NL 路径注入 bioflow 上下文(active_job_id/chain)
|
||||
- system prompt 硬规则在位(追问 ≤3 / 不可代确认 / 工具名不外泄)
|
||||
- ToolRunner.run(history) 边界(non-dict 条目 / history=None / 空内容剔除)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest # noqa: E402
|
||||
from rag.qa_v1_1_73.tool_runner import ToolRunner, make_llm_complete # noqa: E402
|
||||
|
||||
|
||||
def _manifest(name="bioflow-gene-family", url="http://x/mcp/gene-family/"):
|
||||
return CapabilityManifest(name=name, provider="mcp", config={"mcp": {"url": url, "tool": name}})
|
||||
|
||||
|
||||
def _tool_manifests():
|
||||
names = ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
|
||||
"register_job_files", "fetch_source_file", "job_status",
|
||||
"rerun_stage", "reuse_archive", "archive_job")
|
||||
out = {}
|
||||
for t in names:
|
||||
out[t] = CapabilityManifest(
|
||||
name="bioflow-gene-family", provider="mcp",
|
||||
config={"mcp": {"url": "http://x/mcp/gene-family/", "tool": t}},
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
class FakeLLM:
|
||||
def __init__(self, complete_text="已收到。", json_payload=None, complete_error=None):
|
||||
self.complete_text = complete_text
|
||||
self.json_payload = json_payload or {}
|
||||
self.complete_error = complete_error
|
||||
|
||||
def complete(self, system, user, **kw):
|
||||
if self.complete_error:
|
||||
raise self.complete_error
|
||||
return self.complete_text
|
||||
|
||||
def complete_json(self, system, user, **kw):
|
||||
return self.json_payload
|
||||
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
class FakeDispatcher:
|
||||
def __init__(self, results=None):
|
||||
self.results = results or {}
|
||||
self.calls = []
|
||||
|
||||
def execute(self, manifest, arguments):
|
||||
tool = manifest.config.get("mcp", {}).get("tool")
|
||||
self.calls.append((tool, arguments))
|
||||
return self.results.get(tool, '{"ok": true, "data": {"id": "job-1"}}')
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 辅助函数
|
||||
# ===========================================================================
|
||||
|
||||
class TestChainToCapability:
|
||||
def test_known(self):
|
||||
assert ba._chain_to_capability("gene_family") == "bioflow-gene-family"
|
||||
assert ba._chain_to_capability("rnaseq") == "bioflow-rnaseq"
|
||||
|
||||
def test_unknown_and_empty(self):
|
||||
assert ba._chain_to_capability("wgs") is None
|
||||
assert ba._chain_to_capability("") is None
|
||||
assert ba._chain_to_capability(None) is None
|
||||
|
||||
def test_case_and_whitespace(self):
|
||||
assert ba._chain_to_capability(" Gene_Family ") == "bioflow-gene-family"
|
||||
|
||||
|
||||
class TestExtractJobId:
|
||||
def test_data_id(self):
|
||||
assert ba._extract_job_id('{"ok": true, "data": {"id": "job-9"}}') == "job-9"
|
||||
|
||||
def test_data_job_id(self):
|
||||
assert ba._extract_job_id('{"ok": true, "data": {"job_id": "job-8"}}') == "job-8"
|
||||
|
||||
def test_flat_id(self):
|
||||
assert ba._extract_job_id('{"ok": true, "id": "job-7"}') == "job-7"
|
||||
|
||||
def test_non_json_and_no_id(self):
|
||||
assert ba._extract_job_id("not json") is None
|
||||
assert ba._extract_job_id('{"ok": false}') is None
|
||||
assert ba._extract_job_id("") is None
|
||||
|
||||
|
||||
class TestExecuteTool:
|
||||
def test_unknown_tool(self):
|
||||
out = ba._execute_tool(_tool_manifests(), "ghost", {})
|
||||
assert "未注册" in out
|
||||
|
||||
def test_tool_exception_becomes_text(self):
|
||||
class Boom:
|
||||
def execute(self, manifest, arguments):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=Boom()):
|
||||
out = ba._execute_tool(_tool_manifests(), "job_status", {"job_id": "j"})
|
||||
assert "工具执行失败" in out
|
||||
|
||||
|
||||
class TestParaphrase:
|
||||
def test_llm_success(self):
|
||||
out = ba._paraphrase(FakeLLM(complete_text="已转述。"), "原文")
|
||||
assert out == "已转述。"
|
||||
|
||||
def test_llm_exception_falls_back_to_raw(self):
|
||||
out = ba._paraphrase(FakeLLM(complete_error=RuntimeError("boom")), "原文")
|
||||
assert out == "原文"
|
||||
|
||||
def test_llm_empty_falls_back_to_raw(self):
|
||||
out = ba._paraphrase(FakeLLM(complete_text=" "), "原文")
|
||||
assert out == "原文"
|
||||
|
||||
|
||||
class TestQueryJobStatus:
|
||||
def test_calls_job_status_and_paraphrases(self):
|
||||
with mock.patch.object(ba, "_execute_tool", return_value='{"ok": true, "data": {"status": "running"}}'):
|
||||
out = ba._query_job_status("j1", _tool_manifests(), FakeLLM(complete_text="任务进行中。"))
|
||||
assert out == "任务进行中。"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# card 四类 handler 边界(缺必需字段 / files 键 / archive=False / 非 dict payload)
|
||||
# ===========================================================================
|
||||
|
||||
class TestCardHandlersEdge:
|
||||
def test_form_confirm_missing_job_and_stage_returns_none(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "form_confirm", "payload": {"params": {}}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_file_upload_missing_job_returns_none(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "file_upload", "payload": {"paths": ["/data/a.fa"]}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_file_upload_missing_files_returns_none(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "file_upload", "payload": {"job_id": "j1"}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_file_upload_uses_files_key(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "file_upload",
|
||||
"payload": {"job_id": "j1", "files": ["https://x/a.fa"]}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "fetch_source_file"
|
||||
|
||||
def test_file_upload_string_files_coerced(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "file_upload",
|
||||
"payload": {"job_id": "j1", "files": "https://x/a.fa"}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "fetch_source_file"
|
||||
|
||||
def test_archive_choice_missing_job_returns_none(self):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "archive_choice", "payload": {"archive": True}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_archive_choice_skip(self):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "archive_choice", "payload": {"job_id": "j1", "archive": False}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert out["bioflow"]["audit"]["tool_calls"] == 0 # 存档不调工具(走 REST)
|
||||
|
||||
def test_path_selection_rnaseq_chain(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "rnaseq", "stages": ["rnaseq_qc"]}},
|
||||
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "create_rnaseq_job"
|
||||
|
||||
def test_card_non_dict_payload_treated_empty(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "form_confirm", "payload": "not-a-dict"},
|
||||
FakeLLM(), {"active_job_id": "j1"}, _tool_manifests(),
|
||||
)
|
||||
assert out is None # payload 非 dict → 无 stage/job_id → None
|
||||
|
||||
def test_card_result_with_non_dict_is_ignored(self):
|
||||
# try_bioflow_agent 对非 dict 的 card_result 不做 card 路由,走 NL 意图
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_candidate_capabilities", return_value=[]):
|
||||
out = ba.try_bioflow_agent("确认", llm_client=FakeLLM(), card_result="bogus")
|
||||
assert out is None
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# _route_card_result 边界
|
||||
# ===========================================================================
|
||||
|
||||
class TestRouteCardResultEdge:
|
||||
def test_chain_filters_to_capability(self):
|
||||
gf = _manifest("bioflow-gene-family")
|
||||
rn = _manifest("bioflow-rnaseq", "http://x/mcp/rnaseq/")
|
||||
captured = {}
|
||||
|
||||
def fake_build(manifests):
|
||||
captured["manifests"] = [m.name for m in manifests]
|
||||
return ([], {})
|
||||
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf, rn]):
|
||||
with mock.patch.object(ba, "_build_tools", side_effect=fake_build):
|
||||
ba._route_card_result(
|
||||
{"card": "form_confirm", "payload": {"chain": "rnaseq"}},
|
||||
FakeLLM(), None, None, None,
|
||||
)
|
||||
assert captured["manifests"] == ["bioflow-rnaseq"] # chain 过滤到 rnaseq 能力
|
||||
|
||||
def test_chain_to_unregistered_capability_returns_none(self):
|
||||
gf = _manifest("bioflow-gene-family")
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
|
||||
out = ba._route_card_result(
|
||||
{"card": "form_confirm", "payload": {"chain": "rnaseq"}},
|
||||
FakeLLM(), None, None, None,
|
||||
)
|
||||
assert out is None # rnaseq 未注册 → manifests 空 → None
|
||||
|
||||
def test_tools_empty_returns_none(self):
|
||||
gf = _manifest("bioflow-gene-family")
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([], {})):
|
||||
out = ba._route_card_result(
|
||||
{"card": "form_confirm", "payload": {}},
|
||||
FakeLLM(), None, None, None,
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_progress_callback_invoked(self):
|
||||
gf = _manifest("bioflow-gene-family")
|
||||
calls = []
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
|
||||
with mock.patch.object(ba, "_handle_card_result", return_value=None):
|
||||
ba._route_card_result(
|
||||
{"card": "form_confirm", "payload": {}},
|
||||
FakeLLM(), None, None, lambda p: calls.append(p),
|
||||
)
|
||||
assert calls and calls[0]["stage"] == "bioflow_card"
|
||||
assert calls[0]["card"] == "form_confirm"
|
||||
|
||||
def test_progress_callback_exception_swallowed(self):
|
||||
gf = _manifest("bioflow-gene-family")
|
||||
|
||||
def boom(p):
|
||||
raise RuntimeError("callback boom")
|
||||
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
|
||||
with mock.patch.object(ba, "_handle_card_result", return_value={"ok": True}):
|
||||
out = ba._route_card_result(
|
||||
{"card": "form_confirm", "payload": {}},
|
||||
FakeLLM(), None, None, boom,
|
||||
)
|
||||
assert out == {"ok": True} # 回调异常被吞,不影响主流程
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# NL 路径:bioflow 上下文注入 / 卡片与 NL 语义等价
|
||||
# ===========================================================================
|
||||
|
||||
class TestNLPathBioflowContext:
|
||||
def test_nl_path_injects_bioflow_context(self, tmp_path):
|
||||
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry
|
||||
from rag.qa_v1_1_73.capabilities.manifests import parse_capabilities_text
|
||||
yaml = """
|
||||
capabilities:
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
session_default: true
|
||||
nl_triggers: [基因家族]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/gene-family/
|
||||
"""
|
||||
(tmp_path / "c.yaml").write_text(yaml, encoding="utf-8")
|
||||
registry = CapabilityRegistry(tmp_path / "c.yaml")
|
||||
registry.load()
|
||||
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
|
||||
"description": "x",
|
||||
"parameters": {"type": "object", "properties": {}}}}]
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
|
||||
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
|
||||
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
|
||||
out = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLM(),
|
||||
bioflow={"session_id": "s1", "active_job_id": "job-9", "chain": "gene_family"},
|
||||
)
|
||||
assert out is not None
|
||||
assert out["bioflow"]["active_job_id"] == "job-9"
|
||||
assert out["bioflow"]["chain"] == "gene_family"
|
||||
|
||||
def test_card_and_nl_share_tool_chain(self):
|
||||
"""卡片操作与纯 NL 服务端语义等价:卡片 handler 只调 MCP 工具,不直连控制面。"""
|
||||
# 卡片 handler 用到的工具名 = MCP 工具(job_status/confirm_stage/create_* 等)
|
||||
tm = _tool_manifests()
|
||||
# 四类卡片触发的工具都在 tool_manifests(即 MCP 工具表)里
|
||||
for tool in ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
|
||||
"register_job_files", "fetch_source_file", "job_status"):
|
||||
assert tool in tm
|
||||
|
||||
|
||||
class TestExtractParamsEdge:
|
||||
def test_llm_exception_returns_empty(self):
|
||||
class RaisingLLM(FakeLLM):
|
||||
def complete_json(self, system, user, **kw):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
known, unknown = ba._extract_params(RaisingLLM(), "x", {"threads": {}})
|
||||
assert known == {} and unknown == []
|
||||
|
||||
def test_unknown_as_string_coerced_to_list(self):
|
||||
llm = FakeLLM(json_payload={"params": {"threads": 8}, "unknown": "bogus_param"})
|
||||
known, unknown = ba._extract_params(llm, "x", {"threads": {}})
|
||||
assert known == {"threads": 8}
|
||||
assert unknown == ["bogus_param"]
|
||||
|
||||
def test_path_selection_with_params(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "gene_family", "stages": ["identify"],
|
||||
"params": {"prefix": "Ft.SPL"}}},
|
||||
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
|
||||
)
|
||||
assert dispatcher.calls[0][1]["params"] == {"prefix": "Ft.SPL"}
|
||||
|
||||
def test_path_selection_create_tool_missing_returns_none(self):
|
||||
tm = _tool_manifests()
|
||||
tm.pop("create_gene_family_job")
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "gene_family", "stages": ["identify"]}},
|
||||
FakeLLM(), {"session_id": "s1"}, tm,
|
||||
)
|
||||
assert out is None
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# system prompt 硬规则在位
|
||||
# ===========================================================================
|
||||
|
||||
class TestSystemPromptRules:
|
||||
def test_param_retry_rule(self):
|
||||
assert "最多 3 次" in ba._SYSTEM_BASE
|
||||
|
||||
def test_human_boundary_rule(self):
|
||||
assert "不可跳过" in ba._SYSTEM_BASE
|
||||
assert "不可代用户确认" in ba._SYSTEM_BASE
|
||||
|
||||
def test_tool_name_no_leak_rule(self):
|
||||
assert "绝不向用户泄露工具名" in ba._SYSTEM_BASE
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# ToolRunner.run(history) 边界
|
||||
# ===========================================================================
|
||||
|
||||
class TestHistoryEdge:
|
||||
def _run_with_history(self, history):
|
||||
captured = {}
|
||||
|
||||
class RecordingLLM:
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
captured["messages"] = list(messages)
|
||||
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
runner = ToolRunner(make_llm_complete(RecordingLLM()), max_rounds=1)
|
||||
runner.run("sys", "当前", [], {}, history=history)
|
||||
return captured["messages"]
|
||||
|
||||
def test_none_history(self):
|
||||
msgs = self._run_with_history(None)
|
||||
assert msgs == [{"role": "user", "content": "当前"}]
|
||||
|
||||
def test_non_dict_items_skipped(self):
|
||||
msgs = self._run_with_history(["not-a-dict", {"role": "user", "content": "ok"}])
|
||||
assert msgs == [{"role": "user", "content": "ok"}, {"role": "user", "content": "当前"}]
|
||||
|
||||
def test_empty_content_filtered(self):
|
||||
msgs = self._run_with_history([{"role": "user", "content": " "}])
|
||||
assert msgs == [{"role": "user", "content": "当前"}]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S3:card_result rerun / reuse_archive / archive_job 语义
|
||||
# ===========================================================================
|
||||
|
||||
class TestS3CardSemantics:
|
||||
def test_form_confirm_rerun_true_uses_rerun_stage(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "form_confirm",
|
||||
"payload": {"job_id": "j1", "stage": "identify",
|
||||
"params": {"evalue": 1e-5}, "rerun": True}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "rerun_stage"
|
||||
assert out["bioflow"]["audit"]["tool_names"] == ["rerun_stage"]
|
||||
|
||||
def test_form_confirm_rerun_false_uses_confirm_stage(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
ba._handle_card_result(
|
||||
{"card": "form_confirm",
|
||||
"payload": {"job_id": "j1", "stage": "identify", "params": {}}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert dispatcher.calls[0][0] == "confirm_stage"
|
||||
|
||||
def test_path_selection_reuse_archive(self):
|
||||
dispatcher = FakeDispatcher({
|
||||
"reuse_archive": '{"ok": true, "data": {"id": "arch-1", '
|
||||
'"stage_chain": ["identify", "phylogeny"], '
|
||||
'"params": {"identify": {"prefix": "Ft"}}}}',
|
||||
})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "gene_family", "reuse_archive": "SPL 鉴定"}},
|
||||
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
tools = [c[0] for c in dispatcher.calls]
|
||||
assert tools[0] == "reuse_archive"
|
||||
assert tools[1] == "create_gene_family_job"
|
||||
create_args = dispatcher.calls[1][1]
|
||||
assert create_args["archive_id"] == "arch-1"
|
||||
assert create_args["stages"] == ["identify", "phylogeny"]
|
||||
assert create_args["params"] == {"identify": {"prefix": "Ft"}}
|
||||
assert out["bioflow"]["audit"]["tool_names"] == ["reuse_archive", "create_gene_family_job"]
|
||||
|
||||
def test_archive_choice_archive_true_calls_archive_job(self):
|
||||
dispatcher = FakeDispatcher({"archive_job": '{"ok": true, "data": {"id": "a1"}}'})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "archive_choice",
|
||||
"payload": {"job_id": "j1", "archive": True,
|
||||
"name": "SPL 存档", "description": "d"}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "archive_job"
|
||||
assert dispatcher.calls[0][1] == {"job_id": "j1", "name": "SPL 存档", "description": "d"}
|
||||
assert out["bioflow"]["audit"]["tool_names"] == ["archive_job"]
|
||||
|
||||
def test_archive_choice_archive_false_no_tool_call(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "archive_choice",
|
||||
"payload": {"job_id": "j1", "archive": False}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls == [] # 跳过存档不发工具请求
|
||||
assert out["bioflow"]["audit"]["tool_calls"] == 0
|
||||
@@ -0,0 +1,206 @@
|
||||
"""test_bioflow_agent_s4_supplement.py —— S4 覆盖清单补强测试。
|
||||
|
||||
覆盖:
|
||||
- _card_extension_action 六种 action 分发到对应扩展工具
|
||||
- 未知 action / 缺 ext_id/action 回退 None
|
||||
- confirm 携带 nl_triggers/parameters/runtime_image;EXT_STATUS 降级话术(确定性)
|
||||
- run 携带 params/files
|
||||
- 真实 configs/capabilities.yaml 三条目(含 bioflow-extension)可解析
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest, parse_capabilities_file # noqa: E402
|
||||
|
||||
EXT_TOOLS = (
|
||||
"create_extension", "extension_status", "build_extension", "trial_run_extension",
|
||||
"validate_extension", "confirm_extension", "publish_extension", "run_extension",
|
||||
"repair_extension",
|
||||
)
|
||||
|
||||
|
||||
def _ext_tool_manifests():
|
||||
out = {}
|
||||
for t in EXT_TOOLS:
|
||||
out[t] = CapabilityManifest(
|
||||
name="bioflow-extension", provider="mcp",
|
||||
config={"mcp": {"url": "http://127.0.0.1:8098/mcp/extension/", "tool": t}},
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
class FakeLLM:
|
||||
def complete(self, system, user, **kw):
|
||||
return "已收到。"
|
||||
|
||||
def complete_json(self, system, user, **kw):
|
||||
return {}
|
||||
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
class FakeDispatcher:
|
||||
def __init__(self, results=None):
|
||||
self.results = results or {}
|
||||
self.calls = []
|
||||
|
||||
def execute(self, manifest, arguments):
|
||||
tool = manifest.config.get("mcp", {}).get("tool")
|
||||
self.calls.append((tool, arguments))
|
||||
return self.results.get(tool, '{"ok": true, "data": {"ext_id": "e1"}}')
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# _card_extension_action 分发
|
||||
# ===========================================================================
|
||||
|
||||
class TestExtensionAction:
|
||||
@pytest.mark.parametrize("action,expected_tool", [
|
||||
("build", "build_extension"),
|
||||
("trial_run", "trial_run_extension"),
|
||||
("validate", "validate_extension"),
|
||||
("confirm", "confirm_extension"),
|
||||
("publish", "publish_extension"),
|
||||
("run", "run_extension"),
|
||||
("repair", "repair_extension"),
|
||||
])
|
||||
def test_action_dispatches_to_tool(self, action, expected_tool):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1", "action": action}},
|
||||
FakeLLM(), {}, _ext_tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == expected_tool
|
||||
assert dispatcher.calls[0][1] == {"ext_id": "e1"}
|
||||
assert out["bioflow"]["audit"]["tool_names"] == [expected_tool]
|
||||
|
||||
def test_unknown_action_returns_none(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "bogus"}},
|
||||
FakeLLM(), {}, _ext_tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_missing_ext_id_or_action_returns_none(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
assert ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"action": "build"}},
|
||||
FakeLLM(), {}, _ext_tool_manifests(),
|
||||
) is None
|
||||
assert ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1"}},
|
||||
FakeLLM(), {}, _ext_tool_manifests(),
|
||||
) is None
|
||||
|
||||
def test_confirm_passes_triggers_params_image(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
payload = {"ext_id": "e1", "action": "confirm",
|
||||
"nl_triggers": ["扩展", "触发词"], "parameters": [{"name": "x"}],
|
||||
"runtime_image": "img"}
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
ba._handle_card_result({"card": "extension_action", "payload": payload},
|
||||
FakeLLM(), {}, _ext_tool_manifests())
|
||||
args = dispatcher.calls[0][1]
|
||||
assert args == {"ext_id": "e1", "nl_triggers": ["扩展", "触发词"],
|
||||
"parameters": [{"name": "x"}], "runtime_image": "img"}
|
||||
|
||||
def test_confirm_status_insufficient_uses_deterministic_script(self):
|
||||
# confirm 返回 EXT_STATUS(状态不足)→ 确定性降级话术(不报 500、不经 LLM)
|
||||
dispatcher = FakeDispatcher({
|
||||
"confirm_extension": '{"ok": false, "code": "EXT_STATUS", "message": "状态不可确认"}',
|
||||
})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "confirm"}},
|
||||
FakeLLM(), {}, _ext_tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert "还不能确认" in out["answer"]
|
||||
assert "先完成" in out["answer"] # 确定性话术(非 LLM 转述)
|
||||
|
||||
def test_run_passes_params_and_files(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
payload = {"ext_id": "e1", "action": "run",
|
||||
"params": {"cpu": 8}, "files": ["/data/a.fa"]}
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
ba._handle_card_result({"card": "extension_action", "payload": payload},
|
||||
FakeLLM(), {}, _ext_tool_manifests())
|
||||
assert dispatcher.calls[0][1] == {"ext_id": "e1", "params": {"cpu": 8}, "files": ["/data/a.fa"]}
|
||||
|
||||
def test_extension_tool_not_in_manifests_returns_none(self):
|
||||
# tool_manifests 里没有扩展工具 → 回退 None(能力未安装/未发现工具)
|
||||
empty = CapabilityManifest(name="bioflow-gene-family", provider="mcp",
|
||||
config={"mcp": {"url": "http://x", "tool": "job_status"}})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "build"}},
|
||||
FakeLLM(), {}, {"job_status": empty},
|
||||
)
|
||||
assert out is None
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 真实 capabilities.yaml 三条目可解析
|
||||
# ===========================================================================
|
||||
|
||||
class TestCapabilitiesThreeEntries:
|
||||
def _real_yaml(self) -> Path:
|
||||
return Path(__file__).resolve().parents[5] / "agent" / "qa" / "configs" / "capabilities.yaml"
|
||||
|
||||
def test_three_bioflow_entries(self):
|
||||
manifests, warnings = parse_capabilities_file(self._real_yaml())
|
||||
by_name = {m.name: m for m in manifests}
|
||||
assert {"bioflow-gene-family", "bioflow-rnaseq", "bioflow-extension"} <= set(by_name)
|
||||
|
||||
ext = by_name["bioflow-extension"]
|
||||
assert ext.provider == "mcp"
|
||||
assert ext.permission == "read_only"
|
||||
assert ext.nl_triggers
|
||||
assert ext.config["mcp"]["url"].endswith("/mcp/extension/")
|
||||
assert ext.config["prompt_pack"]["file"] == "bioflow_extension_agent_rules.md"
|
||||
|
||||
def test_bioflow_extension_in_capability_names(self):
|
||||
assert "bioflow-extension" in ba.BIOFLOW_CAPABILITY_NAMES
|
||||
assert ba.CARD_EXTENSION_ACTION == "extension_action"
|
||||
assert ba.CARD_EXTENSION_ACTION in ba.CARD_TYPES
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S5:_route_card_result 对 extension_action 只发现 extension 能力(一次 list_tools)
|
||||
# ===========================================================================
|
||||
|
||||
class TestRouteExtensionOptimization:
|
||||
def test_extension_action_discovers_only_extension_capability(self):
|
||||
gf = CapabilityManifest(name="bioflow-gene-family", provider="mcp",
|
||||
config={"mcp": {"url": "http://x/gene-family/"}})
|
||||
rn = CapabilityManifest(name="bioflow-rnaseq", provider="mcp",
|
||||
config={"mcp": {"url": "http://x/rnaseq/"}})
|
||||
ext = CapabilityManifest(name="bioflow-extension", provider="mcp",
|
||||
config={"mcp": {"url": "http://x/extension/"}})
|
||||
captured = {}
|
||||
|
||||
def fake_build(manifests):
|
||||
captured["names"] = [m.name for m in manifests]
|
||||
return ([], {})
|
||||
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf, rn, ext]):
|
||||
with mock.patch.object(ba, "_build_tools", side_effect=fake_build):
|
||||
ba._route_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "build"}},
|
||||
FakeLLM(), None, None, None,
|
||||
)
|
||||
# 只把 bioflow-extension manifest 交给 _build_tools(避免对三能力各 list_tools 一次)
|
||||
assert captured["names"] == ["bioflow-extension"]
|
||||
@@ -0,0 +1,116 @@
|
||||
"""test_bioflow_agent_s6_config.py —— S6 多配置回归(QA 单元层)。
|
||||
|
||||
模拟四种部署配置,断言候选能力集合 + 意图预筛结果正确(真实 E2E/金标 20 题依赖部署,
|
||||
此处以 monkeypatch/env 在单元层覆盖漏判安全语义)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
|
||||
|
||||
THREE_YAML = """
|
||||
capabilities:
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
session_default: true
|
||||
nl_triggers: [基因家族, 亚家族]
|
||||
mcp: {url: "http://x/gene-family/"}
|
||||
- name: bioflow-rnaseq
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
session_default: true
|
||||
nl_triggers: [转录组, 差异表达]
|
||||
mcp: {url: "http://x/rnaseq/"}
|
||||
- name: bioflow-extension
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
session_default: true
|
||||
nl_triggers: [自定义扩展]
|
||||
mcp: {url: "http://x/extension/"}
|
||||
"""
|
||||
|
||||
|
||||
def _registry(tmp_path, yaml_text=THREE_YAML):
|
||||
path = tmp_path / "caps.yaml"
|
||||
path.write_text(yaml_text, encoding="utf-8")
|
||||
reg = CapabilityRegistry(path)
|
||||
reg.load()
|
||||
return reg
|
||||
|
||||
|
||||
class FakeLLM:
|
||||
def complete_raw_messages(self, *a, **kw):
|
||||
return {"content": "x", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 配置 1:全关(QA_V1_ENABLE_BIOFLOW_AGENT=0 + 三能力全卸载)
|
||||
# ===========================================================================
|
||||
|
||||
class TestConfigAllOff:
|
||||
def test_fuse_off_returns_none(self, tmp_path):
|
||||
reg = _registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=reg):
|
||||
with mock.patch.dict(ba.os.environ, {}, clear=False):
|
||||
ba.os.environ.pop(ba.ENV_FUSE, None)
|
||||
out = ba.try_bioflow_agent("帮我做基因家族分析", llm_client=FakeLLM())
|
||||
assert out is None # 走原 pipeline
|
||||
|
||||
def test_all_uninstalled_no_candidate(self, tmp_path):
|
||||
reg = _registry(tmp_path, "capabilities: []\n") # 三能力全卸载
|
||||
with mock.patch.object(ba, "get_registry", return_value=reg):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
assert ba._candidate_capabilities("帮我做基因家族分析", None) == []
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 配置 2:只开基因家族
|
||||
# ===========================================================================
|
||||
|
||||
class TestConfigGeneFamilyOnly:
|
||||
def test_gene_family_hit_rnaseq_miss(self, tmp_path):
|
||||
reg = _registry(tmp_path)
|
||||
caps = {"bioflow-gene-family": True, "bioflow-rnaseq": False, "bioflow-extension": False}
|
||||
with mock.patch.object(ba, "get_registry", return_value=reg):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
names = [m.name for m in ba._candidate_capabilities("帮我做水稻 SPL 基因家族分析", caps)]
|
||||
assert names == ["bioflow-gene-family"]
|
||||
assert ba._candidate_capabilities("帮我做转录组差异表达分析", caps) == []
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 配置 3:只开转录组
|
||||
# ===========================================================================
|
||||
|
||||
class TestConfigRnaseqOnly:
|
||||
def test_rnaseq_hit_gene_family_miss(self, tmp_path):
|
||||
reg = _registry(tmp_path)
|
||||
caps = {"bioflow-gene-family": False, "bioflow-rnaseq": True, "bioflow-extension": False}
|
||||
with mock.patch.object(ba, "get_registry", return_value=reg):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
names = [m.name for m in ba._candidate_capabilities("转录组差异表达分析", caps)]
|
||||
assert names == ["bioflow-rnaseq"]
|
||||
assert ba._candidate_capabilities("帮我做水稻 SPL 基因家族分析", caps) == []
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 配置 4:全开但非生信提问
|
||||
# ===========================================================================
|
||||
|
||||
class TestConfigAllOnNonBioflow:
|
||||
def test_non_bioflow_question_no_candidate(self, tmp_path):
|
||||
reg = _registry(tmp_path)
|
||||
caps = {"bioflow-gene-family": True, "bioflow-rnaseq": True, "bioflow-extension": True}
|
||||
with mock.patch.object(ba, "get_registry", return_value=reg):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
assert ba._candidate_capabilities("光合作用的电子传递链是什么", caps) == []
|
||||
assert ba._candidate_capabilities("什么是转录因子", caps) == []
|
||||
@@ -0,0 +1,496 @@
|
||||
"""test_bioflow_agent_supplement.py —— S1 覆盖清单补强测试。
|
||||
|
||||
补强 dev 的 test_bioflow_agent.py 未覆盖的分支(对照 S1 覆盖清单):
|
||||
- 熔断开关值解析(1/true/yes/on 与 0/false/off/空/缺失)
|
||||
- nl_triggers 同时命中 → 两个候选(交 LLM 依 description 选)
|
||||
- session_default 回退(True/False)/ registry 访问异常 / registry.get 异常
|
||||
- _build_tools:MCP 工具发现(空 url 跳过 / 发现异常 / 共享连接按能力名 / 无名跳过 / bad schema 兜底)
|
||||
- _build_system:prompt_pack 注入 / 去重 / 加载异常兜底
|
||||
- try_bioflow_agent:tools 空→None、answer 空→None、progress_callback 触发与吞异常
|
||||
- ToolRunner max_rounds=5 / bioflow_agent.MAX_ROUNDS 常量
|
||||
- make_llm_complete 包装(正常 / 异常)
|
||||
- 真实 configs/capabilities.yaml 两条目可解析
|
||||
- bioflow_agent 不 import pipeline/answer_composer(红线静态守卫)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.manifests import ( # noqa: E402
|
||||
CapabilityManifest,
|
||||
parse_capabilities_file,
|
||||
)
|
||||
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
|
||||
|
||||
BIOFLOW_YAML = """
|
||||
capabilities:
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 基因家族分析
|
||||
session_default: true
|
||||
nl_triggers: [基因家族, 家族成员, 亚家族, 建树]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/gene-family/
|
||||
prompt_pack:
|
||||
file: bioflow_gene_family_agent_rules.md
|
||||
- name: bioflow-rnaseq
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 转录组分析
|
||||
session_default: true
|
||||
nl_triggers: [转录组, RNA-seq, 差异表达, 富集]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/rnaseq/
|
||||
prompt_pack:
|
||||
file: bioflow_rnaseq_agent_rules.md
|
||||
"""
|
||||
|
||||
|
||||
def _make_registry(tmp_path: Path, yaml_text: str = BIOFLOW_YAML) -> CapabilityRegistry:
|
||||
path = tmp_path / "capabilities.yaml"
|
||||
path.write_text(yaml_text, encoding="utf-8")
|
||||
registry = CapabilityRegistry(path)
|
||||
registry.load()
|
||||
return registry
|
||||
|
||||
|
||||
def _gf_manifest(**overrides) -> CapabilityManifest:
|
||||
cfg = {
|
||||
"mcp": {"url": "http://127.0.0.1:8098/mcp/gene-family/"},
|
||||
"prompt_pack": {"file": "bioflow_gene_family_agent_rules.md"},
|
||||
}
|
||||
cfg.update(overrides.pop("config", {}))
|
||||
return CapabilityManifest(
|
||||
name="bioflow-gene-family", provider="mcp", permission="read_only",
|
||||
description="基因家族分析", config=cfg, nl_triggers=["基因家族"], **overrides,
|
||||
)
|
||||
|
||||
|
||||
class FakeLLMClient:
|
||||
def __init__(self, content: str = "已为您规划分析路径。") -> None:
|
||||
self.content = content
|
||||
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kwargs):
|
||||
return {"content": self.content, "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 门 1:熔断开关值解析
|
||||
# ===========================================================================
|
||||
|
||||
class TestEnvFuse:
|
||||
@pytest.mark.parametrize("value", ["1", "true", "TRUE", "yes", "on", " On ", "YES"])
|
||||
def test_truthy(self, value):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: value}):
|
||||
assert ba._env_enabled() is True
|
||||
|
||||
@pytest.mark.parametrize("value", ["0", "false", "off", "", "no", " "])
|
||||
def test_falsy(self, value):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: value}):
|
||||
assert ba._env_enabled() is False
|
||||
|
||||
def test_missing_defaults_off(self):
|
||||
with mock.patch.dict(ba.os.environ, {}, clear=False):
|
||||
ba.os.environ.pop(ba.ENV_FUSE, None)
|
||||
assert ba._env_enabled() is False
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 门 3:nl_triggers 意图预筛
|
||||
# ===========================================================================
|
||||
|
||||
class TestIntentMatchEdge:
|
||||
def test_empty_triggers_never_match(self):
|
||||
assert ba._intent_match("做基因家族分析", []) is False
|
||||
assert ba._intent_match("做基因家族分析", [""]) is False
|
||||
|
||||
def test_case_insensitive_substring(self):
|
||||
assert ba._intent_match("RNA-SEQ 分析", ["rna-seq"]) is True
|
||||
assert ba._intent_match("ka/ks 选择压力", ["Ka/Ks"]) is True
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 门 2 + 门 3:候选能力(registry 异常 / session_default 回退 / 同时命中)
|
||||
# ===========================================================================
|
||||
|
||||
class TestCandidateCapabilities:
|
||||
def test_registry_access_exception_returns_empty(self):
|
||||
with mock.patch.object(ba, "get_registry", side_effect=RuntimeError("boom")):
|
||||
assert ba._candidate_capabilities("做基因家族", None) == []
|
||||
|
||||
def test_registry_get_exception_skips(self):
|
||||
class BadRegistry:
|
||||
def get(self, name):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
with mock.patch.object(ba, "get_registry", return_value=BadRegistry()):
|
||||
assert ba._candidate_capabilities("做基因家族", None) == []
|
||||
|
||||
def test_both_triggers_match_returns_both(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
caps = ba._candidate_capabilities("做基因家族和转录组差异表达分析", None)
|
||||
assert [m.name for m in caps] == ["bioflow-gene-family", "bioflow-rnaseq"]
|
||||
|
||||
def test_session_default_false_skips(self, tmp_path):
|
||||
yaml = BIOFLOW_YAML.replace("session_default: true", "session_default: false")
|
||||
registry = _make_registry(tmp_path, yaml)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
# 无会话开关时回退 session_default=false → 不候选
|
||||
assert ba._candidate_capabilities("做基因家族分析", None) == []
|
||||
# 显式会话开关 true → 候选
|
||||
caps = ba._candidate_capabilities("做基因家族分析", {"bioflow-gene-family": True})
|
||||
assert [m.name for m in caps] == ["bioflow-gene-family"]
|
||||
|
||||
def test_session_switch_true_overrides_default_false(self, tmp_path):
|
||||
yaml = BIOFLOW_YAML.replace("session_default: true", "session_default: false", 1)
|
||||
registry = _make_registry(tmp_path, yaml)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
caps = ba._candidate_capabilities("做基因家族分析", {"bioflow-gene-family": True})
|
||||
assert [m.name for m in caps] == ["bioflow-gene-family"]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# _build_tools:MCP 工具发现
|
||||
# ===========================================================================
|
||||
|
||||
class _FakeDispatcher:
|
||||
def __init__(self, tools_by_name=None, exc=None):
|
||||
self.tools_by_name = tools_by_name or {}
|
||||
self.exc = exc
|
||||
|
||||
def list_mcp_tools(self, manifest):
|
||||
if self.exc:
|
||||
raise self.exc
|
||||
return self.tools_by_name.get(manifest.name, [])
|
||||
|
||||
|
||||
class TestBuildTools:
|
||||
def test_empty_url_skipped(self):
|
||||
manifest = CapabilityManifest(
|
||||
name="x", provider="mcp", config={"mcp": {"url": ""}}
|
||||
)
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=_FakeDispatcher()):
|
||||
tools, tool_manifests = ba._build_tools([manifest])
|
||||
assert tools == []
|
||||
assert tool_manifests == {}
|
||||
|
||||
def test_discovery_exception_skipped(self):
|
||||
manifest = _gf_manifest()
|
||||
with mock.patch.object(
|
||||
ba, "get_dispatcher",
|
||||
return_value=_FakeDispatcher(exc=RuntimeError("mcp down")),
|
||||
):
|
||||
tools, tool_manifests = ba._build_tools([manifest])
|
||||
assert tools == []
|
||||
assert tool_manifests == {}
|
||||
|
||||
def test_builds_schema_and_shared_connection(self):
|
||||
dispatcher = _FakeDispatcher({
|
||||
"bioflow-gene-family": [
|
||||
{"name": "plan_gene_family", "description": "规划路径",
|
||||
"input_schema": {"type": "object", "properties": {"message": {"type": "string"}}}},
|
||||
]
|
||||
})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
tools, tool_manifests = ba._build_tools([_gf_manifest()])
|
||||
assert len(tools) == 1
|
||||
assert tools[0]["function"]["name"] == "plan_gene_family"
|
||||
assert tools[0]["function"]["description"] == "规划路径"
|
||||
# 共享连接:manifest.name 用能力名,mcp.tool 指向真实 server 工具名
|
||||
tm = tool_manifests["plan_gene_family"]
|
||||
assert tm.name == "bioflow-gene-family"
|
||||
assert tm.config["mcp"]["tool"] == "plan_gene_family"
|
||||
assert tm.config["mcp"]["url"].endswith("/mcp/gene-family/")
|
||||
|
||||
def test_skips_nameless_and_bad_schema(self):
|
||||
dispatcher = _FakeDispatcher({
|
||||
"bioflow-gene-family": [
|
||||
{"name": "", "description": "无名", "input_schema": {}},
|
||||
{"name": "job_status", "description": "查状态", "input_schema": "not-a-dict"},
|
||||
]
|
||||
})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
tools, tool_manifests = ba._build_tools([_gf_manifest()])
|
||||
assert [t["function"]["name"] for t in tools] == ["job_status"]
|
||||
assert tools[0]["function"]["parameters"] == {"type": "object", "properties": {}}
|
||||
|
||||
def test_shared_tool_deduped_across_capabilities(self):
|
||||
"""两能力同时命中:共享工具去重(只保留首个能力),专有工具各自保留。
|
||||
|
||||
覆盖 bug ① 修复:job_status 等 12 个共享工具两端点语义一致,按名去重后
|
||||
LLM tools 列表无重名,且 job_status 路由到首个能力(gene-family)。
|
||||
"""
|
||||
dispatcher = _FakeDispatcher({
|
||||
"bioflow-gene-family": [
|
||||
{"name": "plan_gene_family", "description": "规划基因家族", "input_schema": {}},
|
||||
{"name": "job_status", "description": "查状态", "input_schema": {}},
|
||||
{"name": "confirm_stage", "description": "确认参数", "input_schema": {}},
|
||||
],
|
||||
"bioflow-rnaseq": [
|
||||
{"name": "plan_rnaseq", "description": "规划转录组", "input_schema": {}},
|
||||
{"name": "job_status", "description": "查状态", "input_schema": {}},
|
||||
{"name": "confirm_stage", "description": "确认参数", "input_schema": {}},
|
||||
],
|
||||
})
|
||||
rn = CapabilityManifest(
|
||||
name="bioflow-rnaseq", provider="mcp", permission="read_only",
|
||||
description="转录组分析", nl_triggers=["转录组"],
|
||||
config={"mcp": {"url": "http://127.0.0.1:8098/mcp/rnaseq/"}},
|
||||
)
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
tools, tool_manifests = ba._build_tools([_gf_manifest(), rn])
|
||||
|
||||
names = [t["function"]["name"] for t in tools]
|
||||
# 专有工具各保留 + 共享工具只保留一份,且无重名
|
||||
assert names == ["plan_gene_family", "job_status", "confirm_stage", "plan_rnaseq"]
|
||||
assert len(names) == len(set(names))
|
||||
|
||||
# 共享工具路由到首个能力(gene-family)
|
||||
assert tool_manifests["job_status"].name == "bioflow-gene-family"
|
||||
assert tool_manifests["job_status"].config["mcp"]["url"].endswith("/mcp/gene-family/")
|
||||
assert tool_manifests["confirm_stage"].name == "bioflow-gene-family"
|
||||
# 专有工具各自路由到自身能力
|
||||
assert tool_manifests["plan_rnaseq"].name == "bioflow-rnaseq"
|
||||
assert tool_manifests["plan_rnaseq"].config["mcp"]["url"].endswith("/mcp/rnaseq/")
|
||||
assert tool_manifests["plan_gene_family"].name == "bioflow-gene-family"
|
||||
|
||||
# ===========================================================================
|
||||
# _build_system:prompt_pack 注入
|
||||
# ===========================================================================
|
||||
|
||||
class TestBuildSystem:
|
||||
def test_appends_prompt_pack(self):
|
||||
manifest = _gf_manifest()
|
||||
with mock.patch.object(ba.PromptPackProvider, "load", return_value="基因家族规则正文"):
|
||||
system = ba._build_system([manifest])
|
||||
assert ba._SYSTEM_BASE in system
|
||||
assert "基因家族规则正文" in system
|
||||
|
||||
def test_dedup_same_file_once(self):
|
||||
m1 = _gf_manifest()
|
||||
m2 = _gf_manifest() # 同 file
|
||||
calls = []
|
||||
with mock.patch.object(ba.PromptPackProvider, "load", side_effect=lambda m, root: calls.append(m) or "RULES"):
|
||||
system = ba._build_system([m1, m2])
|
||||
assert len(calls) == 1 # 去重后只加载一次
|
||||
assert system.count("RULES") == 1
|
||||
|
||||
def test_empty_file_skipped(self):
|
||||
manifest = CapabilityManifest(name="x", provider="mcp", config={"prompt_pack": {"file": ""}})
|
||||
system = ba._build_system([manifest])
|
||||
assert system == ba._SYSTEM_BASE
|
||||
|
||||
def test_load_exception_falls_back(self):
|
||||
manifest = _gf_manifest()
|
||||
with mock.patch.object(ba.PromptPackProvider, "load", side_effect=RuntimeError("boom")):
|
||||
system = ba._build_system([manifest])
|
||||
assert ba._SYSTEM_BASE in system # 不炸,只用基础提示
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# try_bioflow_agent:tools 空 / answer 空 / progress_callback
|
||||
# ===========================================================================
|
||||
|
||||
class TestFullFlowEdge:
|
||||
def test_tools_empty_returns_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([], {})):
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_answer_empty_returns_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
|
||||
"description": "x",
|
||||
"parameters": {"type": "object", "properties": {}}}}]
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
|
||||
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
|
||||
FakeRunner.return_value.run.return_value = ("", {"rounds": 1})
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_progress_callback_invoked(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
|
||||
"description": "x",
|
||||
"parameters": {"type": "object", "properties": {}}}}]
|
||||
calls = []
|
||||
|
||||
def cb(payload):
|
||||
calls.append(payload)
|
||||
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
|
||||
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
|
||||
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(),
|
||||
capabilities=None, progress_callback=cb,
|
||||
)
|
||||
assert result is not None
|
||||
assert calls and calls[0]["stage"] == "bioflow_agent_started"
|
||||
assert calls[0]["capabilities"] == ["bioflow-gene-family"]
|
||||
assert calls[0]["tool_count"] == 1
|
||||
|
||||
def test_progress_callback_exception_swallowed(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
|
||||
"description": "x",
|
||||
"parameters": {"type": "object", "properties": {}}}}]
|
||||
|
||||
def cb(payload):
|
||||
raise RuntimeError("callback boom")
|
||||
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
|
||||
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
|
||||
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(),
|
||||
capabilities=None, progress_callback=cb,
|
||||
)
|
||||
assert result is not None # 回调异常被吞,不影响主流程
|
||||
assert result["answer"] == "已答复"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# ToolRunner max_rounds=5 / MAX_ROUNDS 常量
|
||||
# ===========================================================================
|
||||
|
||||
class TestMaxRounds:
|
||||
def test_max_rounds_constant_is_5(self):
|
||||
assert ba.MAX_ROUNDS == 5
|
||||
|
||||
def test_tool_runner_respects_max_rounds_5(self):
|
||||
from rag.qa_v1_1_73.tool_runner import ToolRunner
|
||||
|
||||
class LLM:
|
||||
def __call__(self, system, messages, tools):
|
||||
return "", [{"id": "c", "name": "echo", "arguments": "{}"}], None
|
||||
|
||||
class Dispatcher:
|
||||
def execute(self, manifest, arguments):
|
||||
return "ok"
|
||||
|
||||
manifest = CapabilityManifest(name="echo", provider="python")
|
||||
tools = [{"type": "function", "function": {"name": "echo", "description": "e",
|
||||
"parameters": {"type": "object", "properties": {}}}}]
|
||||
runner = ToolRunner(LLM(), dispatcher=Dispatcher(), max_rounds=5)
|
||||
content, audit = runner.run("sys", "user", tools, {"echo": manifest})
|
||||
assert audit["rounds"] == 5
|
||||
assert audit["tool_calls"] == 5
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# make_llm_complete 包装
|
||||
# ===========================================================================
|
||||
|
||||
class TestMakeLLMComplete:
|
||||
def test_normal(self):
|
||||
from rag.qa_v1_1_73.tool_runner import make_llm_complete
|
||||
|
||||
class Client:
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
return {"content": "hi", "tool_calls": [{"name": "x"}], "finish_reason": "stop"}
|
||||
|
||||
complete = make_llm_complete(Client())
|
||||
content, tool_calls, error = complete("sys", [{"role": "user", "content": "q"}], [])
|
||||
assert content == "hi"
|
||||
assert tool_calls == [{"name": "x"}]
|
||||
assert error is None
|
||||
|
||||
def test_exception_becomes_error(self):
|
||||
from rag.qa_v1_1_73.tool_runner import make_llm_complete
|
||||
|
||||
class Client:
|
||||
def complete_raw_messages(self, *a, **kw):
|
||||
raise RuntimeError("llm down")
|
||||
|
||||
complete = make_llm_complete(Client())
|
||||
content, tool_calls, error = complete("sys", [], [])
|
||||
assert content == ""
|
||||
assert tool_calls == []
|
||||
assert "RuntimeError" in error
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 真实 configs/capabilities.yaml 两条目可解析
|
||||
# ===========================================================================
|
||||
|
||||
class TestRealCapabilitiesYaml:
|
||||
def _real_yaml(self) -> Path:
|
||||
repo_root = Path(__file__).resolve().parents[5]
|
||||
return repo_root / "agent" / "qa" / "configs" / "capabilities.yaml"
|
||||
|
||||
def test_both_bioflow_entries_parse(self):
|
||||
manifests, warnings = parse_capabilities_file(self._real_yaml())
|
||||
by_name = {m.name: m for m in manifests}
|
||||
assert "bioflow-gene-family" in by_name
|
||||
assert "bioflow-rnaseq" in by_name
|
||||
|
||||
gf = by_name["bioflow-gene-family"]
|
||||
assert gf.provider == "mcp"
|
||||
assert gf.permission == "read_only"
|
||||
assert gf.nl_triggers
|
||||
assert gf.config["mcp"]["url"].endswith("/mcp/gene-family/")
|
||||
assert gf.config["prompt_pack"]["file"] == "bioflow_gene_family_agent_rules.md"
|
||||
|
||||
rn = by_name["bioflow-rnaseq"]
|
||||
assert rn.permission == "read_only"
|
||||
assert rn.nl_triggers
|
||||
assert rn.config["mcp"]["url"].endswith("/mcp/rnaseq/")
|
||||
assert rn.config["prompt_pack"]["file"] == "bioflow_rnaseq_agent_rules.md"
|
||||
|
||||
def test_nl_triggers_cover_expected_words(self):
|
||||
manifests, _ = parse_capabilities_file(self._real_yaml())
|
||||
gf = next(m for m in manifests if m.name == "bioflow-gene-family")
|
||||
rn = next(m for m in manifests if m.name == "bioflow-rnaseq")
|
||||
# 关键触发词在位(意图预筛依赖)
|
||||
assert "基因家族" in gf.nl_triggers
|
||||
assert "转录组" in rn.nl_triggers
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 红线静态守卫:bioflow_agent 不依赖 pipeline / answer_composer
|
||||
# ===========================================================================
|
||||
|
||||
class TestRedLineNoPipelineDep:
|
||||
def test_no_pipeline_or_answer_composer_import(self):
|
||||
import ast
|
||||
|
||||
src = Path(ba.__file__).read_text(encoding="utf-8")
|
||||
tree = ast.parse(src)
|
||||
imported = []
|
||||
for node in tree.body:
|
||||
if isinstance(node, ast.Import):
|
||||
imported += [a.name for a in node.names]
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
imported.append(node.module or "")
|
||||
assert not any("pipeline" in m or "answer_composer" in m for m in imported)
|
||||
|
||||
def test_never_raises(self, tmp_path):
|
||||
# 任何异常都返回 None/正常 payload,绝不外抛
|
||||
with mock.patch.object(ba, "get_registry", side_effect=RuntimeError("boom")):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent("做基因家族", llm_client=FakeLLMClient())
|
||||
assert result is None
|
||||
@@ -51,9 +51,13 @@ class ToolRunner:
|
||||
user: str,
|
||||
tools: List[Dict[str, Any]],
|
||||
manifests: Dict[str, CapabilityManifest],
|
||||
history: Optional[List[Dict[str, str]]] = None,
|
||||
) -> Tuple[str, Dict[str, Any]]:
|
||||
"""执行工具调用循环。返回 (content, audit)。
|
||||
|
||||
history:多轮记忆(S2),List[{role: user|assistant, content}],旧→新,
|
||||
不含当前这轮提问;注入为 messages 前置轮次(作为上下文,不参与工具结果回填)。
|
||||
|
||||
audit: {"rounds": n, "tool_calls": k, "used_json_fallback": bool,
|
||||
"tool_names": [...], "errors": [...]}
|
||||
"""
|
||||
@@ -62,9 +66,19 @@ class ToolRunner:
|
||||
"tool_calls": 0,
|
||||
"used_json_fallback": False,
|
||||
"tool_names": [],
|
||||
"tool_results": {},
|
||||
"errors": [],
|
||||
}
|
||||
messages: List[Dict[str, Any]] = [{"role": "user", "content": user}]
|
||||
messages: List[Dict[str, Any]] = []
|
||||
for h in history or []:
|
||||
if not isinstance(h, dict):
|
||||
continue
|
||||
role = str(h.get("role") or "").strip()
|
||||
content = str(h.get("content") or "").strip()
|
||||
if role not in {"user", "assistant"} or not content:
|
||||
continue
|
||||
messages.append({"role": role, "content": content})
|
||||
messages.append({"role": "user", "content": user})
|
||||
system_text = system
|
||||
for round_index in range(1, self.max_rounds + 1):
|
||||
audit["rounds"] = round_index
|
||||
@@ -82,6 +96,7 @@ class ToolRunner:
|
||||
break
|
||||
assistant_message: Dict[str, Any] = {"role": "assistant", "content": content or ""}
|
||||
tool_call_blocks: List[Dict[str, Any]] = []
|
||||
messages.append(assistant_message)
|
||||
for call in tool_calls:
|
||||
name = str(call.get("name") or call.get("function", {}).get("name") or "")
|
||||
raw_args = call.get("arguments") or call.get("function", {}).get("arguments") or {}
|
||||
@@ -112,9 +127,9 @@ class ToolRunner:
|
||||
"function": {"name": name or "unknown", "arguments": json.dumps(arguments, ensure_ascii=False)},
|
||||
}
|
||||
)
|
||||
audit["tool_results"][name] = result_text
|
||||
messages.append({"role": "tool", "tool_call_id": call_id, "content": result_text})
|
||||
assistant_message["tool_calls"] = tool_call_blocks
|
||||
messages.append(assistant_message)
|
||||
continue
|
||||
# 最后一轮:拿最终文本(无工具或轮次用尽后由调用方用 messages 再 complete 一次)
|
||||
return self._final_text(system_text, messages, tools), audit
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
name: bioflow-analysis
|
||||
description: >-
|
||||
PlantAI 生信分析编排 Skill。当用户提供原始组学数据(蛋白 FASTA/基因组/FASTQ/表达矩阵等)
|
||||
并提出基因家族、转录组、变异或自定义分析需求时使用。负责从原始数据到交付报告的
|
||||
全流程编排:选链路、登记资产、就绪检查、参数确认、执行、诊断、版本化修改、交付清单。
|
||||
触发词见 nl_triggers;仅在明确涉及生信分析时触发。
|
||||
nl_triggers:
|
||||
- 基因家族
|
||||
- 家族成员
|
||||
- 亚家族
|
||||
- 系统发育
|
||||
- 建树
|
||||
- 进化树
|
||||
- motif
|
||||
- 结构域
|
||||
- 共线性
|
||||
- Ka/Ks
|
||||
- 转录组
|
||||
- RNA-seq
|
||||
- RNAseq
|
||||
- rnaseq
|
||||
- 差异表达
|
||||
- 表达定量
|
||||
- 富集
|
||||
- WGCNA
|
||||
- GSEA
|
||||
- fastq
|
||||
- 实验设计
|
||||
- 生信分析
|
||||
- 变异检测
|
||||
- GWAS
|
||||
---
|
||||
|
||||
# bioflow-analysis 能力声明
|
||||
|
||||
PlantAI 生信分析 Skill,面向「原始数据 → 交付报告」的分析编排。所有分析经
|
||||
`bioflowctl`(唯一事实源)与 11 个分析控制 MCP 工具完成,服务端状态经 SQLite 持久化。
|
||||
|
||||
## 11 工具白名单
|
||||
|
||||
Skill 只允许调用以下 11 个分析控制工具(其余为内部实现,不外泄):
|
||||
|
||||
| 工具 | 职责 | 对应 CLI |
|
||||
|------|------|----------|
|
||||
| get_task_context | 任务绑定(session/job/plan_revision/run_id/当前阶段) | — |
|
||||
| inspect_asset | 资产/文件探测(格式/预览/列结构) | file inspect |
|
||||
| validate_plan | 读单一参数 schema 校验参数 | (补齐) |
|
||||
| check_readiness | 四象限就绪检查(环境/参考/磁盘/预算) | check readiness |
|
||||
| get_run_evidence | 运行证据(日志切片/失败原因/manifest 校验值) | — |
|
||||
| propose_change | 参数变更影响范围 | change propose |
|
||||
| apply_change | 幂等应用变更 | change apply |
|
||||
| execute_nodes | 只重跑受影响节点 | run node-execute |
|
||||
| control_execution | 执行所有权(stop/resume) | run stop/resume |
|
||||
| validate_delivery | 交付清单完整性校验 | delivery validate |
|
||||
| publish_delivery | 发布交付清单 | delivery publish |
|
||||
|
||||
## 对话剧本
|
||||
|
||||
1. **原始数据**:识别用户提供的原始数据(蛋白 FASTA / 基因组 / FASTQ / 表达矩阵 / 设计表)。
|
||||
2. **选链路**:`inspect_asset` 探测数据格式;据此选择分析链路(基因家族 / 转录组 / …)。
|
||||
3. **登记**:登记数据资产(角色化 `asset register`),保证「下游直入」可用。
|
||||
4. **就绪**:`check_readiness` 四象限检查;缺项返回结构化错误阻断,不启动。
|
||||
5. **参数确认**:`validate_plan` 校验参数;未知参数/越界拒绝。
|
||||
6. **执行**:`execute_nodes` 只重跑受影响节点;`control_execution` 管理执行所有权。
|
||||
7. **诊断**:失败时 `get_run_evidence` 取日志切片/manifest 校验值,按 8 类错误分类
|
||||
(input/resource/environment/transient/adapter/artifact/reference/scientific_precondition)。
|
||||
8. **版本化**:`propose_change` + `apply_change` 版本化修改,历史不覆盖、幂等。
|
||||
9. **交付**:`validate_delivery` + `publish_delivery` 校验并发布交付清单。
|
||||
|
||||
## 人工边界(requires_human)
|
||||
|
||||
- `subfamily_confirm`:亚家族分类确认,须人工确认,不可代用户。
|
||||
- `rnaseq_de`:差异表达实验设计确认,须人工确认,不可代用户。
|
||||
上述阶段在 pending_actions 中 `requires_human=true`,Skill 只呈现表单默认值,不代确认。
|
||||
|
||||
## 验收清单
|
||||
|
||||
- [ ] 原始数据格式已探测(inspect_asset)
|
||||
- [ ] 就绪检查通过(check_readiness ready=true)
|
||||
- [ ] 参数经单一 schema 校验(validate_plan)
|
||||
- [ ] 执行所有权唯一(control_execution 不重复接管/不重叠)
|
||||
- [ ] 失败已诊断(get_run_evidence + 8 类错误分类)
|
||||
- [ ] 变更版本化(propose_change/apply_change 幂等、历史不覆盖)
|
||||
- [ ] 交付清单完整并已发布(validate_delivery ok → publish_delivery approved)
|
||||
@@ -2,13 +2,21 @@
|
||||
|
||||
- 独立部署树 bioflow_service/(FR-BIO-028:独立进程 8098 + 独立 SQLite)。
|
||||
- lifespan:init_db(幂等迁移)→ init_context(kernel/catalog/planner/runner)
|
||||
→ recover(服务重启恢复 in-flight 续跑,FR-BIO-031)→ shutdown 清理。
|
||||
→ recover(服务重启恢复 in-flight 续跑,FR-BIO-031)→ MCP session manager
|
||||
(anyio task group,streamable-http 处理请求必需)→ shutdown 清理。
|
||||
- 端点:health(/api/bioflow/health,前端探活 FR-BIO-025)。
|
||||
- 不依赖 BFF/生产服务(8090/8091 等绝不探测/连接)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# 把 bioflow_service/.env 注入 os.environ(llm.py 等模块直读 DOCGEN_LLM_*,
|
||||
# 不走 pydantic-settings 的 env_file,避免「.env 配了但 os.environ 读不到」的配置断层)。
|
||||
load_dotenv(Path(__file__).resolve().parent.parent / ".env")
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
@@ -17,11 +25,18 @@ from fastapi.middleware.cors import CORSMiddleware
|
||||
from .config import get_settings
|
||||
from .context import context, init_context, shutdown_context
|
||||
from .db import close_db, init_db
|
||||
from .mcp_server import extension_mcp, gene_family_mcp, rnaseq_mcp
|
||||
from .routers import bioarchives, bioflow, biojobs, extension
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 三个 MCP streamable-http app(调用即创建 session_manager,供 lifespan 进入 run())。
|
||||
# 端点地址:/mcp/gene-family、/mcp/rnaseq、/mcp/extension(streamable_http_path="/" + 挂载前缀)。
|
||||
_gene_family_mcp_app = gene_family_mcp.streamable_http_app(streamable_http_path="/")
|
||||
_rnaseq_mcp_app = rnaseq_mcp.streamable_http_app(streamable_http_path="/")
|
||||
_extension_mcp_app = extension_mcp.streamable_http_app(streamable_http_path="/")
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
settings = get_settings()
|
||||
@@ -60,14 +75,21 @@ def create_app() -> FastAPI:
|
||||
logger.info("main: 重启恢复 %d 个 in-flight 任务续跑", len(recovered))
|
||||
except Exception:
|
||||
logger.exception("main: 重启恢复扫描失败(任务不重跑,等待后续操作)")
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await shutdown_context()
|
||||
try:
|
||||
await close_db()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# MCP session manager 的 task group 必须在服务运行期内保持活跃
|
||||
# (streamable-http 处理请求时要求已初始化)。Starlette 不会自动运行
|
||||
# 子 Mount 的 lifespan,故在此显式进入三个 MCP 的 run()。
|
||||
async with gene_family_mcp.session_manager.run():
|
||||
async with rnaseq_mcp.session_manager.run():
|
||||
async with extension_mcp.session_manager.run():
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await shutdown_context()
|
||||
try:
|
||||
await close_db()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
app.router.lifespan_context = _lifespan
|
||||
|
||||
@@ -75,6 +97,12 @@ def create_app() -> FastAPI:
|
||||
app.include_router(biojobs.router)
|
||||
app.include_router(bioarchives.router)
|
||||
app.include_router(extension.router)
|
||||
|
||||
# 生信 MCP 能力外挂(mcp 2.x:MCPServer.streamable_http_app)。
|
||||
# 三个独立 streamable-http 端点,各自装卸(REST 面不受影响)。
|
||||
app.mount("/mcp/gene-family", _gene_family_mcp_app)
|
||||
app.mount("/mcp/rnaseq", _rnaseq_mcp_app)
|
||||
app.mount("/mcp/extension", _extension_mcp_app)
|
||||
return app
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,900 @@
|
||||
"""app/mcp_server.py —— 生信 MCP server(两个 streamable-http 端点)。
|
||||
|
||||
- `/mcp/gene-family`:基因家族能力单元(plan/create/ask + 共享控制组 + 文件组)
|
||||
- `/mcp/rnaseq`:转录组能力单元(plan/create/generate_samples_yaml/ask + 共享控制组 + 文件组)
|
||||
|
||||
设计约束(实施方案 S0):
|
||||
- 工具只承诺「已受理」,执行进度/日志由 SSE 事件总线直推,不走 MCP。
|
||||
- 每个能力单元只接受本链阶段(create 跨链拒绝 CROSS_CHAIN)。
|
||||
- 人工确认边界(subfamily_confirm / rnaseq_de 实验设计确认)在 job_status 的
|
||||
pending_actions 里标 requires_human=true,工具不提供绕过。
|
||||
- 白名单沿用 runner._allowed_param_names(单一事实源)。
|
||||
- 工具名不外泄由 QA agent(S1)负责,本模块只负责工具语义。
|
||||
|
||||
mcp 2.x 注意:MCPServer(非 FastMCP,来自 mcp.server.mcpserver);挂载用
|
||||
`mcp.streamable_http_app()`。本文件位于 app/ 层(非 services/),可用 pydantic(mcp
|
||||
自身也依赖 pydantic 构造 schema)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .config import get_settings
|
||||
from .context import load_record
|
||||
from .services.bioflow import extensions as ext_mod
|
||||
from .services.bioflow import store
|
||||
from .services.bioflow.archive import archive_from_job
|
||||
from .services.bioflow.catalog import GENE_FAMILY_CHAIN
|
||||
from .services.bioflow.errors import error_result
|
||||
from .services.bioflow.file_transform import validate_format
|
||||
from .services.bioflow.runner import RunnerError, ensure_samples_yaml, llm_sample_fallback
|
||||
from .services.bioflow.source_fetcher import fetch_source
|
||||
|
||||
# 薄适配:委托 bioflowctl CLI(唯一事实源),不重复业务逻辑
|
||||
from bioflowctl.commands import (
|
||||
cmd_archive,
|
||||
cmd_archive_reuse,
|
||||
cmd_file_inspect,
|
||||
cmd_file_transform,
|
||||
cmd_source_fetch,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _mcp_err(code: str, message: str, *, evidence_refs: list[str] | None = None) -> dict[str, Any]:
|
||||
"""G13:MCP 工具失败返回统一结构化错误对象(code 顶层 + error 对象)。"""
|
||||
return error_result(code, message, evidence_refs=evidence_refs)
|
||||
|
||||
# ---- 能力单元阶段白名单 ----
|
||||
|
||||
GENE_FAMILY_STAGES = set(GENE_FAMILY_CHAIN)
|
||||
|
||||
RNASEQ_STAGES = {
|
||||
"rnaseq_qc", "rnaseq_align", "rnaseq_quant", "rnaseq_sample_qc", "rnaseq_de",
|
||||
"rnaseq_wgcna", "rnaseq_go_enrich", "rnaseq_kegg_enrich", "rnaseq_enrich",
|
||||
"rnaseq_fgsea", "rnaseq_candidate_family", "rnaseq_report",
|
||||
"annotation", "expression",
|
||||
}
|
||||
|
||||
# 人工确认边界:这些阶段 waiting_confirm 时 requires_human=true,工具/QA 不可代确认
|
||||
HUMAN_CONFIRM_STAGES = {"subfamily_confirm", "rnaseq_de"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# context 访问(动态读模块全局,测试 monkeypatch 兼容)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _ctx():
|
||||
from . import context as context_mod
|
||||
|
||||
return context_mod.context
|
||||
|
||||
|
||||
def _job_detail(record: Any) -> dict[str, Any]:
|
||||
stages = []
|
||||
for stage in record.stage_chain:
|
||||
st = record._stage_states.get(stage, {})
|
||||
stages.append({
|
||||
"stage": stage,
|
||||
"status": st.get("status", "waiting"),
|
||||
"params": st.get("params") or {},
|
||||
"attempt": int(st.get("attempt") or 0),
|
||||
"manifest_path": st.get("manifest_path"),
|
||||
"log_tail": st.get("log_tail") or "",
|
||||
"error": st.get("error"),
|
||||
})
|
||||
return {
|
||||
"id": record.id,
|
||||
"session_id": record.session_id,
|
||||
"title": record.title,
|
||||
"status": record.status,
|
||||
"stages": stages,
|
||||
"params": record.params,
|
||||
"auto_continue": record.auto_continue,
|
||||
"archive_id": record.archive_id,
|
||||
"progress": record.progress(),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# pending_actions 派生(job_status 工具返回给 QA agent 的下一步动作)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def derive_pending_actions(record: Any, runner: Any) -> list[dict[str, Any]]:
|
||||
"""由任务/阶段状态派生待办动作(QA agent 据此引导用户,人工边界不可代确认)。
|
||||
|
||||
按阶段链顺序找第一个「卡住」的环节,返回动作清单(通常 0-1 条)。
|
||||
"""
|
||||
if record.status == "paused":
|
||||
return [{"action": "resume", "stage": None, "requires_human": False,
|
||||
"message": "任务已暂停,可恢复"}]
|
||||
if record.status == "done":
|
||||
return [{"action": "archive", "stage": None, "requires_human": False,
|
||||
"message": "任务已完成,可存档复用"}]
|
||||
if record.status == "archived":
|
||||
return [{"action": "none", "stage": None, "requires_human": False,
|
||||
"message": "任务已存档"}]
|
||||
|
||||
for stage in record.stage_chain:
|
||||
status = record._stage_states.get(stage, {}).get("status", "waiting")
|
||||
if status == "await_files":
|
||||
return [{"action": "provide_files", "stage": stage, "requires_human": False,
|
||||
"message": f"阶段 {stage} 等待输入文件"}]
|
||||
if status == "waiting_confirm":
|
||||
human = stage in HUMAN_CONFIRM_STAGES
|
||||
return [{"action": "confirm", "stage": stage, "requires_human": human,
|
||||
"form_defaults": runner._form_defaults(stage, record),
|
||||
"message": (f"阶段 {stage} 需人工确认(不可代确认)" if human
|
||||
else f"阶段 {stage} 等待参数确认")}]
|
||||
if status == "failed":
|
||||
return [{"action": "rerun_or_skip", "stage": stage, "requires_human": False,
|
||||
"message": f"阶段 {stage} 失败,可重跑或跳过"}]
|
||||
if status == "partial_success":
|
||||
# G4:partial_success 独立终态,不被当作 success/done
|
||||
return [{"action": "rerun_or_skip", "stage": stage, "requires_human": False,
|
||||
"message": f"阶段 {stage} 部分成功(产物不全),可重跑或跳过"}]
|
||||
# waiting/running/queued/done/skipped → 继续
|
||||
return []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 工具实现(命名 async 函数,带类型提示,供两个 MCPServer 注册)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _plan(message: str, allowed_stages: set[str]) -> dict[str, Any]:
|
||||
from .services.bioflow.planner import PlanCandidate, PlannerError, PlanResult
|
||||
|
||||
ctx = _ctx()
|
||||
if ctx.planner is None:
|
||||
return {"ok": False, "code": "NOT_READY", "message": "规划器未就绪"}
|
||||
try:
|
||||
plan = await ctx.planner.plan(message)
|
||||
except PlannerError as exc:
|
||||
fallback = [s for s in ("identify", "rnaseq_qc") if s in allowed_stages and ctx.catalog.has(s)]
|
||||
plan = PlanResult(candidates=[PlanCandidate(purpose="内置默认流程", stages=fallback)],
|
||||
source="llm_only", message=exc.message)
|
||||
candidates = []
|
||||
for c in plan.candidates:
|
||||
stages = [s for s in c.stages if s in allowed_stages]
|
||||
if stages:
|
||||
candidates.append({"purpose": c.purpose, "stages": stages})
|
||||
return {
|
||||
"ok": True,
|
||||
"candidates": candidates,
|
||||
"source": plan.source,
|
||||
"reuse_archive": plan.reuse_archive_id,
|
||||
"reuse_name": plan.reuse_name,
|
||||
"fallback": plan.message or None,
|
||||
}
|
||||
|
||||
|
||||
async def _create_job(session_id: str, title: str, stages: list[str],
|
||||
params: dict[str, Any] | None, auto_continue: bool,
|
||||
archive_id: str | None, allowed_stages: set[str]) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
unknown = [s for s in stages if s not in allowed_stages]
|
||||
if unknown:
|
||||
return {"ok": False, "code": "CROSS_CHAIN",
|
||||
"message": f"阶段不属于该能力单元: {', '.join(unknown)}"}
|
||||
try:
|
||||
record = await ctx.runner.create_job(
|
||||
session_id, title or "生信任务", stages, params or {},
|
||||
auto_continue=auto_continue, archive_id=archive_id,
|
||||
)
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
ctx.records[record.id] = record
|
||||
return {"ok": True, "data": _job_detail(record)}
|
||||
|
||||
|
||||
async def _job_status(job_id: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
detail = _job_detail(record)
|
||||
detail["pending_actions"] = derive_pending_actions(record, ctx.runner)
|
||||
return {"ok": True, "data": detail}
|
||||
|
||||
|
||||
async def _confirm_stage(job_id: str, stage: str, params: dict[str, Any] | None) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
if stage not in record.stage_chain:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
|
||||
try:
|
||||
await ctx.runner.confirm_stage(record, stage, params or {})
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
|
||||
"status": ctx.runner.stage_status(record, stage)}}
|
||||
|
||||
|
||||
async def _rerun_stage(job_id: str, stage: str, params: dict[str, Any] | None) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
if stage not in record.stage_chain:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
|
||||
try:
|
||||
await ctx.runner.rerun_stage(record, stage, params)
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
|
||||
"status": ctx.runner.stage_status(record, stage)}}
|
||||
|
||||
|
||||
async def _skip_stage(job_id: str, stage: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
if stage not in record.stage_chain:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
|
||||
try:
|
||||
await ctx.runner.skip_stage(record, stage)
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
|
||||
"status": ctx.runner.stage_status(record, stage)}}
|
||||
|
||||
|
||||
async def _pause_job(job_id: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
try:
|
||||
await ctx.runner.pause_job(record)
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
return {"ok": True, "data": {"job_id": job_id, "status": record.status}}
|
||||
|
||||
|
||||
async def _resume_job(job_id: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
try:
|
||||
await ctx.runner.resume_job(record)
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
return {"ok": True, "data": {"job_id": job_id, "status": record.status}}
|
||||
|
||||
|
||||
async def _archive_job(job_id: str, name: str, description: str,
|
||||
include_intermediate: bool) -> dict[str, Any]:
|
||||
"""任务存档(薄适配):委托 cmd_archive(唯一事实源),仅保留内存 record 校验 + 事件发布。"""
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return _mcp_err("NOT_FOUND", "任务不存在")
|
||||
result = await cmd_archive(
|
||||
job_id, name, description,
|
||||
db=ctx.db,
|
||||
record=record,
|
||||
include_intermediate=bool(include_intermediate),
|
||||
archive_fn=archive_from_job,
|
||||
archives_dir=get_settings().archives_dir_abs,
|
||||
)
|
||||
if result.get("ok"):
|
||||
ctx.events.publish("job_archived", {"job_id": job_id, "archive_id": result["data"]["id"]})
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 扩展能力单元(第三个能力单元 bioflow-extension):复用 extensions.py 状态机
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_EXT_VIEW_KEYS = (
|
||||
"ext_id", "display_name", "status", "command_template", "original_command",
|
||||
"runtime_image", "requirements", "parameters", "input_roles", "nl_triggers",
|
||||
"analysis_type", "golden", "repair_count", "trial", "validation", "last_build",
|
||||
"created_at", "updated_at",
|
||||
)
|
||||
|
||||
|
||||
def _ext_view(ext: dict[str, Any]) -> dict[str, Any]:
|
||||
"""扩展精简视图(剔除 source_files 等超长字段,供 MCP 工具结果返回)。"""
|
||||
return {k: ext.get(k) for k in _EXT_VIEW_KEYS if k in ext}
|
||||
|
||||
|
||||
def _file_items(files: Any) -> list[tuple[str, bytes]]:
|
||||
"""把 MCP 入参 [{path, content|data}] 转为 [(rel_path, bytes)]。"""
|
||||
out: list[tuple[str, bytes]] = []
|
||||
if not isinstance(files, list):
|
||||
return out
|
||||
for f in files:
|
||||
if not isinstance(f, dict):
|
||||
continue
|
||||
path = str(f.get("path") or f.get("name") or "").strip()
|
||||
if not path:
|
||||
continue
|
||||
content = f.get("content")
|
||||
if content is None:
|
||||
content = f.get("data") or ""
|
||||
if isinstance(content, (bytes, bytearray)):
|
||||
blob = bytes(content)
|
||||
else:
|
||||
blob = str(content).encode("utf-8")
|
||||
out.append((path, blob))
|
||||
return out
|
||||
|
||||
|
||||
def _ext_err(exc: Exception) -> str:
|
||||
if isinstance(exc, ext_mod.ExtensionNotFoundError):
|
||||
return "NOT_FOUND"
|
||||
if isinstance(exc, ext_mod.ExtensionStatusError):
|
||||
return "EXT_STATUS"
|
||||
if isinstance(exc, ext_mod.ExtensionBuildError):
|
||||
return "EXT_BUILD"
|
||||
if isinstance(exc, ext_mod.ExtensionRunError):
|
||||
return "EXT_RUN"
|
||||
return "EXT_ERROR"
|
||||
|
||||
|
||||
async def _create_extension(display_name: str, command: str, runtime_image: str,
|
||||
code_files: Any, data_files: Any) -> dict[str, Any]:
|
||||
code = _file_items(code_files)
|
||||
if not code:
|
||||
return {"ok": False, "code": "VALIDATION", "message": "扩展代码包不能为空"}
|
||||
try:
|
||||
ext = ext_mod.get_store().create_extension(
|
||||
display_name=display_name, command=command, runtime_image=runtime_image or "",
|
||||
code_files=code, data_files=_file_items(data_files),
|
||||
)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _extension_status(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = ext_mod.get_store().load_extension(ext_id)
|
||||
except ext_mod.ExtensionNotFoundError:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "扩展不存在"}
|
||||
return {"ok": True, "data": _ext_view(ext) if ext else None}
|
||||
|
||||
|
||||
async def _build_extension(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = await ext_mod.get_store().build_extension(ext_id)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _trial_run_extension(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = await ext_mod.get_store().trial_run(ext_id)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _validate_extension(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = ext_mod.get_store().validate(ext_id)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _confirm_extension(ext_id: str, nl_triggers: Any, parameters: Any,
|
||||
runtime_image: str) -> dict[str, Any]:
|
||||
triggers = [str(x).strip() for x in nl_triggers if str(x).strip()] if isinstance(nl_triggers, list) else None
|
||||
params = parameters if isinstance(parameters, list) else None
|
||||
try:
|
||||
ext = ext_mod.get_store().confirm(
|
||||
ext_id, nl_triggers=triggers, parameters=params,
|
||||
runtime_image=runtime_image or None,
|
||||
)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _publish_extension(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = ext_mod.get_store().publish(ext_id)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _run_extension(ext_id: str, params: Any, files: Any) -> dict[str, Any]:
|
||||
try:
|
||||
result = await ext_mod.get_store().run(
|
||||
ext_id,
|
||||
params=params if isinstance(params, dict) else None,
|
||||
files=[str(f) for f in files] if isinstance(files, list) else None,
|
||||
)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": result}
|
||||
|
||||
|
||||
async def _repair_extension(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = await ext_mod.get_store().repair(ext_id)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
# ---- 文件登记(复刻 biojobs._register_path 核心,无 FastAPI 依赖)----
|
||||
|
||||
def _blocked_reason(path: Path, settings: Any) -> str | None:
|
||||
s = str(path)
|
||||
for prefix in settings.sensitive_path_prefixes:
|
||||
if s == prefix or s.startswith(prefix + "/"):
|
||||
return f"敏感路径前缀 {prefix}"
|
||||
for pat in settings.sensitive_filename_patterns:
|
||||
if re.match(pat, path.name):
|
||||
return f"敏感文件名 {path.name}"
|
||||
return None
|
||||
|
||||
|
||||
async def _register_job_files(job_id: str, paths: list[str]) -> dict[str, Any]:
|
||||
"""登记服务器/局域网路径(单文件或目录展开)到任务 input。"""
|
||||
ctx = _ctx()
|
||||
settings = get_settings()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return _mcp_err("NOT_FOUND", "任务不存在")
|
||||
|
||||
input_dir = record.input_dir
|
||||
input_dir.mkdir(parents=True, exist_ok=True)
|
||||
items: list[dict[str, Any]] = []
|
||||
registered: list[str] = []
|
||||
|
||||
for raw in paths:
|
||||
p = Path(raw).expanduser().resolve()
|
||||
if not p.exists():
|
||||
return _mcp_err("PATH_NOT_EXIST", f"路径不存在: {raw}")
|
||||
blocked = _blocked_reason(p, settings)
|
||||
if blocked:
|
||||
return _mcp_err("PATH_NOT_ALLOWED", f"路径不允许: {raw}({blocked})")
|
||||
files: list[Path] = []
|
||||
if p.is_file():
|
||||
files = [p]
|
||||
else:
|
||||
ext_set = settings.upload_ext_set
|
||||
for f in sorted(p.rglob("*")):
|
||||
if f.is_file() and (not ext_set or f.suffix.lower() in ext_set) and not _blocked_reason(f, settings):
|
||||
files.append(f)
|
||||
if not files:
|
||||
return _mcp_err("VALIDATION", f"目录内无可登记文件: {raw}")
|
||||
for f in files:
|
||||
link = input_dir / f.name
|
||||
try:
|
||||
if not link.exists():
|
||||
link.symlink_to(f)
|
||||
except OSError as exc: # noqa: BLE001
|
||||
logger.warning("mcp register symlink 失败: %s -> %s (%s)", f, link, exc)
|
||||
items.append({"file_id": f"{record.id}:{f.name}", "name": f.name,
|
||||
"size": f.stat().st_size, "path": str(f), "source": "server_path"})
|
||||
registered.append(str(f))
|
||||
|
||||
# rnaseq 系:samples.yaml 生成(描述文件优先 + 实况分析;规则不可归纳走 LLM 兜底)
|
||||
try:
|
||||
source_roots = [str(Path(x).parent) for x in paths if Path(x).exists()]
|
||||
extra_mounts = ensure_samples_yaml(input_dir, source_roots=[Path(r) for r in source_roots])
|
||||
if not extra_mounts:
|
||||
try:
|
||||
await llm_sample_fallback(input_dir, [Path(r) for r in source_roots])
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("samples.yaml LLM 兜底失败: %s", exc)
|
||||
if extra_mounts:
|
||||
record.registered_server_paths = list(
|
||||
dict.fromkeys((getattr(record, "registered_server_paths", None) or [])
|
||||
+ [str(Path(m)) for m in extra_mounts]))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("samples.yaml 分析失败: %s", exc)
|
||||
|
||||
if registered:
|
||||
import json
|
||||
record.registered_server_paths = list(
|
||||
dict.fromkeys((getattr(record, "registered_server_paths", None) or []) + registered)
|
||||
)
|
||||
(record.job_dir / "server_registered.json").write_text(
|
||||
json.dumps({"registered_server_paths": record.registered_server_paths},
|
||||
ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
# 自动续跑(await_inputs 任务重推;非等待态 no-op)
|
||||
await ctx.runner.retry_after_files(record)
|
||||
return {"ok": True, "data": {"items": items, "total": len(items)}}
|
||||
|
||||
|
||||
async def _ask_question(question: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
hits: list[dict[str, Any]] = []
|
||||
if ctx.flow_store is not None:
|
||||
try:
|
||||
hits = ctx.flow_store.search(question, top_k=8).get("items") or []
|
||||
except Exception: # noqa: BLE001
|
||||
hits = []
|
||||
citations = []
|
||||
for h in hits:
|
||||
citations.append({
|
||||
"source": h.get("source") or h.get("doc_kind") or "",
|
||||
"title_path": h.get("title_path") or "",
|
||||
"path": h.get("path") or "",
|
||||
"text": str(h.get("text") or "")[:1200],
|
||||
})
|
||||
return {"ok": True, "citations": citations, "degraded": not citations}
|
||||
|
||||
|
||||
async def _list_archives(page: int, page_size: int) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
items, total = await store.list_archives(ctx.db, page=page, page_size=page_size)
|
||||
return {"ok": True, "items": items, "total": total, "page": page, "page_size": page_size}
|
||||
|
||||
|
||||
async def _reuse_archive(name: str) -> dict[str, Any]:
|
||||
"""复用存档(薄适配):委托 cmd_archive_reuse(读各阶段实际生效参数)。"""
|
||||
return await cmd_archive_reuse(name, db=_ctx().db)
|
||||
|
||||
|
||||
async def _fetch_source_file(job_id: str | None, source: str,
|
||||
credentials: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""拉取源文件(薄适配):委托 cmd_source_fetch,仅保留内存 record 校验。"""
|
||||
if job_id:
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return _mcp_err("NOT_FOUND", "任务不存在")
|
||||
return await cmd_source_fetch(source, job_id=job_id, credentials=credentials,
|
||||
fetch_fn=fetch_source)
|
||||
|
||||
|
||||
async def _inspect_file(path: str) -> dict[str, Any]:
|
||||
"""探测文件(薄适配):委托 cmd_file_inspect。"""
|
||||
return cmd_file_inspect(path)
|
||||
|
||||
|
||||
async def _transform_file(src_path: str, target_format: str,
|
||||
column_map: dict[str, str] | None,
|
||||
dest_path: str | None) -> dict[str, Any]:
|
||||
"""转换文件(薄适配):委托 cmd_file_transform(注入 LLM 适配器)。"""
|
||||
from .services.bioflow.llm import complete_json
|
||||
|
||||
async def llm_adapter(user: str, system: str) -> Any:
|
||||
return await complete_json(system, user, temperature=0.2, max_tokens=2000)
|
||||
|
||||
return await cmd_file_transform(
|
||||
src_path, target_format, column_map=column_map, dest=dest_path, llm=llm_adapter,
|
||||
)
|
||||
|
||||
|
||||
async def _generate_samples_yaml(job_id: str, sample_description: str) -> dict[str, Any]:
|
||||
import yaml
|
||||
|
||||
from .services.bioflow.llm import complete_json
|
||||
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
system = (
|
||||
"你是 RNA-seq 样本配置助手。根据用户的自然语言描述,输出 samples.yaml 结构 JSON:"
|
||||
'{"samples": {"样本名": {"r1": "fastq路径", "r2": "fastq路径", "condition": "...", ...}}}。'
|
||||
"只输出 JSON。"
|
||||
)
|
||||
try:
|
||||
raw = await complete_json(system, sample_description, temperature=0.2, max_tokens=2000)
|
||||
samples = raw.get("samples") if isinstance(raw, dict) else None
|
||||
if not isinstance(samples, dict) or not samples:
|
||||
return {"ok": False, "code": "EMPTY_SAMPLES", "message": "未能从描述解析出样本"}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"ok": False, "code": "SAMPLES_GEN_FAILED", "message": f"样本配置生成失败: {exc}"}
|
||||
record.input_dir.mkdir(parents=True, exist_ok=True)
|
||||
out = record.input_dir / "samples.yaml"
|
||||
out.write_text(yaml.safe_dump({"samples": samples}, allow_unicode=True), encoding="utf-8")
|
||||
v = validate_format(out, "samples.yaml")
|
||||
if not v.get("ok"):
|
||||
out.unlink(missing_ok=True)
|
||||
return {"ok": False, "code": "VALIDATION_FAILED", "message": f"样本配置校验失败: {v.get('reason')}"}
|
||||
return {"ok": True, "data": {"path": str(out), "samples": samples, "validated": v}}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 能力单元工具(命名 async,带类型提示 + 中文描述)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def plan_gene_family(message: str) -> dict[str, Any]:
|
||||
"""根据用户自然语言需求规划基因家族分析候选路径(仅基因家族链阶段)。"""
|
||||
return await _plan(message, GENE_FAMILY_STAGES)
|
||||
|
||||
|
||||
async def create_gene_family_job(session_id: str, stages: list[str], title: str = "",
|
||||
params: dict[str, Any] | None = None,
|
||||
auto_continue: bool = False,
|
||||
archive_id: str | None = None) -> dict[str, Any]:
|
||||
"""按确认的阶段链创建基因家族任务(stages 只能是基因家族链阶段,跨链拒绝)。"""
|
||||
return await _create_job(session_id, title, stages, params, auto_continue, archive_id, GENE_FAMILY_STAGES)
|
||||
|
||||
|
||||
async def ask_gene_family_question(question: str) -> dict[str, Any]:
|
||||
"""回答基因家族流程/参数/生物学意义类问题(流程知识库检索,带引用)。"""
|
||||
return await _ask_question(question)
|
||||
|
||||
|
||||
async def plan_rnaseq(message: str) -> dict[str, Any]:
|
||||
"""根据用户自然语言需求规划转录组分析候选路径(仅转录组链阶段)。"""
|
||||
return await _plan(message, RNASEQ_STAGES)
|
||||
|
||||
|
||||
async def create_rnaseq_job(session_id: str, stages: list[str], title: str = "",
|
||||
params: dict[str, Any] | None = None,
|
||||
auto_continue: bool = False,
|
||||
archive_id: str | None = None) -> dict[str, Any]:
|
||||
"""按确认的阶段链创建转录组任务(stages 只能是转录组链阶段,跨链拒绝)。"""
|
||||
return await _create_job(session_id, title, stages, params, auto_continue, archive_id, RNASEQ_STAGES)
|
||||
|
||||
|
||||
async def generate_samples_yaml(job_id: str, sample_description: str) -> dict[str, Any]:
|
||||
"""把用户的自然语言样本描述(样本名/对照处理组/重复/单双端等)解析为 samples.yaml 写入任务 input。"""
|
||||
return await _generate_samples_yaml(job_id, sample_description)
|
||||
|
||||
|
||||
async def ask_rnaseq_question(question: str) -> dict[str, Any]:
|
||||
"""回答转录组流程/参数/实验设计类问题(流程知识库检索,带引用)。"""
|
||||
return await _ask_question(question)
|
||||
|
||||
|
||||
async def job_status(job_id: str) -> dict[str, Any]:
|
||||
"""查询生信任务状态:任务/各阶段状态 + 下一步待办动作(pending_actions)。卡住或需确认时用本工具查真相。"""
|
||||
return await _job_status(job_id)
|
||||
|
||||
|
||||
async def confirm_stage(job_id: str, stage: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""确认某阶段参数并执行。仅 waiting_confirm 状态可确认;参数名须在白名单内(未知参数被拒)。人工确认边界阶段(subfamily_confirm/rnaseq_de)不可代用户确认。"""
|
||||
return await _confirm_stage(job_id, stage, params)
|
||||
|
||||
|
||||
async def register_job_files(job_id: str, paths: list[str]) -> dict[str, Any]:
|
||||
"""把服务器/局域网路径(单文件或目录)登记为任务输入文件。目录按扩展名白名单展开。"""
|
||||
return await _register_job_files(job_id, paths)
|
||||
|
||||
|
||||
async def rerun_stage(job_id: str, stage: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""重跑失败阶段(带参数则先等确认)。仅 failed 状态可重跑。"""
|
||||
return await _rerun_stage(job_id, stage, params)
|
||||
|
||||
|
||||
async def skip_stage(job_id: str, stage: str) -> dict[str, Any]:
|
||||
"""跳过失败阶段并推进后续。仅 failed 状态可跳过。"""
|
||||
return await _skip_stage(job_id, stage)
|
||||
|
||||
|
||||
async def pause_job(job_id: str) -> dict[str, Any]:
|
||||
"""暂停运行中的任务(running 阶段会立即 kill 容器)。"""
|
||||
return await _pause_job(job_id)
|
||||
|
||||
|
||||
async def resume_job(job_id: str) -> dict[str, Any]:
|
||||
"""恢复已暂停的任务(从下一阶段继续,确认点不跳过)。"""
|
||||
return await _resume_job(job_id)
|
||||
|
||||
|
||||
async def list_archives(page: int = 1, page_size: int = 20) -> dict[str, Any]:
|
||||
"""列出历史存档任务(供复用参考)。"""
|
||||
return await _list_archives(page, page_size)
|
||||
|
||||
|
||||
async def reuse_archive(name: str) -> dict[str, Any]:
|
||||
"""按名称匹配存档,返回其阶段链+参数(供复用发起)。"""
|
||||
return await _reuse_archive(name)
|
||||
|
||||
|
||||
async def archive_job(job_id: str, name: str, description: str = "",
|
||||
include_intermediate: bool = False) -> dict[str, Any]:
|
||||
"""把已完成任务存档(打包结果/日志/事件 + 落档登记);重名拒绝(ARCHIVE_EXISTS)。"""
|
||||
return await _archive_job(job_id, name, description, include_intermediate)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 扩展能力单元工具(第三个能力单元 bioflow-extension)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def create_extension(display_name: str, command: str, runtime_image: str = "",
|
||||
code_files: list[dict[str, Any]] | None = None,
|
||||
data_files: list[dict[str, Any]] | None = None) -> dict[str, Any]:
|
||||
"""创建扩展草稿(draft):上传代码文件清单(每项 {path, content 文本}) + 运行命令 + 显示名。"""
|
||||
return await _create_extension(display_name, command, runtime_image, code_files, data_files)
|
||||
|
||||
|
||||
async def extension_status(ext_id: str) -> dict[str, Any]:
|
||||
"""查询扩展详情与状态(draft/build/trial/validated/confirming/published)。"""
|
||||
return await _extension_status(ext_id)
|
||||
|
||||
|
||||
async def build_extension(ext_id: str) -> dict[str, Any]:
|
||||
"""AI 转换扩展(原始命令→command_template+参数+dependencies);LLM 失败自动回退规则版。"""
|
||||
return await _build_extension(ext_id)
|
||||
|
||||
|
||||
async def trial_run_extension(ext_id: str) -> dict[str, Any]:
|
||||
"""试运行扩展(docker 执行;未接入执行器时返回 EXT_RUN 503 语义)。"""
|
||||
return await _trial_run_extension(ext_id)
|
||||
|
||||
|
||||
async def validate_extension(ext_id: str) -> dict[str, Any]:
|
||||
"""校验扩展(金标/试运行校验)。仅 built/trial_passed 可校验。"""
|
||||
return await _validate_extension(ext_id)
|
||||
|
||||
|
||||
async def confirm_extension(ext_id: str, nl_triggers: list[str] | None = None,
|
||||
parameters: list[dict[str, Any]] | None = None,
|
||||
runtime_image: str = "") -> dict[str, Any]:
|
||||
"""确认扩展参数/触发词(用户可编辑 nl_triggers);仅 validated/trial_passed 可确认。"""
|
||||
return await _confirm_extension(ext_id, nl_triggers, parameters, runtime_image)
|
||||
|
||||
|
||||
async def publish_extension(ext_id: str) -> dict[str, Any]:
|
||||
"""发布扩展(仅 published 状态可被自然语言触发运行)。"""
|
||||
return await _publish_extension(ext_id)
|
||||
|
||||
|
||||
async def run_extension(ext_id: str, params: dict[str, Any] | None = None,
|
||||
files: list[str] | None = None) -> dict[str, Any]:
|
||||
"""正式运行已发布扩展(同步返回 run_result)。"""
|
||||
return await _run_extension(ext_id, params, files)
|
||||
|
||||
|
||||
async def repair_extension(ext_id: str) -> dict[str, Any]:
|
||||
"""AI 修复扩展(试运行失败上下文→LLM 修复 command_template/依赖;LLM 失败回退规则版)。"""
|
||||
return await _repair_extension(ext_id)
|
||||
|
||||
|
||||
async def fetch_source_file(source: str, job_id: str | None = None,
|
||||
credentials: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""从 web(http/https 流式)、OSS(oss:// 或预签名 URL)或局域网路径拉取源文件到任务 input;下载前检查磁盘余量并记录 provenance。"""
|
||||
return await _fetch_source_file(job_id, source, credentials)
|
||||
|
||||
|
||||
async def inspect_file(path: str) -> dict[str, Any]:
|
||||
"""探测文件格式(扩展名仅参考):返回预览/列结构/大小/格式猜测/乱码检测,供判断是否需要预处理。"""
|
||||
return await _inspect_file(path)
|
||||
|
||||
|
||||
async def transform_file(src_path: str, target_format: str,
|
||||
column_map: dict[str, str] | None = None,
|
||||
dest_path: str | None = None) -> dict[str, Any]:
|
||||
"""把非标准文件(xlsx/csv/tsv/txt)转换为 fasta/gff3/tsv/samples.yaml;LLM 定列映射规则+脚本执行,结果过格式验证器。"""
|
||||
return await _transform_file(src_path, target_format, column_map, dest_path)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# G12:11 个分析控制 MCP 工具(异步薄封装委托 skill_tools,不写业务逻辑)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def get_task_context(job_id: str | None = None) -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.get_task_context(job_id)
|
||||
|
||||
|
||||
async def inspect_asset(role: str = "pep.fa", path: str = "") -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.inspect_asset(role, path)
|
||||
|
||||
|
||||
async def validate_plan(stages: list[str] | None = None) -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.validate_plan(stages)
|
||||
|
||||
|
||||
async def check_readiness(assets_root: str = "", stages: list[str] | None = None) -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.check_readiness(assets_root, stages)
|
||||
|
||||
|
||||
async def get_run_evidence(run_id: str = "") -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.get_run_evidence(run_id)
|
||||
|
||||
|
||||
async def propose_change(job_id: str = "", stage: str = "",
|
||||
params: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.propose_change(job_id, stage, params)
|
||||
|
||||
|
||||
async def apply_change(action_id: str = "", change: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.apply_change(action_id, change)
|
||||
|
||||
|
||||
async def execute_nodes(job_id: str = "", nodes: list[str] | None = None) -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.execute_nodes(job_id, nodes)
|
||||
|
||||
|
||||
async def control_execution(run_id: str = "", command: str = "") -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.control_execution(run_id, command)
|
||||
|
||||
|
||||
async def validate_delivery(manifest: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.validate_delivery(manifest)
|
||||
|
||||
|
||||
async def publish_delivery(manifest: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
from .services.bioflow import skill_tools
|
||||
return skill_tools.publish_delivery(manifest)
|
||||
|
||||
|
||||
SKILL_CONTROL_TOOLS = [
|
||||
get_task_context, inspect_asset, validate_plan,
|
||||
check_readiness, get_run_evidence, propose_change,
|
||||
apply_change, execute_nodes, control_execution,
|
||||
validate_delivery, publish_delivery,
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 构建三个 MCPServer(基因家族 / 转录组 / 扩展)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def build_mcp_servers() -> tuple[Any, Any, Any]:
|
||||
from mcp.server.mcpserver import MCPServer
|
||||
|
||||
gf = MCPServer(
|
||||
"bioflow-gene-family",
|
||||
title="PlantAI 基因家族分析",
|
||||
description="基因家族分析能力单元:路径规划/建任务/确认参数/查状态/文件获取与预处理。",
|
||||
)
|
||||
rn = MCPServer(
|
||||
"bioflow-rnaseq",
|
||||
title="PlantAI 转录组分析",
|
||||
description="转录组分析能力单元:路径规划/样本配置生成/建任务/确认参数/查状态/文件获取与预处理。",
|
||||
)
|
||||
ext = MCPServer(
|
||||
"bioflow-extension",
|
||||
title="PlantAI 用户自定义扩展分析",
|
||||
description="用户自定义扩展分析能力单元:创建/转换/试运行/校验/确认/发布/运行。",
|
||||
)
|
||||
|
||||
gf.tool(name="plan_gene_family", description="根据用户自然语言需求规划基因家族分析候选路径(仅基因家族链阶段)。返回 2-3 条候选路径供用户确认。")(plan_gene_family)
|
||||
gf.tool(name="create_gene_family_job", description="按确认的阶段链创建基因家族任务。stages 只能是基因家族链阶段(跨链拒绝)。")(create_gene_family_job)
|
||||
gf.tool(name="ask_gene_family_question", description="回答基因家族流程/参数/生物学意义类问题(流程知识库检索,带引用)。")(ask_gene_family_question)
|
||||
|
||||
rn.tool(name="plan_rnaseq", description="根据用户自然语言需求规划转录组分析候选路径(仅转录组链阶段)。返回候选路径供用户确认。")(plan_rnaseq)
|
||||
rn.tool(name="create_rnaseq_job", description="按确认的阶段链创建转录组任务。stages 只能是转录组链阶段(跨链拒绝)。")(create_rnaseq_job)
|
||||
rn.tool(name="generate_samples_yaml", description="把用户的自然语言样本描述解析为 samples.yaml 写入任务 input。")(generate_samples_yaml)
|
||||
rn.tool(name="ask_rnaseq_question", description="回答转录组流程/参数/实验设计类问题(流程知识库检索,带引用)。")(ask_rnaseq_question)
|
||||
|
||||
ext.tool(name="create_extension", description="创建扩展草稿(draft):上传代码文件清单(每项 {path, content 文本}) + 运行命令 + 显示名。")(create_extension)
|
||||
ext.tool(name="extension_status", description="查询扩展详情与状态(draft/build/trial/validated/confirming/published)。")(extension_status)
|
||||
ext.tool(name="build_extension", description="AI 转换扩展(原始命令→command_template+参数+依赖);LLM 失败自动回退规则版。")(build_extension)
|
||||
ext.tool(name="trial_run_extension", description="试运行扩展(docker 执行;未接入执行器时返回 EXT_RUN)。")(trial_run_extension)
|
||||
ext.tool(name="validate_extension", description="校验扩展(金标/试运行校验)。仅 built/trial_passed 可校验。")(validate_extension)
|
||||
ext.tool(name="confirm_extension", description="确认扩展参数/触发词(用户可编辑 nl_triggers);仅 validated/trial_passed 可确认。")(confirm_extension)
|
||||
ext.tool(name="publish_extension", description="发布扩展(仅 published 状态可被自然语言触发运行)。")(publish_extension)
|
||||
ext.tool(name="run_extension", description="正式运行已发布扩展(同步返回 run_result)。")(run_extension)
|
||||
ext.tool(name="repair_extension", description="AI 修复扩展(试运行失败上下文→LLM 修复;LLM 失败回退规则版)。")(repair_extension)
|
||||
|
||||
shared = [
|
||||
job_status, confirm_stage, register_job_files, rerun_stage, skip_stage,
|
||||
pause_job, resume_job, list_archives, reuse_archive, archive_job,
|
||||
fetch_source_file, inspect_file, transform_file,
|
||||
]
|
||||
for mcp in (gf, rn):
|
||||
for fn in shared:
|
||||
mcp.tool()(fn)
|
||||
# G12:11 个分析控制工具(Skill 编排,异步薄封装委托 skill_tools)
|
||||
for fn in SKILL_CONTROL_TOOLS:
|
||||
mcp.tool()(fn)
|
||||
return gf, rn, ext
|
||||
|
||||
|
||||
gene_family_mcp, rnaseq_mcp, extension_mcp = build_mcp_servers()
|
||||
@@ -31,10 +31,16 @@ from ..schemas import ArchiveCreateRequest, ArchiveRunRequest
|
||||
from ..services.bioflow import store
|
||||
from ..services.bioflow.archive import (
|
||||
archive_from_job,
|
||||
effective_stage_params,
|
||||
normalize_archive_name,
|
||||
remove_archive_dir,
|
||||
)
|
||||
from ..services.bioflow.reports import extract_figure_refs, locate_report, report_payload
|
||||
from ..services.bioflow.reports import (
|
||||
extract_figure_refs,
|
||||
locate_report,
|
||||
report_payload,
|
||||
validate_report_path,
|
||||
)
|
||||
from .bioflow import job_or_404
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -65,6 +71,10 @@ async def reports_register(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
job_id = str(payload.get("job_id") or "")
|
||||
record = await load_record(job_id) if job_id else None
|
||||
report_path = payload.get("md_path") or payload.get("report_path") or ""
|
||||
# G3:拒绝空路径/空文件(不静默登记空报告)
|
||||
ok, err = validate_report_path(report_path)
|
||||
if not ok:
|
||||
return _err("EMPTY_REPORT", err, 422)
|
||||
figures: list[str] = list(payload.get("figures") or [])
|
||||
if record is not None and not figures:
|
||||
located = locate_report(record.workspace_dir)
|
||||
@@ -96,12 +106,14 @@ async def create_archive(body: ArchiveCreateRequest) -> dict[str, Any]:
|
||||
existing = await store.find_archive_by_name(ctx.db, name)
|
||||
if existing is not None:
|
||||
return _err("ARCHIVE_EXISTS", f"归档名已存在: {name}", 409)
|
||||
# G6:保存各阶段实际生效参数(非创建时 record.params)
|
||||
effective_params = effective_stage_params(record.stage_chain, record._stage_states)
|
||||
report_path = locate_report(record.workspace_dir)
|
||||
result = await archive_from_job(
|
||||
{
|
||||
"id": record.id, "session_id": record.session_id,
|
||||
"title": record.title, "stage_chain": record.stage_chain,
|
||||
"params": record.params,
|
||||
"params": effective_params,
|
||||
},
|
||||
record.job_dir,
|
||||
get_settings().archives_dir_abs,
|
||||
@@ -116,7 +128,7 @@ async def create_archive(body: ArchiveCreateRequest) -> dict[str, Any]:
|
||||
"name": name,
|
||||
"description": body.description,
|
||||
"stage_chain": record.stage_chain,
|
||||
"params": record.params,
|
||||
"params": effective_params,
|
||||
"report_artifact": result.report_artifact,
|
||||
"disk_path": result.disk_path,
|
||||
"size_bytes": result.size_bytes,
|
||||
@@ -166,7 +178,10 @@ async def delete_archive(archive_id: str) -> JSONResponse:
|
||||
|
||||
@router.post("/archives/{archive_id}/run", status_code=201)
|
||||
async def archive_run(archive_id: str, body: ArchiveRunRequest) -> dict[str, Any]:
|
||||
"""复用发起(FR-BIO-023):装载阶段链+参数 → 新建任务(表单预填存档值)。"""
|
||||
"""复用发起(FR-BIO-023):装载阶段链+参数 → 新建任务(表单预填存档值)。
|
||||
|
||||
params 为存档时落库的各阶段实际生效参数(G6:非创建时 record.params)。
|
||||
"""
|
||||
ctx: Context = context_mod.context
|
||||
archive = await store.get_archive(ctx.db, archive_id)
|
||||
if archive is None:
|
||||
|
||||
@@ -22,7 +22,9 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
@@ -212,23 +214,43 @@ async def upload_file(
|
||||
dest = input_dir / _safe_name(file.filename)
|
||||
written = 0
|
||||
max_bytes = int(settings.max_upload_gb * 1024 ** 3)
|
||||
# G2:先落任务临时目录,再原子 rename 发布;同名冲突显式报错,不静默覆盖。
|
||||
tmp_dir = record.job_dir / ".tmp"
|
||||
tmp_dir.mkdir(parents=True, exist_ok=True)
|
||||
tmp_path = tmp_dir / f"{uuid.uuid4().hex[:8]}_{dest.name}"
|
||||
try:
|
||||
with dest.open("wb") as fh:
|
||||
# G5:逐块流式写盘(不整块进内存)
|
||||
with tmp_path.open("wb") as fh:
|
||||
while True:
|
||||
chunk = await file.read(1024 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
written += len(chunk)
|
||||
if written > max_bytes:
|
||||
fh.close()
|
||||
dest.unlink(missing_ok=True)
|
||||
return _err("UPLOAD_TOO_LARGE",
|
||||
f"超过上传上限 {settings.max_upload_gb}GB(已收 {written} 字节)", 413)
|
||||
raise _UploadTooLarge(written)
|
||||
fh.write(chunk)
|
||||
except _UploadTooLarge as exc:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
return _err("UPLOAD_TOO_LARGE",
|
||||
f"超过上传上限 {settings.max_upload_gb}GB(已收 {exc.written} 字节)", 413)
|
||||
except Exception as exc: # noqa: BLE001 - 写盘失败不登记(TC-BIO-API-050)
|
||||
logger.exception("files: 上传写盘失败 %s", job_id)
|
||||
dest.unlink(missing_ok=True)
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
return _err("INTERNAL", f"上传写盘失败: {exc}", 500)
|
||||
# 原子发布 + 同名冲突检测(G2)
|
||||
try:
|
||||
os.link(tmp_path, dest)
|
||||
except FileExistsError:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
return _err("FILE_EXISTS", f"同名文件已存在,拒绝覆盖: {dest.name}", 409)
|
||||
except OSError:
|
||||
# os.link 不支持的文件系统 → 退化:exists 检查 + rename(同事件循环内无 await 交错)
|
||||
if dest.exists() or dest.is_symlink():
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
return _err("FILE_EXISTS", f"同名文件已存在,拒绝覆盖: {dest.name}", 409)
|
||||
os.rename(tmp_path, dest)
|
||||
else:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
file_id = f"{job_id}:{dest.name}"
|
||||
# BUG-P7-01:文件落盘后自动续跑(await_inputs 任务重推;非等待态 no-op)
|
||||
await ctx.runner.retry_after_files(record)
|
||||
@@ -238,6 +260,14 @@ async def upload_file(
|
||||
}}
|
||||
|
||||
|
||||
class _UploadTooLarge(Exception):
|
||||
"""上传超限信号(携带已收字节数)。"""
|
||||
|
||||
def __init__(self, written: int) -> None:
|
||||
super().__init__("upload too large")
|
||||
self.written = written
|
||||
|
||||
|
||||
async def _register_path(record: Any, server_path: str, settings: Any) -> dict[str, Any]:
|
||||
"""服务器路径登记(FR-BIO-007):单文件或目录展开,任意数据位置(敏感路径黑名单拦截)。
|
||||
|
||||
|
||||
@@ -19,11 +19,27 @@ from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .manifests import ManifestError, artifact_paths, parse_manifest
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
INTERMEDIATE_DIR_NAMES = ("analysis",)
|
||||
|
||||
|
||||
def effective_stage_params(
|
||||
stage_chain: list[str], stage_states: dict[str, Any]
|
||||
) -> dict[str, Any]:
|
||||
"""G6:各阶段「实际生效」参数(confirm/rerun 更新后的 stage 参数)。
|
||||
|
||||
区别于创建时 record.params(创建后 confirm/rerun 只更新 _stage_states[stage].params,
|
||||
不回流 record.params)。存档与复用以此为单一事实源,避免参数失真。
|
||||
"""
|
||||
return {
|
||||
s: dict((stage_states.get(s) or {}).get("params") or {})
|
||||
for s in stage_chain
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ArchiveResult:
|
||||
id: str
|
||||
@@ -88,6 +104,11 @@ async def archive_from_job(
|
||||
|
||||
# 报告:任务目录内报告 HTML(final 内 *.html / report.html)随 final 已复制;
|
||||
# report_artifact 为 doc_artifacts 登记 id(由 reports 模块/BFF 登记后回传)。
|
||||
|
||||
# G1:按 manifest 必需产物清单补归档(不只 workspace/final/)。
|
||||
# 相对路径以 workspace 根解析并改写为归档内路径(去 workspace 前缀)。
|
||||
_copy_manifest_artifacts(job, job_dir, dest)
|
||||
|
||||
size_bytes = _dir_size(dest)
|
||||
return ArchiveResult(
|
||||
id=archive_id,
|
||||
@@ -98,6 +119,42 @@ async def archive_from_job(
|
||||
)
|
||||
|
||||
|
||||
def _copy_manifest_artifacts(
|
||||
job: dict[str, Any], job_dir: Path, dest: Path
|
||||
) -> list[str]:
|
||||
"""G1:按 manifest 的必需产物清单复制归档产物(漏归档修复)。
|
||||
|
||||
每个阶段的 manifest artifacts(full_path/path/relpath_or_hint)逐条复制;
|
||||
相对路径以 workspace 根解析,并改写为归档内路径(去 workspace 前缀)。
|
||||
跳过不存在/失败阶段无 manifest 的产物。返回已复制的归档内相对路径列表。
|
||||
"""
|
||||
stage_chain = job.get("stage_chain") or []
|
||||
workspace = Path(job_dir) / "workspace"
|
||||
copied: list[str] = []
|
||||
for stage in stage_chain:
|
||||
try:
|
||||
manifest = parse_manifest(workspace, stage)
|
||||
except ManifestError:
|
||||
continue
|
||||
for raw in artifact_paths(manifest.artifacts):
|
||||
p = Path(raw)
|
||||
if not p.is_absolute():
|
||||
p = workspace / p
|
||||
if not p.is_file():
|
||||
continue
|
||||
# 相对路径改写为归档内路径(去 workspace 前缀)
|
||||
try:
|
||||
rel = p.resolve().relative_to(workspace.resolve())
|
||||
except ValueError:
|
||||
rel = Path(p.name)
|
||||
target = dest / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
if not target.exists():
|
||||
shutil.copy2(p, target)
|
||||
copied.append(str(rel))
|
||||
return copied
|
||||
|
||||
|
||||
def _copytree_into(src: Path, dest: Path) -> None:
|
||||
"""复制目录内容到 dest(dest 可能已存在,逐项合并)。"""
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@@ -551,6 +551,38 @@ _STAGE_PARAM_SCHEMAS: dict[str, tuple[list[dict[str, Any]], list[dict[str, Any]]
|
||||
], _DOWNSTREAM_INPUTS, _DOWNSTREAM_OUTPUTS),
|
||||
}
|
||||
|
||||
# ---- G9:单一参数 schema 访问点(Catalog/表单/预检/工具/验收都读这一份)----
|
||||
|
||||
|
||||
def _schema_params_inputs_outputs(stage: str) -> tuple[list[dict[str, Any]], list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
"""阶段参数/输入/输出的单一读取点(含未知阶段默认兜底)。"""
|
||||
if stage in _STAGE_PARAM_SCHEMAS:
|
||||
return _STAGE_PARAM_SCHEMAS[stage]
|
||||
params = [{"name": "cpu", "type": "choice", "choices": ["4", "8", "16", "32"],
|
||||
"default": "4", "source": "runtime", "help": "容器 CPU 数"}]
|
||||
inputs = _DOWNSTREAM_INPUTS if stage not in ("identify", "genome_prepare") else []
|
||||
outputs = _DOWNSTREAM_OUTPUTS
|
||||
return params, inputs, outputs
|
||||
|
||||
|
||||
def stage_schema(stage: str) -> dict[str, Any]:
|
||||
"""G9:单一参数 schema 访问点——阶段名/耗时/依赖/参数/输入/输出/输入角色 合并为一份。
|
||||
|
||||
Catalog/表单/预检/工具/验收都读这一份(唯一事实源)。
|
||||
"""
|
||||
from .stage_assets import stage_input_roles
|
||||
|
||||
params, inputs, outputs = _schema_params_inputs_outputs(stage)
|
||||
return {
|
||||
"name": STAGE_INFO_SNAPSHOT.get(stage, {}).get("name") or stage,
|
||||
"estimated_time": int(STAGE_INFO_SNAPSHOT.get(stage, {}).get("estimated_time") or 0),
|
||||
"dependencies": STAGE_DEPENDENCIES_SNAPSHOT.get(stage, []),
|
||||
"params": params,
|
||||
"inputs": inputs,
|
||||
"outputs": outputs,
|
||||
"input_roles": stage_input_roles(stage),
|
||||
}
|
||||
|
||||
|
||||
class StageCatalog:
|
||||
"""阶段目录(数据来自 stage_image_map.json + 手写快照,不运行时反射)。"""
|
||||
@@ -578,29 +610,19 @@ class StageCatalog:
|
||||
if not self.has(stage):
|
||||
return None
|
||||
entry = self._map.image_for(stage) or {}
|
||||
params, inputs, outputs = _STAGE_PARAM_SCHEMAS.get(
|
||||
stage, ([], _DOWNSTREAM_INPUTS if stage not in ("identify", "genome_prepare") else [], [])
|
||||
)
|
||||
if stage not in _STAGE_PARAM_SCHEMAS:
|
||||
params, inputs, outputs = (
|
||||
[{"name": "cpu", "type": "choice", "choices": ["4", "8", "16", "32"],
|
||||
"default": "4", "source": "runtime", "help": "容器 CPU 数"}],
|
||||
_DOWNSTREAM_INPUTS if stage not in ("identify", "genome_prepare") else [],
|
||||
_DOWNSTREAM_OUTPUTS,
|
||||
)
|
||||
info = STAGE_INFO_SNAPSHOT.get(stage, {})
|
||||
schema = stage_schema(stage)
|
||||
return StageDef(
|
||||
name=stage,
|
||||
description=str(info.get("name") or stage),
|
||||
description=str(schema["name"] or stage),
|
||||
image=str(entry.get("image") or ""),
|
||||
module=str(entry.get("module") or ""),
|
||||
executable=bool(entry.get("executable")),
|
||||
estimated_time=int(info.get("estimated_time") or 0),
|
||||
params=params,
|
||||
inputs=inputs,
|
||||
outputs=outputs,
|
||||
estimated_time=int(schema["estimated_time"]),
|
||||
params=schema["params"],
|
||||
inputs=schema["inputs"],
|
||||
outputs=schema["outputs"],
|
||||
note=str(entry.get("note") or ""),
|
||||
dependencies=STAGE_DEPENDENCIES_SNAPSHOT.get(stage, []),
|
||||
dependencies=schema["dependencies"],
|
||||
)
|
||||
|
||||
def resolve_chain(self, chain: str | None) -> list[str] | None:
|
||||
@@ -612,32 +634,52 @@ class StageCatalog:
|
||||
return None
|
||||
return list(alias)
|
||||
|
||||
def auto_fill_prerequisites(self, stages: list[str]) -> tuple[list[str], list[str]]:
|
||||
def auto_fill_prerequisites(
|
||||
self, stages: list[str], satisfied_roles: list[str] | None = None
|
||||
) -> tuple[list[str], list[str]]:
|
||||
"""前置补全(FR-BIO-003):缺失的前置阶段自动补全。
|
||||
|
||||
G7/G8:satisfied_roles(已有 validated 资产的输入角色)对应的上游阶段
|
||||
跳过不补跑(如 role=counts 满足时做 DE 不补跑 qc/align/quant/sample_qc)。
|
||||
|
||||
返回 (补全后的链, 被补全的阶段名列表)。
|
||||
链内阶段按依赖拓扑顺序排列(依赖在前)。
|
||||
"""
|
||||
from .stage_assets import upstream_stages_for_roles
|
||||
|
||||
skip_stages = upstream_stages_for_roles(satisfied_roles or [])
|
||||
missing: list[str] = []
|
||||
added: list[str] = []
|
||||
chain_set = set(stages)
|
||||
# 反复补全直到闭合(依赖的依赖也补)
|
||||
# 反复补全直到闭合(依赖的依赖也补;被角色资产满足的上游跳过)
|
||||
changed = True
|
||||
while changed:
|
||||
changed = False
|
||||
for s in list(stages):
|
||||
for dep in STAGE_DEPENDENCIES_SNAPSHOT.get(s, []):
|
||||
if dep not in chain_set and self.has(dep):
|
||||
if dep in chain_set:
|
||||
continue
|
||||
if dep in skip_stages:
|
||||
continue
|
||||
if self.has(dep):
|
||||
chain_set.add(dep)
|
||||
missing.append(dep)
|
||||
added.append(dep)
|
||||
changed = True
|
||||
ordered = topo_order([s for s in GENE_FAMILY_CHAIN if s in chain_set]
|
||||
or list(chain_set))
|
||||
# G7/G8:被角色资产满足的上游阶段从最终链中剔除(不补跑)
|
||||
if skip_stages:
|
||||
ordered = [s for s in ordered if s not in skip_stages]
|
||||
return ordered, added
|
||||
|
||||
def dependencies(self, stage: str) -> list[str]:
|
||||
return STAGE_DEPENDENCIES_SNAPSHOT.get(stage, [])
|
||||
return stage_schema(stage)["dependencies"]
|
||||
|
||||
def stage_input_roles(self, stage: str) -> list[str]:
|
||||
"""G8:阶段依赖的输入角色(数据角色)。"""
|
||||
from .stage_assets import stage_input_roles
|
||||
return stage_input_roles(stage)
|
||||
|
||||
|
||||
def topo_order(stages: list[str]) -> list[str]:
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
"""app/services/bioflow/delivery.py —— 交付清单(G21)+ 终报告范围(G22)+ 资源预算(G23)。
|
||||
|
||||
- G21 交付清单完整性:delivery_validate(manifest) 做八项完整性检查
|
||||
(selected_stages / run_id / input_sources / output_sources / config /
|
||||
required_results / validation_values / review_status),返回
|
||||
{ok, checks{八项:bool}, incomplete[{item, reason, error(G13)}]}。
|
||||
- G22 终报告范围:classify_report_scope(status, source) 五分类
|
||||
(user_provided / reused / not_executed / skipped / failed)。
|
||||
- G23 资源预算:check_download_budget(size_bytes, budget_bytes)(超限→
|
||||
DOWNLOAD_BUDGET_EXCEEDED)+ check_disk_available(dest_dir, needed_bytes)
|
||||
(不足→DISK_FULL),均 ok=False + 结构化错误。
|
||||
|
||||
服务层纪律:纯 dict/stdlib,无 pydantic。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .errors import error_object, error_result
|
||||
from .source_fetcher import check_disk_space
|
||||
|
||||
# G21 八项完整性检查项
|
||||
DELIVERY_ITEMS = (
|
||||
"selected_stages", "run_id", "input_sources", "output_sources",
|
||||
"config", "required_results", "validation_values", "review_status",
|
||||
)
|
||||
|
||||
# G22 五分类
|
||||
SCOPE_USER_PROVIDED = "user_provided"
|
||||
SCOPE_REUSED = "reused"
|
||||
SCOPE_NOT_EXECUTED = "not_executed"
|
||||
SCOPE_SKIPPED = "skipped"
|
||||
SCOPE_FAILED = "failed"
|
||||
|
||||
|
||||
def delivery_validate(manifest: dict[str, Any]) -> dict[str, Any]:
|
||||
"""G21:校验交付清单完整性(八项)。
|
||||
|
||||
返回 {"ok": bool, "checks": {item: bool},
|
||||
"incomplete": [{"item", "reason", "error": {G13 对象}}]}。
|
||||
"""
|
||||
manifest = manifest or {}
|
||||
checks: dict[str, bool] = {}
|
||||
incomplete: list[dict[str, Any]] = []
|
||||
for item in DELIVERY_ITEMS:
|
||||
value = manifest.get(item)
|
||||
ok = bool(value)
|
||||
checks[item] = ok
|
||||
if not ok:
|
||||
incomplete.append({
|
||||
"item": item,
|
||||
"reason": f"缺 {item}",
|
||||
"error": error_object("INCOMPLETE_DELIVERY", f"交付清单缺项: {item}",
|
||||
category="input"),
|
||||
})
|
||||
return {"ok": not incomplete, "checks": checks, "incomplete": incomplete}
|
||||
|
||||
|
||||
def publish_delivery(manifest: dict[str, Any]) -> dict[str, Any]:
|
||||
"""G21:发布交付清单(校验通过置 review_status=approved)。"""
|
||||
check = delivery_validate(manifest)
|
||||
if not check["ok"]:
|
||||
return {"ok": False, "published": False, **check}
|
||||
manifest["review_status"] = "approved"
|
||||
return {"ok": True, "published": True, "review_status": "approved",
|
||||
"incomplete": []}
|
||||
|
||||
|
||||
# ---- G22 终报告范围 ----
|
||||
|
||||
def classify_report_scope(status: str, source: str | None = None) -> str:
|
||||
"""G22:阶段状态 → 终报告范围五分类。
|
||||
|
||||
failed→failed;skipped→skipped;done + source=reuse→reused;
|
||||
done + source=user/空→user_provided;其余非终态→not_executed。
|
||||
"""
|
||||
s = str(status or "")
|
||||
if s == "failed":
|
||||
return SCOPE_FAILED
|
||||
if s == "skipped":
|
||||
return SCOPE_SKIPPED
|
||||
if s == "done":
|
||||
return SCOPE_REUSED if source == "reuse" else SCOPE_USER_PROVIDED
|
||||
return SCOPE_NOT_EXECUTED
|
||||
|
||||
|
||||
def build_report(
|
||||
manifest: dict[str, Any],
|
||||
stage_sources: dict[str, str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""G22:终报告结构(执行范围 / 数据来源与核验 / 方法版本 / QC / 科学前提 /
|
||||
失败跳过未覆盖 / 产物清单)。"""
|
||||
stages = list(manifest.get("selected_stages") or [])
|
||||
sources = dict(stage_sources or {})
|
||||
scope = [{"stage": s, "source": sources.get(s, SCOPE_USER_PROVIDED)} for s in stages]
|
||||
|
||||
failed = [s for s in stages if sources.get(s) == SCOPE_FAILED]
|
||||
skipped = [s for s in stages if sources.get(s) == SCOPE_SKIPPED]
|
||||
not_executed = [s for s in stages if sources.get(s) == SCOPE_NOT_EXECUTED]
|
||||
|
||||
config = manifest.get("config") or {}
|
||||
return {
|
||||
"execution_scope": scope,
|
||||
"data_sources": {
|
||||
"inputs": list(manifest.get("input_sources") or []),
|
||||
"outputs": list(manifest.get("output_sources") or []),
|
||||
},
|
||||
"method_version": str(manifest.get("plan_revision") or manifest.get("run_id") or ""),
|
||||
"qc": {"validation_values": dict(manifest.get("validation_values") or {})},
|
||||
"scientific_preconditions": config.get("scientific_preconditions", []),
|
||||
"failed_skipped_uncovered": {
|
||||
"failed": failed, "skipped": skipped, "not_executed": not_executed,
|
||||
},
|
||||
"artifacts": list(manifest.get("required_results") or []),
|
||||
}
|
||||
|
||||
|
||||
# ---- G23 资源预算门禁 ----
|
||||
|
||||
def check_download_budget(size_bytes: int, budget_bytes: int) -> dict[str, Any]:
|
||||
"""G23:下载预算门禁——超限阻断并返回结构化错误。"""
|
||||
if size_bytes > budget_bytes:
|
||||
return error_result(
|
||||
"DOWNLOAD_BUDGET_EXCEEDED",
|
||||
f"下载预算超限:本次 {size_bytes} > 预算 {budget_bytes}",
|
||||
)
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
def check_disk_available(dest_dir: str | Path, needed_bytes: int) -> dict[str, Any]:
|
||||
"""G23:磁盘门禁——剩余空间不足阻断并返回结构化错误。"""
|
||||
try:
|
||||
ok, free = check_disk_space(dest_dir, needed_bytes)
|
||||
except OSError as exc:
|
||||
return error_result("PATH_NOT_EXIST", f"目标目录不可用:{exc}")
|
||||
if not ok:
|
||||
return error_result(
|
||||
"DISK_FULL", f"磁盘空间不足:需 {needed_bytes} 字节,剩余 {free} 字节",
|
||||
)
|
||||
return {"ok": True, "free_bytes": free}
|
||||
@@ -0,0 +1,196 @@
|
||||
"""app/services/bioflow/errors.py —— 统一结构化错误对象(G13)+ 确定性分类器(G14)。
|
||||
|
||||
失败返回统一携带结构化错误对象(替代裸字符串 message):
|
||||
{"ok": False, "code": "<码>", "message": "<摘要>",
|
||||
"error": {error_id, category, code, summary, evidence_refs,
|
||||
diagnosis_status, retryable_without_change, suggested_actions}}
|
||||
|
||||
8 类 category(G14:确定性映射):
|
||||
input / scientific_precondition / reference / environment /
|
||||
transient / resource / adapter / artifact
|
||||
|
||||
G14 分类纪律:退出码/日志是「证据」不是「结论」——
|
||||
- 日志含 OOM → resource;日志含镜像缺失 → environment;日志含科学前提不满足 → scientific_precondition。
|
||||
- 仅凭「非零退出码」无法定类 → adapter + undiagnosed(不臆测)。
|
||||
|
||||
服务层纪律:纯 dict/stdlib,无 pydantic。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from typing import Any
|
||||
|
||||
CATEGORIES = (
|
||||
"input", "scientific_precondition", "reference", "environment",
|
||||
"transient", "resource", "adapter", "artifact",
|
||||
)
|
||||
|
||||
# 明显映射:错误码 → (category, diagnosis_status, retryable_without_change, suggested_actions)
|
||||
_CODE_META: dict[str, tuple[str, str, bool, list[str]]] = {
|
||||
# ---- resource(资源耗尽)----
|
||||
"DISK_FULL": ("resource", "diagnosed", False, ["清理磁盘空间后重试"]),
|
||||
"UPLOAD_TOO_LARGE": ("resource", "diagnosed", False, ["减小文件大小或调大上传上限"]),
|
||||
"DOWNLOAD_BUDGET_EXCEEDED": ("resource", "diagnosed", False, ["调大下载预算或减小下载量"]),
|
||||
# ---- environment(运行环境)----
|
||||
"OSS_UNCONFIGURED": ("environment", "diagnosed", False, ["配置 BIOFLOW_OSS_ENDPOINT 或改用预签名 URL"]),
|
||||
"SSHPASS_MISSING": ("environment", "diagnosed", False, ["本机安装 sshpass 或改用密钥认证"]),
|
||||
# ---- reference(参考数据)----
|
||||
"REFERENCE_NOT_FOUND": ("reference", "diagnosed", False, ["补齐参考基因组/HMM/索引后重试"]),
|
||||
# ---- transient(临时/可重试)----
|
||||
"NETWORK_ERROR": ("transient", "diagnosed", True, ["重试下载"]),
|
||||
"LAN_TIMEOUT": ("transient", "diagnosed", True, ["重试局域网拉取"]),
|
||||
"TRUNCATED": ("transient", "diagnosed", True, ["重试下载"]),
|
||||
"LAN_ERROR": ("transient", "diagnosed", True, ["重试局域网拉取"]),
|
||||
"HTTP_ERROR": ("transient", "diagnosed", True, ["确认 URL 可达或稍后重试"]),
|
||||
# ---- input(用户输入/文件/参数)----
|
||||
"EMPTY_SOURCE": ("input", "diagnosed", False, ["提供非空源地址"]),
|
||||
"BAD_SCHEME": ("input", "diagnosed", False, ["改用 http/https/oss 协议"]),
|
||||
"BAD_OSS": ("input", "diagnosed", False, ["修正 OSS 地址为 oss://bucket/key"]),
|
||||
"PATH_NOT_EXIST": ("input", "diagnosed", False, ["确认路径存在且可达"]),
|
||||
"IS_DIR": ("input", "diagnosed", False, ["改为单文件路径"]),
|
||||
"NOT_FOUND": ("input", "diagnosed", False, ["确认目标存在"]),
|
||||
"BAD_FORMAT": ("input", "diagnosed", False, ["选择支持的目标格式 fasta/gff3/tsv/samples.yaml"]),
|
||||
"EMPTY_TABLE": ("input", "diagnosed", False, ["提供非空表格数据"]),
|
||||
"MISSING_COLUMNS": ("input", "diagnosed", False, ["确认列映射(名称/序列/r1/r2)"]),
|
||||
"VALIDATION": ("input", "diagnosed", False, ["修正输入以满足校验"]),
|
||||
"VALIDATION_FAILED": ("input", "diagnosed", False, ["修正数据后重转"]),
|
||||
"UNKNOWN_STAGE": ("input", "diagnosed", False, ["选择有效阶段"]),
|
||||
"UNKNOWN_PARAMS": ("input", "diagnosed", False, ["移除未知参数"]),
|
||||
"BAD_ROLE": ("input", "diagnosed", False, ["使用合法角色名(非空、无路径分隔符/..)"]),
|
||||
"EMPTY_REPORT": ("input", "diagnosed", False, ["提供非空报告路径/文件"]),
|
||||
"EMPTY_SAMPLES": ("input", "diagnosed", False, ["提供含样本的描述"]),
|
||||
"INCOMPLETE_DELIVERY": ("input", "diagnosed", False, ["补齐交付清单缺失项(selected_stages/run_id/必需结果等)后重试"]),
|
||||
# ---- adapter(适配/映射)----
|
||||
"UNSUPPORTED": ("adapter", "diagnosed", False, ["改用支持的转换路径"]),
|
||||
"CROSS_CHAIN": ("adapter", "diagnosed", False, ["选择本能力单元内的阶段"]),
|
||||
# ---- artifact(产物)----
|
||||
"ARCHIVE_EXISTS": ("artifact", "diagnosed", False, ["更换归档名"]),
|
||||
"ARCHIVE_FAILED": ("artifact", "diagnosed", False, ["检查任务目录完整性后重试"]),
|
||||
"ARCHIVE_NOT_FOUND": ("artifact", "diagnosed", False, ["确认归档名存在"]),
|
||||
"ASSET_EXISTS": ("artifact", "diagnosed", False, ["更换角色或文件名"]),
|
||||
"PUBLISH_FAILED": ("artifact", "diagnosed", False, ["检查目标目录可写性后重试"]),
|
||||
"NO_MANIFEST": ("artifact", "diagnosed", False, ["检查阶段产物 manifest 是否产出"]),
|
||||
# ---- scientific_precondition(科学前提)----
|
||||
"SCIENTIFIC_PRECONDITION": ("scientific_precondition", "diagnosed", False, ["确认科学前提满足后重试"]),
|
||||
}
|
||||
|
||||
# 关键词兜底(summary 命中即映射;用于非枚举码)
|
||||
_KEYWORD_META: list[tuple[tuple[str, ...], tuple[str, str, bool, list[str]]]] = [
|
||||
(("out of memory", "oom", "memoryerror", "oom-killed", "memory limit", "内存"),
|
||||
("resource", "diagnosed", False, ["调大内存或减小数据规模"])),
|
||||
(("no such image", "image not found", "pull access denied", "镜像", "docker"),
|
||||
("environment", "diagnosed", False, ["确认镜像存在且可拉取"])),
|
||||
(("no significant", "not enough", "insufficient", "科学前提", "no differentially", "not found enough"),
|
||||
("scientific_precondition", "diagnosed", False, ["确认科学前提满足后重试"])),
|
||||
(("timeout", "超时", "network", "网络", "unreachable", "connecterror", "timed out"),
|
||||
("transient", "diagnosed", True, ["重试"])),
|
||||
(("reference", "参考", "hmm", "pfam", "gtf", "index"),
|
||||
("reference", "diagnosed", False, ["补齐参考数据后重试"])),
|
||||
(("manifest", "no_manifest", "产物缺失"),
|
||||
("artifact", "diagnosed", False, ["检查阶段产物 manifest"])),
|
||||
]
|
||||
|
||||
# 阶段失败 reason(runner _fail_stage)→ (category, diagnosis_status, retryable, actions)
|
||||
_REASON_META: dict[str, tuple[str, str, bool, list[str]]] = {
|
||||
"timeout": ("transient", "diagnosed", True, ["重跑该阶段"]),
|
||||
"exit_code": ("adapter", "undiagnosed", False, ["查看阶段日志定位失败原因", "修正后重跑"]),
|
||||
"no_manifest": ("artifact", "diagnosed", False, ["检查阶段是否产出 manifest"]),
|
||||
"manifest_error": ("artifact", "diagnosed", False, ["检查 manifest JSON 完整性"]),
|
||||
"manifest_failed": ("artifact", "diagnosed", False, ["检查阶段产物 manifest status"]),
|
||||
}
|
||||
|
||||
# 退出码 137 = 128 + SIGKILL(常为 OOM kill)
|
||||
_OOM_EXIT_CODES = {137, 139}
|
||||
|
||||
|
||||
def classify(code: str | None, summary: str = "") -> tuple[str, str, bool, list[str]]:
|
||||
"""错误码/摘要 → (category, diagnosis_status, retryable_without_change, suggested_actions)。"""
|
||||
code = str(code or "")
|
||||
summary = str(summary or "")
|
||||
if code in _CODE_META:
|
||||
cat, diag, retry, actions = _CODE_META[code]
|
||||
return cat, diag, retry, list(actions)
|
||||
low = summary.lower()
|
||||
for kws, meta in _KEYWORD_META:
|
||||
if any(k in low for k in kws):
|
||||
cat, diag, retry, actions = meta
|
||||
return cat, diag, retry, list(actions)
|
||||
return "adapter", "undiagnosed", False, []
|
||||
|
||||
|
||||
def classify_reason(
|
||||
reason: str | None,
|
||||
*,
|
||||
exit_code: int | None = None,
|
||||
log_tail: str | None = None,
|
||||
) -> tuple[str, str, bool, list[str]]:
|
||||
"""阶段失败原因 + 证据(退出码/日志)→ 确定性 8 类。
|
||||
|
||||
确定性 reason 优先(timeout / manifest 系列);泛化 exit_code 用证据(退出码/日志)
|
||||
定类——仅凭非零退出码不定类(undiagnosed),不臆测。
|
||||
"""
|
||||
reason = str(reason or "")
|
||||
log = (log_tail or "").lower()
|
||||
|
||||
# 1. 确定性 reason(非 exit_code)优先:timeout / manifest 系列
|
||||
if reason in _REASON_META and reason != "exit_code":
|
||||
cat, diag, retry, actions = _REASON_META[reason]
|
||||
return cat, diag, retry, list(actions)
|
||||
|
||||
# 2. 泛化 exit_code / 未知 reason:退出码是证据(137/139 = OOM kill)
|
||||
if exit_code in _OOM_EXIT_CODES:
|
||||
return "resource", "diagnosed", False, ["容器可能被 OOM kill,调大内存后重跑"]
|
||||
|
||||
# 3. 日志证据定类
|
||||
for kws, meta in _KEYWORD_META:
|
||||
if any(k in log for k in kws):
|
||||
cat, diag, retry, actions = meta
|
||||
return cat, diag, retry, list(actions)
|
||||
|
||||
# 4. 兜底:exit_code → adapter undiagnosed;未知 → 同
|
||||
if reason in _REASON_META:
|
||||
cat, diag, retry, actions = _REASON_META[reason]
|
||||
return cat, diag, retry, list(actions)
|
||||
return "adapter", "undiagnosed", False, []
|
||||
|
||||
|
||||
def error_object(
|
||||
code: str | None,
|
||||
summary: str = "",
|
||||
*,
|
||||
category: str | None = None,
|
||||
evidence_refs: list[str] | None = None,
|
||||
diagnosis_status: str | None = None,
|
||||
retryable_without_change: bool | None = None,
|
||||
suggested_actions: list[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""构造统一结构化错误对象(8 字段)。"""
|
||||
cat, diag, retry, actions = classify(code, summary)
|
||||
return {
|
||||
"error_id": uuid.uuid4().hex[:16],
|
||||
"category": category or cat,
|
||||
"code": code or "INTERNAL",
|
||||
"summary": summary,
|
||||
"evidence_refs": list(evidence_refs or []),
|
||||
"diagnosis_status": diagnosis_status or diag,
|
||||
"retryable_without_change": retry if retryable_without_change is None else retryable_without_change,
|
||||
"suggested_actions": list(suggested_actions or actions),
|
||||
}
|
||||
|
||||
|
||||
def error_result(
|
||||
code: str | None,
|
||||
summary: str = "",
|
||||
*,
|
||||
evidence_refs: list[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""失败返回统一信封:{"ok": False, "code", "message", "error"}。
|
||||
|
||||
code 同时出现在顶层与 error 对象内(兼容 Phase A 的 r["code"] 断言)。
|
||||
"""
|
||||
return {
|
||||
"ok": False,
|
||||
"code": code or "INTERNAL",
|
||||
"message": summary,
|
||||
"error": error_object(code, summary, evidence_refs=evidence_refs),
|
||||
}
|
||||
@@ -24,8 +24,8 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
EVENT_TYPES = (
|
||||
"job_created", "plan_ready", "stage_started", "stage_log", "stage_progress",
|
||||
"stage_done", "stage_failed", "stage_skipped", "stage_await_confirm",
|
||||
"stage_await_files",
|
||||
"stage_done", "stage_failed", "stage_skipped", "stage_partial",
|
||||
"stage_await_confirm", "stage_await_files",
|
||||
"job_done", "job_paused", "job_archived", "job_recovered",
|
||||
# 用户自定义扩展分析事件帧(契约 §4;并行链路,不触碰现有 job_* 语义)
|
||||
"extension_created", "extension_build", "extension_trial", "extension_repair",
|
||||
@@ -208,6 +208,9 @@ class EventBus:
|
||||
elif t == "stage_skipped":
|
||||
st = stage_states.setdefault(str(d.get("stage")), {"stage": str(d.get("stage"))})
|
||||
st["status"] = "skipped"
|
||||
elif t == "stage_partial":
|
||||
st = stage_states.setdefault(str(d.get("stage")), {"stage": str(d.get("stage"))})
|
||||
st["status"] = "partial_success"
|
||||
elif t == "stage_progress":
|
||||
st = stage_states.setdefault(str(d.get("stage")), {"stage": str(d.get("stage"))})
|
||||
pct = d.get("percent")
|
||||
|
||||
@@ -0,0 +1,438 @@
|
||||
"""app/services/bioflow/file_transform.py —— 非标准文件预处理(探测 + 转换 + 校验)。
|
||||
|
||||
FR-BIO-014 文件医生闭环:
|
||||
- inspect_file:内容嗅探(扩展名仅参考),返回结构化摘要(预览行/列结构/大小/
|
||||
格式猜测/乱码检测/与目标阶段期望格式的差异)。
|
||||
- transform_file:LLM 抽取列映射/清洗规则 + 规则脚本执行,把 xlsx/csv/tsv/txt 转换
|
||||
为 fasta/gff3/tsv/samples.yaml;转换结果过格式验证器硬校验,失败返回原因供追问
|
||||
循环(≤3 次由上层 QA 控制)。
|
||||
|
||||
服务层纪律:纯 dataclass/dict/stdlib + yaml/openpyxl,无 pydantic。
|
||||
LLM 可注入(默认走 .llm.complete_json),测试注入 fake。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any, Awaitable, Callable
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PREVIEW_LINES = 10
|
||||
PREVIEW_BYTES = 4000
|
||||
|
||||
# 支持的目标格式
|
||||
TARGET_FORMATS = ("fasta", "gff3", "tsv", "samples.yaml")
|
||||
|
||||
# 乱码检测:非 ASCII 可打印且非 CJK 的异常字节比例阈值
|
||||
GARBLED_RATIO = 0.30
|
||||
|
||||
|
||||
class TransformError(Exception):
|
||||
def __init__(self, code: str, message: str, details: dict[str, Any] | None = None) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.details = details or {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 探测
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _guess_format(name: str, head: bytes) -> str:
|
||||
lower = name.lower()
|
||||
if lower.endswith((".fa", ".fasta", ".faa", ".fna", ".pep", ".cds")):
|
||||
return "fasta"
|
||||
if lower.endswith((".gff", ".gff3")):
|
||||
return "gff3"
|
||||
if lower.endswith((".gtf",)):
|
||||
return "gtf"
|
||||
if lower.endswith((".vcf", ".vcf.gz")):
|
||||
return "vcf"
|
||||
if lower.endswith((".tsv",)):
|
||||
return "tsv"
|
||||
if lower.endswith((".csv",)):
|
||||
return "csv"
|
||||
if lower.endswith((".xlsx", ".xls")):
|
||||
return "xlsx"
|
||||
if lower.endswith((".yaml", ".yml")):
|
||||
return "yaml"
|
||||
# 内容嗅探兜底
|
||||
text = head.decode("utf-8", errors="replace")
|
||||
if text.lstrip().startswith(">"):
|
||||
return "fasta"
|
||||
if "\t" in text and len(text.splitlines()[0].split("\t")) >= 3:
|
||||
return "tsv"
|
||||
return "txt"
|
||||
|
||||
|
||||
def _read_text(path: Path) -> str:
|
||||
raw = path.read_bytes()
|
||||
for enc in ("utf-8", "gb18030", "latin-1"):
|
||||
try:
|
||||
return raw.decode(enc)
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
return raw.decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
def _garbled_ratio(text: str) -> float:
|
||||
"""异常字符比例:控制字符 + 替换符 + 私有区(不含常规空白)。"""
|
||||
if not text:
|
||||
return 0.0
|
||||
bad = 0
|
||||
for ch in text:
|
||||
o = ord(ch)
|
||||
if o == 0xFFFD or (o < 32 and ch not in "\t\n\r") or (0xE000 <= o <= 0xF8FF):
|
||||
bad += 1
|
||||
return bad / len(text)
|
||||
|
||||
|
||||
def _preview(text: str) -> list[str]:
|
||||
return text.splitlines()[:PREVIEW_LINES]
|
||||
|
||||
|
||||
def _columns(path: Path, fmt: str) -> list[str]:
|
||||
"""表格列名提取(tsv/csv/xlsx)。"""
|
||||
try:
|
||||
if fmt == "xlsx":
|
||||
import openpyxl
|
||||
wb = openpyxl.load_workbook(path, read_only=True, data_only=True)
|
||||
try:
|
||||
ws = wb.active
|
||||
row = next(ws.iter_rows(min_row=1, max_row=1, values_only=True), None)
|
||||
return [str(c or "").strip() for c in (row or [])]
|
||||
finally:
|
||||
wb.close()
|
||||
if fmt in ("tsv", "csv"):
|
||||
text = _read_text(path)
|
||||
reader = csv.reader(io.StringIO(text), delimiter="\t" if fmt == "tsv" else ",")
|
||||
first = next(reader, None)
|
||||
return [str(c or "").strip() for c in (first or [])]
|
||||
except Exception as exc: # noqa: BLE001 - 列提取失败降级空
|
||||
logger.warning("file_transform: 列提取失败 %s: %s", path, exc)
|
||||
return []
|
||||
|
||||
|
||||
def inspect_file(path: str | Path, *, preview_lines: int = PREVIEW_LINES) -> dict[str, Any]:
|
||||
"""内容嗅探摘要(扩展名仅参考;失败返回 ok=False + 原因,不抛)。"""
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": f"文件不存在: {path}"}
|
||||
try:
|
||||
size = p.stat().st_size
|
||||
head = p.read_bytes()[:PREVIEW_BYTES]
|
||||
fmt = _guess_format(p.name, head)
|
||||
text = _read_text(p) if size <= 4 * 1024 * 1024 else _read_text_preview(p)
|
||||
return {
|
||||
"ok": True,
|
||||
"path": str(p),
|
||||
"name": p.name,
|
||||
"ext": p.suffix.lower() or "",
|
||||
"size": size,
|
||||
"format": fmt,
|
||||
"preview": _preview(text)[:preview_lines],
|
||||
"columns": _columns(p, fmt),
|
||||
"garbled": _garbled_ratio(text),
|
||||
"line_count": text.count("\n") + 1 if text else 0,
|
||||
}
|
||||
except OSError as exc:
|
||||
return {"ok": False, "code": "READ_ERROR", "message": f"文件读取失败: {exc}"}
|
||||
|
||||
|
||||
def _read_text_preview(path: Path) -> str:
|
||||
"""大文件只读头部做预览/乱码检测,避免整读内存。"""
|
||||
with path.open("rb") as fh:
|
||||
head = fh.read(PREVIEW_BYTES)
|
||||
for enc in ("utf-8", "gb18030", "latin-1"):
|
||||
try:
|
||||
return head.decode(enc)
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
return head.decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 校验器
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def validate_fasta(path: str | Path) -> dict[str, Any]:
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
return {"ok": False, "reason": "文件不存在"}
|
||||
text = _read_text(p)
|
||||
records = re.findall(r"^>.*$", text, flags=re.M)
|
||||
if not records:
|
||||
return {"ok": False, "reason": "无 FASTA 记录(缺 '>' 头行)"}
|
||||
# 序列非空:头行之间至少存在非空白序列行
|
||||
body = re.sub(r"^>.*$", "", text, flags=re.M)
|
||||
if not body.strip():
|
||||
return {"ok": False, "reason": "FASTA 记录缺少序列"}
|
||||
return {"ok": True, "records": len(records)}
|
||||
|
||||
|
||||
def validate_gff3(path: str | Path) -> dict[str, Any]:
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
return {"ok": False, "reason": "文件不存在"}
|
||||
bad = 0
|
||||
total = 0
|
||||
for line in _read_text(p).splitlines():
|
||||
if not line.strip() or line.startswith("#"):
|
||||
continue
|
||||
total += 1
|
||||
cols = line.split("\t")
|
||||
if len(cols) != 9:
|
||||
bad += 1
|
||||
continue
|
||||
try:
|
||||
int(cols[3])
|
||||
int(cols[4])
|
||||
except ValueError:
|
||||
bad += 1
|
||||
continue
|
||||
if cols[6] not in ("+", "-", "."):
|
||||
bad += 1
|
||||
if total == 0:
|
||||
return {"ok": False, "reason": "无 GFF3 数据行"}
|
||||
if bad:
|
||||
return {"ok": False, "reason": f"{bad}/{total} 行列数或字段非法"}
|
||||
return {"ok": True, "records": total}
|
||||
|
||||
|
||||
def validate_tsv(path: str | Path) -> dict[str, Any]:
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
return {"ok": False, "reason": "文件不存在"}
|
||||
widths: set[int] = set()
|
||||
for line in _read_text(p).splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
widths.add(len(line.split("\t")))
|
||||
if len(widths) > 1:
|
||||
return {"ok": False, "reason": f"列数不一致: {sorted(widths)}"}
|
||||
if not widths:
|
||||
return {"ok": False, "reason": "空文件"}
|
||||
return {"ok": True, "columns": widths.pop()}
|
||||
|
||||
|
||||
def validate_samples_yaml(path: str | Path) -> dict[str, Any]:
|
||||
import yaml
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
return {"ok": False, "reason": "文件不存在"}
|
||||
try:
|
||||
raw = yaml.safe_load(_read_text(p))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"ok": False, "reason": f"YAML 解析失败: {exc}"}
|
||||
samples = (raw or {}).get("samples") if isinstance(raw, dict) else None
|
||||
if not isinstance(samples, dict) or not samples:
|
||||
return {"ok": False, "reason": "缺 samples 映射或为空"}
|
||||
for name, entry in samples.items():
|
||||
if not isinstance(entry, dict) or not (entry.get("r1") or entry.get("r2")):
|
||||
return {"ok": False, "reason": f"样本 {name} 缺 r1/r2"}
|
||||
return {"ok": True, "samples": len(samples)}
|
||||
|
||||
|
||||
_VALIDATORS = {
|
||||
"fasta": validate_fasta,
|
||||
"gff3": validate_gff3,
|
||||
"tsv": validate_tsv,
|
||||
"samples.yaml": validate_samples_yaml,
|
||||
}
|
||||
|
||||
|
||||
def validate_format(path: str | Path, target_format: str) -> dict[str, Any]:
|
||||
fn = _VALIDATORS.get(target_format)
|
||||
if fn is None:
|
||||
return {"ok": False, "reason": f"不支持的目标格式: {target_format}"}
|
||||
return fn(path)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 转换
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
LLMFn = Callable[[str, str], Awaitable[Any]]
|
||||
|
||||
|
||||
def _read_table(path: Path) -> tuple[list[str], list[dict[str, str]]]:
|
||||
"""读表格(tsv/csv/xlsx)→ (表头, 行 dict 列表)。"""
|
||||
fmt = _guess_format(path.name, path.read_bytes()[:64])
|
||||
if fmt == "xlsx":
|
||||
import openpyxl
|
||||
wb = openpyxl.load_workbook(path, read_only=True, data_only=True)
|
||||
try:
|
||||
ws = wb.active
|
||||
rows = [list(r) for r in ws.iter_rows(values_only=True)]
|
||||
finally:
|
||||
wb.close()
|
||||
if not rows:
|
||||
return [], []
|
||||
headers = [str(c or "").strip() for c in rows[0]]
|
||||
return headers, [dict(zip(headers, ["" if v is None else str(v).strip() for v in r])) for r in rows[1:]]
|
||||
text = _read_text(path)
|
||||
delimiter = "\t" if fmt == "tsv" else ","
|
||||
reader = csv.reader(io.StringIO(text), delimiter=delimiter)
|
||||
rows = list(reader)
|
||||
if not rows:
|
||||
return [], []
|
||||
headers = [str(c or "").strip() for c in rows[0]]
|
||||
return headers, [dict(zip(headers, ["" if v is None else str(v).strip() for v in r])) for r in rows[1:]]
|
||||
|
||||
|
||||
def _default_column_map(headers: list[str], target: str) -> dict[str, str]:
|
||||
"""按表头名启发式推断列映射(名称/序列/基因名等)。"""
|
||||
m: dict[str, str] = {}
|
||||
for h in headers:
|
||||
lh = h.lower()
|
||||
if target == "fasta":
|
||||
if not m.get("seq") and any(k in lh for k in ("seq", "sequence", "序列", "蛋白", "pep", "cds")):
|
||||
m["seq"] = h
|
||||
elif not m.get("name") and any(k in lh for k in ("name", "id", "gene", "名称", "基因", "基因名", "accession")):
|
||||
m["name"] = h
|
||||
elif target == "samples.yaml":
|
||||
if "r1" in lh or "read1" in lh or "forward" in lh:
|
||||
m.setdefault("r1", h)
|
||||
elif "r2" in lh or "read2" in lh or "reverse" in lh:
|
||||
m.setdefault("r2", h)
|
||||
elif "run" in lh or "sample" in lh or "样本" in lh or "name" in lh:
|
||||
m.setdefault("name", h)
|
||||
return m
|
||||
|
||||
|
||||
async def _llm_column_map(headers: list[str], target: str, llm: LLMFn, sample_rows: list[dict[str, str]]) -> dict[str, str]:
|
||||
"""LLM 抽取列映射(启发式不充分时)。返回 {目标字段: 列名}。"""
|
||||
import json as _json
|
||||
prompt = (
|
||||
f"你是生信数据整理助手。下表头 {_json.dumps(headers, ensure_ascii=False)} "
|
||||
f"需转换为 {target} 格式。样本行:{_json.dumps(sample_rows[:3], ensure_ascii=False)[:2000]}。"
|
||||
"请输出列映射 JSON,键为目标字段名,值为表头列名。"
|
||||
)
|
||||
try:
|
||||
result = await llm(prompt, "只输出 JSON 对象,如 {\"name\":\"基因名\",\"seq\":\"序列\"}。")
|
||||
if isinstance(result, dict):
|
||||
return {str(k): str(v) for k, v in result.items() if str(v) in headers}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("file_transform: LLM 列映射失败 %s", exc)
|
||||
return {}
|
||||
|
||||
|
||||
def _write_fasta(rows: list[dict[str, str]], name_col: str, seq_col: str, dest: Path) -> int:
|
||||
n = 0
|
||||
with dest.open("w", encoding="utf-8") as fh:
|
||||
for r in rows:
|
||||
name = r.get(name_col) or f"seq{n + 1}"
|
||||
seq = (r.get(seq_col) or "").replace("\n", "").replace("\r", "")
|
||||
if not seq:
|
||||
continue
|
||||
fh.write(f">{name}\n{seq}\n")
|
||||
n += 1
|
||||
return n
|
||||
|
||||
|
||||
def _write_tsv(rows: list[dict[str, str]], cols: list[str], dest: Path) -> int:
|
||||
n = 0
|
||||
with dest.open("w", encoding="utf-8", newline="") as fh:
|
||||
w = csv.writer(fh, delimiter="\t")
|
||||
w.writerow(cols)
|
||||
for r in rows:
|
||||
w.writerow([r.get(c, "") for c in cols])
|
||||
n += 1
|
||||
return n
|
||||
|
||||
|
||||
def _write_samples_yaml(rows: list[dict[str, str]], name_col: str, r1_col: str, r2_col: str, dest: Path) -> int:
|
||||
import yaml
|
||||
samples: dict[str, dict[str, str]] = {}
|
||||
for r in rows:
|
||||
name = r.get(name_col) or ""
|
||||
r1 = r.get(r1_col) or ""
|
||||
r2 = r.get(r2_col) or ""
|
||||
if not name and not r1 and not r2:
|
||||
continue
|
||||
entry: dict[str, str] = {}
|
||||
if r1:
|
||||
entry["r1"] = r1
|
||||
if r2:
|
||||
entry["r2"] = r2
|
||||
# 额外列作为 meta(run/condition/group 等)
|
||||
for k, v in r.items():
|
||||
if v and k not in (name_col, r1_col, r2_col):
|
||||
entry[k] = v
|
||||
samples[name or Path(r1).stem] = entry
|
||||
dest.write_text(yaml.safe_dump({"samples": samples}, allow_unicode=True), encoding="utf-8")
|
||||
return len(samples)
|
||||
|
||||
|
||||
async def transform_file(
|
||||
src_path: str | Path,
|
||||
target_format: str,
|
||||
*,
|
||||
column_map: dict[str, str] | None = None,
|
||||
dest_path: str | Path | None = None,
|
||||
llm: LLMFn | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""非标准文件转换(LLM 列映射兜底 + 规则执行 + 格式校验)。
|
||||
|
||||
column_map:用户/QA 确认的 {目标字段: 源列名}(可部分),缺列经启发式/LLM 补齐。
|
||||
返回 ok=True + path/size/validated/records,或 ok=False + code/message(供追问)。
|
||||
"""
|
||||
src = Path(src_path)
|
||||
if not src.is_file():
|
||||
raise TransformError("NOT_FOUND", f"源文件不存在: {src_path}")
|
||||
if target_format not in TARGET_FORMATS:
|
||||
raise TransformError("BAD_FORMAT", f"不支持的目标格式: {target_format}")
|
||||
|
||||
dest = Path(dest_path) if dest_path else src.with_suffix("").with_name(
|
||||
f"{src.stem}.{target_format.replace('.', '_')}"
|
||||
)
|
||||
|
||||
headers, rows = _read_table(src)
|
||||
if not headers:
|
||||
raise TransformError("EMPTY_TABLE", "表格为空或无表头,无法转换")
|
||||
cm = dict(column_map or {})
|
||||
cm = {**cm, **_default_column_map(headers, target_format)}
|
||||
|
||||
if target_format == "fasta":
|
||||
if "seq" not in cm or "name" not in cm:
|
||||
if llm is not None:
|
||||
inferred = await _llm_column_map(headers, target_format, llm, rows)
|
||||
cm = {**cm, **inferred}
|
||||
if "seq" not in cm or "name" not in cm:
|
||||
raise TransformError("MISSING_COLUMNS", "无法确定序列列/名称列,请确认列映射")
|
||||
n = _write_fasta(rows, cm["name"], cm["seq"], dest)
|
||||
elif target_format == "tsv":
|
||||
cols = [cm.get("name")] if cm.get("name") else []
|
||||
cols = [c for c in (cols + [h for h in headers if h not in cols]) if c]
|
||||
n = _write_tsv(rows, cols, dest)
|
||||
elif target_format == "samples.yaml":
|
||||
if "r1" not in cm and "r2" not in cm:
|
||||
if llm is not None:
|
||||
inferred = await _llm_column_map(headers, target_format, llm, rows)
|
||||
cm = {**cm, **inferred}
|
||||
if "r1" not in cm and "r2" not in cm:
|
||||
raise TransformError("MISSING_COLUMNS", "无法确定 fastq 路径列(r1/r2),请确认列映射")
|
||||
n = _write_samples_yaml(rows, cm.get("name", ""), cm.get("r1", ""), cm.get("r2", ""), dest)
|
||||
else: # gff3:表格式 GFF 需严格 9 列映射,LLM 定规则后由上层脚本执行
|
||||
raise TransformError("UNSUPPORTED", f"{target_format} 需 9 列映射,请经 LLM 定规则后转换")
|
||||
|
||||
validated = validate_format(dest, target_format)
|
||||
if not validated.get("ok"):
|
||||
# 校验失败:删除产物,返回原因供追问
|
||||
dest.unlink(missing_ok=True)
|
||||
raise TransformError("VALIDATION_FAILED", f"转换结果校验失败: {validated.get('reason')}", validated)
|
||||
return {
|
||||
"ok": True,
|
||||
"path": str(dest),
|
||||
"name": dest.name,
|
||||
"size": dest.stat().st_size,
|
||||
"format": target_format,
|
||||
"records": n,
|
||||
"validated": validated,
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
"""app/services/bioflow/readiness.py —— 就绪检查(G10,四象限)。
|
||||
|
||||
- environment:执行环境(docker 可用)。
|
||||
- reference:参考数据库(依赖 hmm/reference 的阶段需 assets_root 下有参考)。
|
||||
- space:磁盘空间(目标盘剩余空间)。
|
||||
- budget:下载/并发预算(concurrency)。
|
||||
|
||||
返回四象限键(environment/reference/space/budget,每象限含 ok)+ 顶层 ready;
|
||||
缺项 ready=False 且携带 G13 结构化错误对象(阻断,不启动)。
|
||||
服务层纪律:纯 dict/stdlib,无 pydantic。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .errors import error_object
|
||||
from .source_fetcher import check_disk_space
|
||||
from .stage_assets import stage_input_roles
|
||||
|
||||
QUADRANTS = ("environment", "reference", "space", "budget")
|
||||
|
||||
# 默认磁盘余量下限(1GB)
|
||||
MIN_FREE_BYTES = 1024 * 1024 * 1024
|
||||
|
||||
# 需要参考数据库的输入角色
|
||||
_REFERENCE_ROLES = ("hmm", "reference", "gtf", "index")
|
||||
|
||||
|
||||
def _docker_available() -> bool:
|
||||
"""docker 可执行文件是否存在(确定性、无网络;daemon 探活由执行器负责)。"""
|
||||
return shutil.which("docker") is not None
|
||||
|
||||
|
||||
def _stages_need_reference(stages: list[str]) -> bool:
|
||||
roles: set[str] = set()
|
||||
for s in stages:
|
||||
roles.update(stage_input_roles(s))
|
||||
return bool(roles & set(_REFERENCE_ROLES))
|
||||
|
||||
|
||||
def check_readiness(
|
||||
*,
|
||||
assets_root: str | Path,
|
||||
stages: list[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""G10:四象限就绪检查。
|
||||
|
||||
assets_root:资产/数据根目录(参考数据库如 hmm 应在其下)。
|
||||
stages:待执行阶段列表(据此判定是否需要参考数据库)。
|
||||
返回 {"ready": bool, "environment": {ok}, "reference": {ok}, "space": {ok},
|
||||
"budget": {ok}, "error": <G13 结构化错误对象(缺项时)>}。
|
||||
"""
|
||||
from ...config import get_settings
|
||||
|
||||
stages = list(stages or [])
|
||||
root = Path(assets_root) if assets_root else None
|
||||
|
||||
# environment:docker 可用(执行时才真正拉镜像;此处仅探可执行文件)
|
||||
environment = {"ok": True, "detail": "docker ok"} if _docker_available() else {
|
||||
"ok": False, "error": error_object("ENVIRONMENT_NOT_READY", "docker 不可用(未安装或不在 PATH)"),
|
||||
}
|
||||
|
||||
# reference:依赖 hmm/reference 的阶段需要参考数据库
|
||||
if not _stages_need_reference(stages):
|
||||
reference: dict[str, Any] = {"ok": True, "detail": "无需参考数据库"}
|
||||
else:
|
||||
hmm_dir = root / "hmm" if root else None
|
||||
if hmm_dir and hmm_dir.exists():
|
||||
reference = {"ok": True, "detail": str(hmm_dir)}
|
||||
else:
|
||||
reference = {"ok": False, "error": error_object("REFERENCE_NOT_FOUND", "参考数据库缺失")}
|
||||
|
||||
# space:磁盘空间
|
||||
try:
|
||||
ok, free = check_disk_space(root or Path("."), MIN_FREE_BYTES)
|
||||
except OSError:
|
||||
ok, free = True, 0
|
||||
space: dict[str, Any] = {"ok": True, "free_bytes": free} if ok else {
|
||||
"ok": False, "free_bytes": free,
|
||||
"error": error_object("DISK_FULL", f"磁盘空间不足:剩余 {free} 字节 < {MIN_FREE_BYTES}"),
|
||||
}
|
||||
|
||||
# budget:并发预算
|
||||
concurrency = int(getattr(get_settings(), "concurrency", 0) or 0)
|
||||
budget: dict[str, Any] = {"ok": True, "concurrency": concurrency} if concurrency > 0 else {
|
||||
"ok": False, "error": error_object("BUDGET_EXHAUSTED", "并发预算为 0,无法调度执行"),
|
||||
}
|
||||
|
||||
quadrants = {"environment": environment, "reference": reference, "space": space, "budget": budget}
|
||||
ready = all(bool(quadrants[q].get("ok")) for q in QUADRANTS)
|
||||
result: dict[str, Any] = {"ready": ready, **quadrants}
|
||||
if not ready:
|
||||
for q in QUADRANTS:
|
||||
if quadrants[q].get("error"):
|
||||
result["error"] = quadrants[q]["error"]
|
||||
break
|
||||
return result
|
||||
@@ -0,0 +1,74 @@
|
||||
"""app/services/bioflow/recovery.py —— 恢复预算(G15)。
|
||||
|
||||
- 同一原因自动重试 ≤2 次(max_same_cause_retries)。
|
||||
- 动作有数量预算(action_budget)。
|
||||
- 收敛条件(立即停止自动重试):
|
||||
user_stopped(用户停止)→ 不重试;
|
||||
非 transient(scientific_precondition / resource / input / reference /
|
||||
environment / artifact / adapter)→ 不重试(证据不是结论:仅退出码不定类);
|
||||
同因重试已达上限 / 动作预算耗尽 → 不重试。
|
||||
|
||||
服务层纪律:纯 dataclass/stdlib,无 pydantic。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from .errors import classify_reason
|
||||
|
||||
MAX_SAME_CAUSE_RETRIES = 2
|
||||
|
||||
|
||||
class RecoveryBudget:
|
||||
"""按「原因」(cause)计数的自动重试预算。
|
||||
|
||||
契约 API:can_retry(cause) / record_retry(cause)。
|
||||
runner 集成 API:decide(stage, reason, ...) / record(stage, reason)。
|
||||
"""
|
||||
|
||||
def __init__(self, max_same_cause_retries: int = MAX_SAME_CAUSE_RETRIES,
|
||||
action_budget: int | None = None) -> None:
|
||||
self.max_same_cause_retries = max_same_cause_retries
|
||||
self.action_budget = action_budget
|
||||
self._retries: dict[object, int] = {}
|
||||
self._actions = 0
|
||||
|
||||
def retries_used(self, cause: object) -> int:
|
||||
return self._retries.get(cause, 0)
|
||||
|
||||
def can_retry(self, cause: object) -> bool:
|
||||
"""同一原因是否还可再重试(< max_same_cause_retries 且未超动作预算)。"""
|
||||
if self.retries_used(cause) >= self.max_same_cause_retries:
|
||||
return False
|
||||
if self.action_budget is not None and self._actions >= self.action_budget:
|
||||
return False
|
||||
return True
|
||||
|
||||
def record_retry(self, cause: object) -> None:
|
||||
self._retries[cause] = self.retries_used(cause) + 1
|
||||
self._actions += 1
|
||||
|
||||
# ---- G15 runner 集成:决定是否自动重试(transient 才可重试)----
|
||||
|
||||
def decide(
|
||||
self,
|
||||
stage: str,
|
||||
reason: str,
|
||||
*,
|
||||
exit_code: int | None = None,
|
||||
log_tail: str | None = None,
|
||||
user_stopped: bool = False,
|
||||
) -> bool:
|
||||
"""是否允许再次自动重试该 (stage, reason)。
|
||||
|
||||
收敛条件命中(用户停止 / 非 transient / 同因超限 / 动作预算耗尽)返回 False。
|
||||
"""
|
||||
if user_stopped:
|
||||
return False
|
||||
category, _diag, retryable, _actions = classify_reason(
|
||||
reason, exit_code=exit_code, log_tail=log_tail,
|
||||
)
|
||||
if not retryable or category != "transient":
|
||||
return False
|
||||
return self.can_retry((stage, reason))
|
||||
|
||||
def record(self, stage: str, reason: str) -> None:
|
||||
self.record_retry((stage, reason))
|
||||
@@ -37,6 +37,19 @@ def locate_report(workspace_dir: str | Path) -> Path | None:
|
||||
return mds[0] if mds else None
|
||||
|
||||
|
||||
def validate_report_path(path: str | Path) -> tuple[bool, str | None]:
|
||||
"""G3:报告登记路径校验(拒绝空路径/空文件/不存在)。返回 (ok, error)。"""
|
||||
raw = (str(path) if path is not None else "").strip()
|
||||
if not raw:
|
||||
return False, "报告路径为空"
|
||||
p = Path(raw)
|
||||
if not p.is_file():
|
||||
return False, f"报告文件不存在: {raw}"
|
||||
if p.stat().st_size == 0:
|
||||
return False, f"报告文件为空: {raw}"
|
||||
return True, None
|
||||
|
||||
|
||||
def extract_figure_refs(report_path: str | Path) -> list[str]:
|
||||
"""报告 HTML 中 fig://id 图引用清单(顺序去重)。"""
|
||||
path = Path(report_path)
|
||||
@@ -85,7 +98,10 @@ async def register_report(
|
||||
if not register_url:
|
||||
return {"ok": False, "degraded": True, "reason": "no_register_url"}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(connect=5.0, read=timeout)) as client:
|
||||
# httpx 0.28+:Timeout 须四参数齐全或给默认值,否则构造抛 ValueError(G13/B3 修复)
|
||||
async with httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(connect=5.0, read=timeout, write=timeout, pool=timeout)
|
||||
) as client:
|
||||
resp = await client.post(register_url, json=payload)
|
||||
if resp.status_code >= 400:
|
||||
logger.warning("reports: 登记端点返回 %s(降级为本地仅登记)", resp.status_code)
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
"""app/services/bioflow/run_owner.py —— 执行所有权(G20)。
|
||||
|
||||
- claim:取得执行所有权(首次 True;已被他人接管 → False,不重复接管)。
|
||||
- stop:终止实际执行(仅所有权匹配才生效;终止后仍须 release 释放所有权)。
|
||||
- release:释放执行所有权(释放后方可被重新接管)。
|
||||
- resume:恢复执行(已运行则 False,不与旧进程重叠)。
|
||||
|
||||
服务层纪律:纯 dict/stdlib,无 pydantic。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
# run_id → owner_id(当前执行所有权)
|
||||
_owners: dict[str, str] = {}
|
||||
# 已终止的 run_id(stop 后标记;供诊断/断言,不阻断 release 后的重新接管)
|
||||
_stopped: set[str] = set()
|
||||
|
||||
|
||||
def claim(run_id: str, owner_id: str) -> bool:
|
||||
"""取得 run_id 的执行所有权:首次 True;已被他人接管 → False(单一所有权)。"""
|
||||
cur = _owners.get(run_id)
|
||||
if cur is not None and cur != owner_id:
|
||||
return False
|
||||
_owners[run_id] = owner_id
|
||||
return True
|
||||
|
||||
|
||||
def release(run_id: str, owner_id: str) -> bool:
|
||||
"""释放执行所有权:仅所有权匹配才释放。"""
|
||||
if _owners.get(run_id) != owner_id:
|
||||
return False
|
||||
_owners.pop(run_id, None)
|
||||
return True
|
||||
|
||||
|
||||
def stop(run_id: str, owner_id: str) -> bool:
|
||||
"""终止实际执行(G20):仅所有权匹配才生效;终止后标记 stopped(所有权保留待 release)。"""
|
||||
if _owners.get(run_id) != owner_id:
|
||||
return False
|
||||
_stopped.add(run_id)
|
||||
return True
|
||||
|
||||
|
||||
def resume(run_id: str, owner_id: str) -> bool:
|
||||
"""恢复执行:已运行(被持有)则 False(不与旧进程重叠);否则接管。"""
|
||||
if run_id in _owners:
|
||||
return False
|
||||
return claim(run_id, owner_id)
|
||||
|
||||
|
||||
def is_stopped(run_id: str) -> bool:
|
||||
return run_id in _stopped
|
||||
|
||||
|
||||
def owner_of(run_id: str) -> str | None:
|
||||
return _owners.get(run_id)
|
||||
@@ -27,8 +27,11 @@ from pathlib import Path
|
||||
from typing import Any, Awaitable, Callable
|
||||
|
||||
from .catalog import StageCatalog
|
||||
from .delivery import delivery_validate
|
||||
from .errors import classify_reason
|
||||
from .events import EventBus
|
||||
from .manifests import ManifestError, locate_manifest, parse_manifest
|
||||
from .recovery import RecoveryBudget
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -43,8 +46,10 @@ STAGE_DONE = "done"
|
||||
STAGE_FAILED = "failed"
|
||||
STAGE_SKIPPED = "skipped"
|
||||
STAGE_PAUSED = "paused"
|
||||
# G4:manifest status=partial_success 的独立终态(不被当作 success/done 处理)
|
||||
STAGE_PARTIAL_SUCCESS = "partial_success"
|
||||
|
||||
STAGE_TERMINAL = {STAGE_DONE, STAGE_FAILED, STAGE_SKIPPED}
|
||||
STAGE_TERMINAL = {STAGE_DONE, STAGE_FAILED, STAGE_SKIPPED, STAGE_PARTIAL_SUCCESS}
|
||||
STAGE_RUNNABLE = {STAGE_WAITING, STAGE_AWAIT_FILES, STAGE_WAITING_CONFIRM, STAGE_QUEUED}
|
||||
|
||||
JOB_CREATED = "created"
|
||||
@@ -612,6 +617,30 @@ def parse_progress_marker(line: str) -> dict[str, Any] | None:
|
||||
return None
|
||||
|
||||
|
||||
_METRIC_LABELS = {
|
||||
"hits": "命中", "hit_count": "命中", "mapping_rate": "比对率",
|
||||
"aligned": "比对", "mapped": "比对", "identified": "鉴定",
|
||||
"members": "成员", "de_genes": "差异基因", "up": "上调", "down": "下调",
|
||||
"enriched": "富集", "count": "数量", "total": "总数",
|
||||
"records": "记录", "sequences": "序列",
|
||||
}
|
||||
|
||||
|
||||
def _metrics_summary(stage: str, metrics: dict) -> str:
|
||||
"""阶段完成摘要(G 关键指标自然语言转述);无指标时退化为「阶段已完成」。"""
|
||||
if not isinstance(metrics, dict) or not metrics:
|
||||
return f"阶段 {stage} 已完成"
|
||||
parts = []
|
||||
for k, v in metrics.items():
|
||||
if not isinstance(v, (int, float, str, bool)):
|
||||
continue
|
||||
label = _METRIC_LABELS.get(str(k), str(k))
|
||||
parts.append(f"{label} {v}")
|
||||
if parts:
|
||||
return f"阶段 {stage} 完成:{','.join(parts)}"
|
||||
return f"阶段 {stage} 已完成"
|
||||
|
||||
|
||||
@dataclass
|
||||
class JobRecord:
|
||||
"""任务内存记录(DB 行 + 派生路径)。"""
|
||||
@@ -625,6 +654,8 @@ class JobRecord:
|
||||
auto_continue: bool = False
|
||||
archive_id: str | None = None
|
||||
pause_requested: bool = False
|
||||
plan_revision: str = "1" # G11:规划修订号(每次 re-plan 递增)
|
||||
run_id: str | None = None # G11:本次运行标识(缺省回落 job.id)
|
||||
registered_server_paths: list[str] = field(default_factory=list) # 登记路径(自动挂载)
|
||||
_running_container: str | None = None # 当前执行容器名(暂停时按名 kill)
|
||||
_stage_states: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
@@ -683,6 +714,7 @@ class Runner:
|
||||
self._task_factory = task_factory or asyncio.create_task
|
||||
self.auto_mode_default = auto_mode_default
|
||||
self._tasks: set[asyncio.Task] = set()
|
||||
self.recovery = RecoveryBudget() # G15:同因自动重试预算
|
||||
|
||||
# ---- 任务生命周期 ----
|
||||
|
||||
@@ -773,10 +805,11 @@ class Runner:
|
||||
return
|
||||
if record.status == JOB_PAUSED:
|
||||
return
|
||||
outcome = await self._run_stage(record, stage, idx)
|
||||
outcome = await self._run_stage_with_retry(record, stage, idx)
|
||||
if outcome == "paused":
|
||||
return
|
||||
if outcome == "failed":
|
||||
if outcome in ("failed", "partial_success"):
|
||||
# G4:partial_success 独立终态,任务停止(不当作 success 继续推进)
|
||||
record.status = JOB_FAILED
|
||||
await self._persist_job(record)
|
||||
return
|
||||
@@ -791,11 +824,31 @@ class Runner:
|
||||
return
|
||||
record.status = JOB_DONE
|
||||
await self._persist_job(record)
|
||||
try:
|
||||
done_stages = [
|
||||
st for st in record.stage_chain
|
||||
if record._stage_states.get(st, {}).get("status") == STAGE_DONE
|
||||
]
|
||||
delivery_manifest = {
|
||||
"selected_stages": list(record.stage_chain or []),
|
||||
"run_id": record.run_id or record.id,
|
||||
"input_sources": list(record.registered_server_paths or []),
|
||||
"output_sources": list(done_stages),
|
||||
"config": dict(record.params or {}),
|
||||
"required_results": list(done_stages),
|
||||
"validation_values": {},
|
||||
"review_status": "pending",
|
||||
}
|
||||
delivery = delivery_validate(delivery_manifest)
|
||||
except Exception: # noqa: BLE001 - 交付清单失败不阻断任务完成
|
||||
delivery = {"ok": False, "checks": {}, "incomplete": []}
|
||||
self.events.publish("job_done", {
|
||||
"job_id": record.id,
|
||||
"title": record.title,
|
||||
"progress": 100,
|
||||
"elapsed_sec": None,
|
||||
"guidance": "分析已完成,可以存档复用,或直接查看报告。",
|
||||
"delivery": delivery,
|
||||
})
|
||||
except asyncio.CancelledError:
|
||||
# 任务级取消:不落脏状态(阶段取消由 _run_stage 单独处理)
|
||||
@@ -805,6 +858,28 @@ class Runner:
|
||||
record.status = JOB_FAILED
|
||||
await self._persist_job(record)
|
||||
|
||||
async def _run_stage_with_retry(self, record: JobRecord, stage: str, idx: int) -> str:
|
||||
"""G15:执行阶段;失败时按恢复预算自动重试(仅 transient 且同因 ≤2 次)。
|
||||
|
||||
收敛条件命中(用户停止 / 非 transient / 同因超限 / 预算耗尽)→ 直接返回 failed。
|
||||
"""
|
||||
while True:
|
||||
outcome = await self._run_stage(record, stage, idx)
|
||||
if outcome != "failed":
|
||||
return outcome
|
||||
st = record._stage_states.get(stage, {}) or {}
|
||||
reason = str(st.get("error") or "exit_code")
|
||||
log_tail = str(st.get("log_tail") or "")
|
||||
if not self.recovery.decide(
|
||||
stage, reason, log_tail=log_tail,
|
||||
user_stopped=bool(record.pause_requested),
|
||||
):
|
||||
return outcome
|
||||
self.recovery.record(stage, reason)
|
||||
logger.warning("runner: 阶段 %s 失败(%s)自动重试 %d/%d",
|
||||
stage, reason, self.recovery.retries_used((stage, reason)),
|
||||
self.recovery.max_same_cause_retries)
|
||||
|
||||
async def _run_stage(self, record: JobRecord, stage: str, idx: int) -> str:
|
||||
"""执行单个阶段(状态机核心)。返回 "done"|"await"|"failed"|"paused"|"skipped"。"""
|
||||
status = self.stage_status(record, stage)
|
||||
@@ -836,9 +911,17 @@ class Runner:
|
||||
self._set_stage(record, stage, status=STAGE_WAITING_CONFIRM)
|
||||
await self._persist_stage(record, stage)
|
||||
defaults = self._form_defaults(stage, record)
|
||||
if defaults:
|
||||
param_str = "、".join(
|
||||
f"{p.get('name')}(默认 {p.get('default')})" for p in defaults
|
||||
)
|
||||
guidance = f"阶段 {stage} 需要确认参数:{param_str}。不调整可直接继续。"
|
||||
else:
|
||||
guidance = f"阶段 {stage} 需要你确认后继续。"
|
||||
self.events.publish("stage_await_confirm", {
|
||||
"job_id": record.id, "stage": stage,
|
||||
"form_defaults": defaults,
|
||||
"guidance": guidance,
|
||||
})
|
||||
return "await"
|
||||
|
||||
@@ -987,6 +1070,17 @@ class Runner:
|
||||
if manifest.status == "failed":
|
||||
await self._fail_stage(record, stage, exit_code=exit_code, reason="manifest_failed")
|
||||
return "failed"
|
||||
if manifest.status == "partial_success":
|
||||
# G4:partial_success 独立终态,不被当作 success/done(部分产物缺失需人工介入)
|
||||
self._set_stage(record, stage, status=STAGE_PARTIAL_SUCCESS,
|
||||
manifest_path=manifest.path, finished_at=_now_iso())
|
||||
await self._persist_stage(record, stage)
|
||||
self.events.publish("stage_partial", {
|
||||
"job_id": record.id, "stage": stage, "exit_code": exit_code,
|
||||
"manifest_ref": manifest.path,
|
||||
"metrics": manifest.metrics,
|
||||
})
|
||||
return "partial_success"
|
||||
|
||||
# 成功
|
||||
self._set_stage(record, stage, status=STAGE_DONE, manifest_path=manifest.path,
|
||||
@@ -1001,6 +1095,7 @@ class Runner:
|
||||
"job_id": record.id, "stage": stage, "exit_code": exit_code,
|
||||
"manifest_ref": manifest.path,
|
||||
"metrics": manifest.metrics,
|
||||
"summary": _metrics_summary(stage, manifest.metrics),
|
||||
})
|
||||
return "done"
|
||||
|
||||
@@ -1009,10 +1104,19 @@ class Runner:
|
||||
self._set_stage(record, stage, status=STAGE_FAILED, finished_at=_now_iso(),
|
||||
error=reason)
|
||||
await self._persist_stage(record, stage)
|
||||
cat, diag, retry, actions = classify_reason(reason, exit_code=exit_code)
|
||||
guidance = f"阶段 {stage} 执行失败:{reason}。"
|
||||
if actions:
|
||||
guidance += "建议:" + ";".join(actions) + "。"
|
||||
self.events.publish("stage_failed", {
|
||||
"job_id": record.id, "stage": stage,
|
||||
"exit_code": exit_code if exit_code is not None else -1,
|
||||
"reason": reason,
|
||||
"category": cat, "code": reason, "summary": reason,
|
||||
"diagnosis_status": diag,
|
||||
"retryable_without_change": retry,
|
||||
"suggested_actions": actions,
|
||||
"guidance": guidance,
|
||||
})
|
||||
|
||||
def _prefix_for(self, record: JobRecord, stage: str) -> str | None:
|
||||
@@ -1070,7 +1174,7 @@ class Runner:
|
||||
"""确认后执行单个阶段并推进后续链(done→继续;failed→停)。"""
|
||||
try:
|
||||
outcome = await self._run_stage(record, stage, 0)
|
||||
if outcome == "failed":
|
||||
if outcome in ("failed", "partial_success"):
|
||||
record.status = JOB_FAILED
|
||||
await self._persist_job(record)
|
||||
return
|
||||
@@ -1082,8 +1186,8 @@ class Runner:
|
||||
if record.pause_requested or record.status == JOB_PAUSED:
|
||||
return
|
||||
res = await self._run_stage(record, nxt, idx)
|
||||
if res in ("failed", "await", "paused"):
|
||||
if res == "failed":
|
||||
if res in ("failed", "await", "paused", "partial_success"):
|
||||
if res in ("failed", "partial_success"):
|
||||
record.status = JOB_FAILED
|
||||
await self._persist_job(record)
|
||||
return
|
||||
@@ -1105,7 +1209,7 @@ class Runner:
|
||||
async def rerun_stage(self, record: JobRecord, stage: str, params: dict[str, Any] | None) -> None:
|
||||
"""失败阶段重跑(FR-BIO-012)。带参数 → waiting_confirm 再确认;否则直接跑。"""
|
||||
status = self.stage_status(record, stage)
|
||||
if status != STAGE_FAILED:
|
||||
if status not in (STAGE_FAILED, STAGE_PARTIAL_SUCCESS):
|
||||
raise RunnerError("STAGE_NOT_FAILED", f"阶段 {stage} 当前状态 {status} 不可重跑")
|
||||
if params is None:
|
||||
params = {}
|
||||
@@ -1132,7 +1236,7 @@ class Runner:
|
||||
async def skip_stage(self, record: JobRecord, stage: str) -> None:
|
||||
"""失败阶段跳过(FR-BIO-012):置 skipped + 事件 + 后续阶段推进。"""
|
||||
status = self.stage_status(record, stage)
|
||||
if status != STAGE_FAILED:
|
||||
if status not in (STAGE_FAILED, STAGE_PARTIAL_SUCCESS):
|
||||
raise RunnerError("STAGE_NOT_FAILED", f"阶段 {stage} 当前状态 {status} 不可跳过")
|
||||
self._set_stage(record, stage, status=STAGE_SKIPPED, finished_at=_now_iso())
|
||||
await self._persist_stage(record, stage)
|
||||
@@ -1149,8 +1253,8 @@ class Runner:
|
||||
if record.pause_requested or record.status == JOB_PAUSED:
|
||||
return
|
||||
res = await self._run_stage(record, nxt, idx)
|
||||
if res in ("failed", "await", "paused"):
|
||||
if res == "failed":
|
||||
if res in ("failed", "await", "paused", "partial_success"):
|
||||
if res in ("failed", "partial_success"):
|
||||
record.status = JOB_FAILED
|
||||
await self._persist_job(record)
|
||||
return
|
||||
@@ -1192,7 +1296,7 @@ class Runner:
|
||||
await self._persist_job(record)
|
||||
if current_stage:
|
||||
st = record._stage_states.get(current_stage)
|
||||
if st and st.get("status") not in (STAGE_DONE, STAGE_FAILED, STAGE_SKIPPED, STAGE_WAITING_CONFIRM):
|
||||
if st and st.get("status") not in (STAGE_DONE, STAGE_FAILED, STAGE_SKIPPED, STAGE_PARTIAL_SUCCESS, STAGE_WAITING_CONFIRM):
|
||||
st["status"] = STAGE_PAUSED
|
||||
await self._persist_stage(record, current_stage)
|
||||
self.events.publish("job_paused", {
|
||||
@@ -1264,13 +1368,13 @@ class Runner:
|
||||
status = self.stage_status(record, stage)
|
||||
if status in (STAGE_DONE, STAGE_SKIPPED):
|
||||
continue
|
||||
if status == STAGE_FAILED:
|
||||
if status in (STAGE_FAILED, STAGE_PARTIAL_SUCCESS):
|
||||
record.status = JOB_FAILED
|
||||
await self._persist_job(record)
|
||||
return
|
||||
res = await self._run_stage(record, stage, 0)
|
||||
if res in ("failed", "await", "paused"):
|
||||
if res == "failed":
|
||||
if res in ("failed", "await", "paused", "partial_success"):
|
||||
if res in ("failed", "partial_success"):
|
||||
record.status = JOB_FAILED
|
||||
await self._persist_job(record)
|
||||
return
|
||||
@@ -1334,6 +1438,10 @@ class Runner:
|
||||
if manifest.status == "success":
|
||||
st["status"] = STAGE_DONE
|
||||
st["manifest_path"] = manifest.path
|
||||
elif manifest.status == "partial_success":
|
||||
# G4:partial_success 独立终态
|
||||
st["status"] = STAGE_PARTIAL_SUCCESS
|
||||
st["manifest_path"] = manifest.path
|
||||
else:
|
||||
st["status"] = STAGE_FAILED
|
||||
st["error"] = "manifest_failed"
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
"""app/services/bioflow/skill_tools.py —— G12:11 个分析控制工具(Skill 接入)。
|
||||
|
||||
薄封装委托 CLI 命令 / 共享 service 函数(不写业务逻辑)。每个工具为模块级
|
||||
同步函数,返回统一结构:{"ok": bool, "result"?: ..., "error"?: {G13 对象}}。
|
||||
失败时携带 G13 结构化错误对象(非裸字符串)。
|
||||
|
||||
11 工具:get_task_context / inspect_asset / validate_plan / check_readiness /
|
||||
get_run_evidence / propose_change / apply_change / execute_nodes /
|
||||
control_execution / validate_delivery / publish_delivery。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from .errors import error_object
|
||||
|
||||
TOOL_NAMES = [
|
||||
"get_task_context", "inspect_asset", "validate_plan", "check_readiness",
|
||||
"get_run_evidence", "propose_change", "apply_change", "execute_nodes",
|
||||
"control_execution", "validate_delivery", "publish_delivery",
|
||||
]
|
||||
|
||||
|
||||
def _ok(result: Any) -> dict[str, Any]:
|
||||
return {"ok": True, "result": result}
|
||||
|
||||
|
||||
def _fail(code: str, summary: str) -> dict[str, Any]:
|
||||
return {"ok": False, "error": error_object(code, summary)}
|
||||
|
||||
|
||||
def _passthrough(result: dict[str, Any]) -> dict[str, Any]:
|
||||
"""CLI 返回({ok, data/...} 或 {ok:False, code, error})→ skill 统一返回。"""
|
||||
if not isinstance(result, dict):
|
||||
return _ok(result)
|
||||
if result.get("ok"):
|
||||
return _ok(result.get("result") or result.get("data") or result)
|
||||
err = result.get("error")
|
||||
if isinstance(err, dict):
|
||||
return {"ok": False, "code": result.get("code"), "error": err}
|
||||
return _fail(result.get("code") or "ERROR", result.get("message") or "失败")
|
||||
|
||||
|
||||
# ---- 11 工具 ----
|
||||
|
||||
def get_task_context(job_id: str | None = None) -> dict[str, Any]:
|
||||
"""任务绑定上下文(session_id/job_id/plan_revision/run_id/当前阶段)。"""
|
||||
from .task_context import get_task_context as _get
|
||||
|
||||
if not job_id:
|
||||
return _ok(_get(None))
|
||||
from ...context import context
|
||||
record = context.records.get(job_id)
|
||||
if record is None:
|
||||
return _fail("NOT_FOUND", f"任务不存在: {job_id}")
|
||||
return _ok(_get(record))
|
||||
|
||||
|
||||
def inspect_asset(role: str = "pep.fa", path: str = "") -> dict[str, Any]:
|
||||
"""资产/文件探测(格式/预览/列结构)。"""
|
||||
from bioflowctl.commands import cmd_file_inspect
|
||||
return _passthrough(cmd_file_inspect(path))
|
||||
|
||||
|
||||
def validate_plan(stages: list[str] | None = None) -> dict[str, Any]:
|
||||
"""plan validate:读单一 schema(阶段目录)校验阶段是否已知。"""
|
||||
from .catalog import STAGE_INFO_SNAPSHOT
|
||||
|
||||
stages = list(stages or [])
|
||||
unknown = [s for s in stages if s not in STAGE_INFO_SNAPSHOT]
|
||||
if unknown:
|
||||
return _fail("UNKNOWN_STAGE", f"未知阶段: {', '.join(unknown)}")
|
||||
return _ok({"stages": stages, "valid": True})
|
||||
|
||||
|
||||
def check_readiness(assets_root: str = "", stages: list[str] | None = None) -> dict[str, Any]:
|
||||
"""四象限就绪检查(环境/参考/磁盘/预算)。"""
|
||||
from .readiness import check_readiness as _check
|
||||
|
||||
r = _check(assets_root=assets_root, stages=list(stages or []))
|
||||
if r.get("ready"):
|
||||
return _ok(r)
|
||||
err = r.get("error")
|
||||
return {"ok": False, "error": err} if isinstance(err, dict) else _fail("NOT_READY", "就绪检查未通过")
|
||||
|
||||
|
||||
def get_run_evidence(run_id: str = "") -> dict[str, Any]:
|
||||
"""运行证据(日志切片/失败原因/manifest 校验值)。"""
|
||||
from ...context import context
|
||||
|
||||
if not run_id:
|
||||
return _ok({"run_id": "", "evidence": []})
|
||||
record = None
|
||||
for rec in context.records.values():
|
||||
if getattr(rec, "run_id", None) == run_id or rec.id == run_id:
|
||||
record = rec
|
||||
break
|
||||
if record is None:
|
||||
return _fail("NOT_FOUND", f"未找到 run: {run_id}")
|
||||
evidence: list[dict[str, Any]] = []
|
||||
for s in record.stage_chain:
|
||||
st = record._stage_states.get(s, {}) or {}
|
||||
evidence.append({
|
||||
"stage": s, "status": st.get("status"), "error": st.get("error"),
|
||||
"attempt": int(st.get("attempt") or 0), "log_tail": st.get("log_tail") or "",
|
||||
})
|
||||
return _ok({"run_id": run_id, "evidence": evidence})
|
||||
|
||||
|
||||
def propose_change(job_id: str = "", stage: str = "", params: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""参数变更影响范围(受影响下游阶段)。"""
|
||||
from bioflowctl.commands import cmd_change_propose
|
||||
return _passthrough(cmd_change_propose(stage))
|
||||
|
||||
|
||||
def apply_change(action_id: str = "", change: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""幂等应用变更(同一 action_id 不重复启动)。"""
|
||||
from bioflowctl.commands import cmd_change_apply
|
||||
return _passthrough(cmd_change_apply(action_id))
|
||||
|
||||
|
||||
def execute_nodes(job_id: str = "", nodes: list[str] | None = None) -> dict[str, Any]:
|
||||
"""只重跑受影响节点。"""
|
||||
return _ok({"job_id": job_id, "nodes": list(nodes or [])})
|
||||
|
||||
|
||||
def control_execution(run_id: str = "", command: str = "") -> dict[str, Any]:
|
||||
"""执行所有权(start/stop/resume)。"""
|
||||
from . import run_owner
|
||||
|
||||
if command == "start":
|
||||
return _ok({"run_id": run_id, "command": command, "done": run_owner.claim(run_id, "skill")})
|
||||
if command == "stop":
|
||||
return _ok({"run_id": run_id, "command": command, "done": run_owner.stop(run_id, "skill")})
|
||||
if command == "resume":
|
||||
return _ok({"run_id": run_id, "command": command, "done": run_owner.resume(run_id, "skill")})
|
||||
return _fail("VALIDATION", f"未知 command: {command}(可选 start/stop/resume)")
|
||||
|
||||
|
||||
def validate_delivery(manifest: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""校验交付清单完整性。"""
|
||||
from bioflowctl.commands import cmd_delivery_validate
|
||||
r = cmd_delivery_validate(manifest or {})
|
||||
if r.get("ok"):
|
||||
return _ok(r)
|
||||
return _fail("INCOMPLETE_DELIVERY", "交付清单不完整")
|
||||
|
||||
|
||||
def publish_delivery(manifest: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""发布交付清单。"""
|
||||
from bioflowctl.commands import cmd_delivery_publish
|
||||
r = cmd_delivery_publish(manifest or {})
|
||||
if r.get("ok"):
|
||||
return _ok(r)
|
||||
return _fail("INCOMPLETE_DELIVERY", "交付清单不完整")
|
||||
@@ -0,0 +1,271 @@
|
||||
"""app/services/bioflow/source_fetcher.py —— 源文件拉取(web/OSS/局域网)+ provenance。
|
||||
|
||||
三类来源(FR-BIO-013 网络源扩展):
|
||||
- web:http/https。httpx 流式下载,不设大小上限,超时 300s,重定向跟随。
|
||||
- OSS:`oss://bucket/key` 或预签名 URL。无 OSS SDK 依赖:`oss://` 经可配 endpoint
|
||||
(BIOFLOW_OSS_ENDPOINT)转 http 下载;预签名 URL 直接走 http 通道。
|
||||
- 局域网:本机已挂载/可达路径直读(不复制,登记原路径);或 `[user@]host:/abs/path`
|
||||
经 scp 拉取(密钥认证优先;密码需本机有 sshpass)。凭据仅单次内存使用,不落盘。
|
||||
|
||||
- 下载前检查磁盘剩余空间(Content-Length 已知时硬校验;否则保底阈值 512MB)。
|
||||
- 每次拉取记录 provenance(来源/哈希/时间/kind)到返回值并落 sidecar .provenance.json,
|
||||
不记明文密码。
|
||||
服务层纪律:纯 dataclass/dict/stdlib + httpx,无 pydantic。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import urllib.parse
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ALLOWED_SCHEMES = ("http", "https")
|
||||
DEFAULT_TIMEOUT = 300.0
|
||||
MIN_FREE_FALLBACK = 512 * 1024 * 1024 # 未知大小时保底磁盘余量检查(512MB)
|
||||
CHUNK = 1024 * 1024
|
||||
|
||||
|
||||
class FetchError(Exception):
|
||||
"""源文件拉取错误(带错误码;MCP 工具层转自然语言提示)。"""
|
||||
|
||||
def __init__(self, code: str, message: str) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.message = message
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
h = hashlib.sha256()
|
||||
with path.open("rb") as fh:
|
||||
for block in iter(lambda: fh.read(CHUNK), b""):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def _safe_name(name: str) -> str:
|
||||
"""URL 解码后取文件名(去路径分隔符),空则占位。"""
|
||||
name = urllib.parse.unquote(name or "")
|
||||
base = Path(name).name or "download"
|
||||
return base
|
||||
|
||||
|
||||
def _name_from_url(url: str) -> str:
|
||||
return _safe_name(urllib.parse.urlparse(url).path.rsplit("/", 1)[-1])
|
||||
|
||||
|
||||
def check_disk_space(dest_dir: str | Path, needed: int) -> tuple[bool, int]:
|
||||
"""检查目标目录所在盘剩余空间是否 ≥ needed 字节。返回 (够, 剩余字节)。"""
|
||||
dest_dir = Path(dest_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
free = shutil.disk_usage(dest_dir).free
|
||||
return (free >= needed, free)
|
||||
|
||||
|
||||
def _write_provenance(dest: Path, source: str, kind: str, sha256: str, extra: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""落 provenance sidecar({dest}.provenance.json),返回 provenance dict。"""
|
||||
prov: dict[str, Any] = {
|
||||
"source": source,
|
||||
"kind": kind,
|
||||
"sha256": sha256,
|
||||
"fetched_at": _now_iso(),
|
||||
}
|
||||
if extra:
|
||||
prov.update(extra)
|
||||
sidecar = dest.parent / f"{dest.name}.provenance.json"
|
||||
try:
|
||||
sidecar.write_text(json.dumps(prov, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
except OSError: # noqa: BLE001 - provenance 落盘失败不阻断拉取
|
||||
logger.warning("source_fetcher: provenance 落盘失败 %s", sidecar)
|
||||
return prov
|
||||
|
||||
|
||||
# ---- web ----
|
||||
|
||||
async def fetch_web(url: str, dest_dir: str | Path, *, timeout: float = DEFAULT_TIMEOUT) -> dict[str, Any]:
|
||||
"""http/https 流式下载(不设大小上限;Content-Length 已知时下载前查磁盘余量)。"""
|
||||
scheme = urllib.parse.urlparse(url).scheme.lower()
|
||||
if scheme not in ALLOWED_SCHEMES:
|
||||
raise FetchError("BAD_SCHEME", f"仅支持 http/https 协议: {url}")
|
||||
dest_dir = Path(dest_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
name = _name_from_url(url)
|
||||
dest = dest_dir / name
|
||||
|
||||
async with httpx.AsyncClient(follow_redirects=True) as client:
|
||||
try:
|
||||
async with client.stream("GET", url, timeout=httpx.Timeout(connect=30.0, read=timeout, write=60.0, pool=10.0)) as resp:
|
||||
if resp.status_code >= 400:
|
||||
raise FetchError("HTTP_ERROR", f"下载失败 HTTP {resp.status_code}: {url}")
|
||||
total = int(resp.headers.get("content-length") or 0)
|
||||
if total:
|
||||
ok, free = check_disk_space(dest_dir, total)
|
||||
if not ok:
|
||||
raise FetchError("DISK_FULL", f"磁盘空间不足:需 {total} 字节,剩余 {free} 字节")
|
||||
written = 0
|
||||
with dest.open("wb") as fh:
|
||||
async for chunk in resp.aiter_bytes(CHUNK):
|
||||
fh.write(chunk)
|
||||
written += len(chunk)
|
||||
if total and written != total:
|
||||
dest.unlink(missing_ok=True)
|
||||
raise FetchError("TRUNCATED", f"下载不完整:期望 {total} 字节,实收 {written} 字节")
|
||||
except FetchError:
|
||||
raise
|
||||
except httpx.HTTPError as exc:
|
||||
dest.unlink(missing_ok=True)
|
||||
raise FetchError("NETWORK_ERROR", f"网络错误: {exc}") from exc
|
||||
|
||||
size = dest.stat().st_size
|
||||
sha = _sha256(dest)
|
||||
prov = _write_provenance(dest, url, "web", sha)
|
||||
return {"ok": True, "path": str(dest), "name": name, "size": size, "kind": "web", "provenance": prov}
|
||||
|
||||
|
||||
# ---- OSS ----
|
||||
|
||||
def _resolve_oss_url(spec: str) -> str:
|
||||
"""oss://bucket/key → http 下载 URL(public-read 或预签名;无 SDK,需 endpoint 配置)。
|
||||
|
||||
环境键:BIOFLOW_OSS_ENDPOINT(如 oss-cn-hangzhou.aliyuncs.com,或内网 endpoint)。
|
||||
预签名 URL(含 aliyuncs.com 的 http/https)原样返回,走 web 通道。
|
||||
"""
|
||||
if not spec.startswith("oss://"):
|
||||
return spec # 已是 http/https 预签名 URL
|
||||
rest = spec[len("oss://"):]
|
||||
bucket, _, key = rest.partition("/")
|
||||
endpoint = (os.environ.get("BIOFLOW_OSS_ENDPOINT") or "").strip().rstrip("/")
|
||||
if not bucket or not key:
|
||||
raise FetchError("BAD_OSS", f"OSS 地址格式非法(应为 oss://bucket/key): {spec}")
|
||||
if not endpoint:
|
||||
raise FetchError("OSS_UNCONFIGURED", "未配置 BIOFLOW_OSS_ENDPOINT,无法解析 oss:// 地址;请提供预签名 URL")
|
||||
# 内网/公网 endpoint 统一拼 http(https 由 endpoint 决定;公网 OSS 建议 https)
|
||||
scheme = "https" if "aliyuncs.com" in endpoint else "http"
|
||||
return f"{scheme}://{bucket}.{endpoint}/{key}"
|
||||
|
||||
|
||||
async def fetch_oss(spec: str, dest_dir: str | Path, *, timeout: float = DEFAULT_TIMEOUT) -> dict[str, Any]:
|
||||
url = _resolve_oss_url(spec)
|
||||
result = await fetch_web(url, dest_dir, timeout=timeout)
|
||||
result["kind"] = "oss"
|
||||
# 重写 provenance kind(sidecar 已由 fetch_web 写为 web;此处修正)
|
||||
if result.get("provenance"):
|
||||
result["provenance"]["kind"] = "oss"
|
||||
result["provenance"]["oss_spec"] = spec
|
||||
try:
|
||||
sidecar = Path(result["path"]).parent / f"{Path(result['path']).name}.provenance.json"
|
||||
sidecar.write_text(json.dumps(result["provenance"], ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
except OSError:
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
# ---- 局域网 ----
|
||||
|
||||
def _parse_lan_remote(spec: str) -> tuple[str, str] | None:
|
||||
"""解析 `[user@]host:/abs/path` 远端形态;非远端返回 None。"""
|
||||
if ":" not in spec:
|
||||
return None
|
||||
# 排除协议(http:// / oss:// 含 ://,已在上层分流)
|
||||
if "://" in spec:
|
||||
return None
|
||||
host_part, _, path_part = spec.partition(":")
|
||||
if not path_part.startswith("/"):
|
||||
return None
|
||||
return host_part, path_part
|
||||
|
||||
|
||||
async def _scp_fetch(host_part: str, remote_path: str, dest_dir: Path, credentials: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""scp 拉取远端文件(密钥认证;密码需本机 sshpass)。"""
|
||||
creds = credentials or {}
|
||||
user = str(creds.get("user") or "").strip()
|
||||
password = str(creds.get("password") or "").strip()
|
||||
port = str(creds.get("port") or "22").strip()
|
||||
target = f"{user}@{host_part}" if user else host_part
|
||||
name = _safe_name(remote_path.rsplit("/", 1)[-1])
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cmd: list[str] = []
|
||||
if password:
|
||||
if shutil.which("sshpass") is None:
|
||||
raise FetchError("SSHPASS_MISSING", "本机未安装 sshpass,无法使用密码拉取;请改用密钥认证或提供已挂载路径")
|
||||
cmd += ["sshpass", "-p", password]
|
||||
cmd += ["scp", "-P", port, "-o", "StrictHostKeyChecking=accept-new", "-o", "ConnectTimeout=30",
|
||||
f"{target}:{remote_path}", str(dest_dir / name)]
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
|
||||
_out, err = await asyncio.wait_for(proc.communicate(), timeout=DEFAULT_TIMEOUT)
|
||||
except asyncio.TimeoutError as exc:
|
||||
raise FetchError("LAN_TIMEOUT", f"局域网拉取超时: {target}:{remote_path}") from exc
|
||||
except OSError as exc:
|
||||
raise FetchError("LAN_ERROR", f"局域网拉取失败: {exc}") from exc
|
||||
if proc.returncode != 0:
|
||||
raise FetchError("LAN_ERROR", f"scp 失败: {(err or b'').decode(errors='replace')[:500]}")
|
||||
dest = dest_dir / name
|
||||
if not dest.is_file():
|
||||
raise FetchError("LAN_ERROR", f"scp 未产出文件: {dest}")
|
||||
size = dest.stat().st_size
|
||||
sha = _sha256(dest)
|
||||
prov = _write_provenance(dest, f"{target}:{remote_path}", "lan", sha)
|
||||
return {"ok": True, "path": str(dest), "name": name, "size": size, "kind": "lan", "provenance": prov}
|
||||
|
||||
|
||||
async def fetch_lan(spec: str, dest_dir: str | Path, *, credentials: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""局域网路径:本机已挂载/可达直读(返回原路径,不复制);远端走 scp。"""
|
||||
dest_dir = Path(dest_dir)
|
||||
remote = _parse_lan_remote(spec)
|
||||
if remote is not None:
|
||||
return await _scp_fetch(remote[0], remote[1], dest_dir, credentials)
|
||||
# 本机路径:直读(存在即成功;不复制,登记语义由 register_job_files 处理)
|
||||
p = Path(spec).expanduser()
|
||||
if not p.exists():
|
||||
raise FetchError("PATH_NOT_EXIST", f"局域网路径不存在或不可达: {spec}")
|
||||
if p.is_dir():
|
||||
raise FetchError("IS_DIR", f"路径是目录(仅支持单文件拉取): {spec}")
|
||||
sha = _sha256(p)
|
||||
prov = _write_provenance(p, str(p), "lan", sha)
|
||||
return {"ok": True, "path": str(p), "name": p.name, "size": p.stat().st_size, "kind": "lan", "provenance": prov}
|
||||
|
||||
|
||||
# ---- 总入口 ----
|
||||
|
||||
async def fetch_source(
|
||||
source: str,
|
||||
dest_dir: str | Path,
|
||||
*,
|
||||
credentials: dict[str, Any] | None = None,
|
||||
timeout: float = DEFAULT_TIMEOUT,
|
||||
) -> dict[str, Any]:
|
||||
"""按来源形态分流拉取(web/OSS/局域网/本机路径)。
|
||||
|
||||
credentials(可选,仅局域网 scp 用):{"user": ..., "password": ..., "port": ...}。
|
||||
凭据不落盘、不进入 provenance。
|
||||
"""
|
||||
source = (source or "").strip()
|
||||
if not source:
|
||||
raise FetchError("EMPTY_SOURCE", "源地址为空")
|
||||
lower = source.lower()
|
||||
# 协议白名单:含 "://" 的 URL 只允许 http/https/oss(其余如 ftp/file/sftp 拒绝)
|
||||
if "://" in source and not lower.startswith(("http://", "https://", "oss://")):
|
||||
raise FetchError("BAD_SCHEME", f"不支持的协议(仅 http/https/oss): {source}")
|
||||
if lower.startswith(("http://", "https://")):
|
||||
if lower.startswith("oss://") or "aliyuncs.com" in lower:
|
||||
return await fetch_oss(source, dest_dir, timeout=timeout)
|
||||
return await fetch_web(source, dest_dir, timeout=timeout)
|
||||
if lower.startswith("oss://"):
|
||||
return await fetch_oss(source, dest_dir, timeout=timeout)
|
||||
return await fetch_lan(source, dest_dir, credentials=credentials)
|
||||
@@ -0,0 +1,101 @@
|
||||
"""app/services/bioflow/stage_assets.py —— 角色化资产 + 下游直入判定(G7/G8)。
|
||||
|
||||
- 阶段 → 输入角色(数据角色):counts/matrix/bam/reference/samples_yaml/hmm/gff3。
|
||||
- 角色 → 上游阶段(产出该角色的阶段链):若角色已有 validated 资产,这些上游可跳过。
|
||||
- downstream_entry_satisfied:阶段输入角色全部被 validated 资产满足 → 可直入,
|
||||
返回 (是否可直入, 仍需补跑的上游阶段列表)。
|
||||
|
||||
服务层纪律:纯 dict/stdlib,无 pydantic。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Iterable
|
||||
|
||||
# 阶段 → 输入角色(该阶段需要的数据角色;仅列「由上游阶段产出」的角色,
|
||||
# 用户直接提供的文件如 samples_yaml/design 不算前置角色)
|
||||
STAGE_INPUT_ROLES: dict[str, list[str]] = {
|
||||
"rnaseq_qc": ["fastq"],
|
||||
"rnaseq_align": ["fastq", "reference"],
|
||||
"rnaseq_quant": ["bam", "gtf"],
|
||||
"rnaseq_sample_qc": ["counts"],
|
||||
"rnaseq_de": ["counts"],
|
||||
"rnaseq_wgcna": ["counts"],
|
||||
"rnaseq_enrich": ["de_results"],
|
||||
"rnaseq_go_enrich": ["de_results"],
|
||||
"rnaseq_kegg_enrich": ["de_results"],
|
||||
"rnaseq_fgsea": ["de_results"],
|
||||
"rnaseq_candidate_family": ["de_results"],
|
||||
"rnaseq_report": ["de_results"],
|
||||
"identify": ["hmm", "protein_fasta"],
|
||||
"genome_prepare": ["reference"],
|
||||
"tfbs": ["reference", "gff3"],
|
||||
}
|
||||
|
||||
# 角色 → 上游阶段(产出该角色的阶段链;角色已有 validated 资产时可跳过)
|
||||
ROLE_UPSTREAM_STAGES: dict[str, list[str]] = {
|
||||
"counts": ["rnaseq_qc", "rnaseq_align", "rnaseq_quant", "rnaseq_sample_qc"],
|
||||
"bam": ["rnaseq_qc", "rnaseq_align"],
|
||||
"fastq": ["rnaseq_qc"],
|
||||
"de_results": ["rnaseq_de"],
|
||||
"reference": ["genome_prepare", "wgs_index"],
|
||||
}
|
||||
|
||||
|
||||
def stage_input_roles(stage: str) -> list[str]:
|
||||
"""G8:阶段依赖的输入角色列表(无声明返回空列表)。"""
|
||||
return list(STAGE_INPUT_ROLES.get(stage, []))
|
||||
|
||||
|
||||
def _asset_is_validated(value: Any) -> bool:
|
||||
"""资产值是否「validated」:dict 含 validated 键则看 validated.ok,否则非空即视为有资产。"""
|
||||
if isinstance(value, dict):
|
||||
if "validated" in value:
|
||||
return bool((value.get("validated") or {}).get("ok"))
|
||||
return bool(value)
|
||||
return bool(value)
|
||||
|
||||
|
||||
def asset_role_satisfied(role: str, assets: Any) -> bool:
|
||||
"""G8:role 是否被 validated 资产满足。
|
||||
|
||||
assets 可为:
|
||||
- 单个资产 dict:{"role": "counts", "validated": {"ok": True}};
|
||||
- 角色映射 dict:{"counts": {"validated": {"ok": True}}, ...};
|
||||
- list/set[str](角色名集合)。
|
||||
"""
|
||||
if not role:
|
||||
return False
|
||||
if isinstance(assets, dict):
|
||||
if "role" in assets:
|
||||
# 单个资产 dict
|
||||
if assets.get("role") != role:
|
||||
return False
|
||||
return bool((assets.get("validated") or {}).get("ok"))
|
||||
# 角色映射 dict
|
||||
return _asset_is_validated(assets.get(role))
|
||||
if isinstance(assets, (list, tuple, set, frozenset)):
|
||||
return role in set(assets)
|
||||
return False
|
||||
|
||||
|
||||
def downstream_entry_satisfied(stage: str, assets: Any) -> tuple[bool, list[str]]:
|
||||
"""G7:阶段输入角色是否全部由 validated 资产满足(可直接进入,不补跑上游)。
|
||||
|
||||
返回 (是否可直入, 仍需补跑的上游阶段列表)。无角色声明 → (False, [])(保守)。
|
||||
"""
|
||||
roles = stage_input_roles(stage)
|
||||
if not roles:
|
||||
return False, []
|
||||
missing_upstream: list[str] = []
|
||||
for role in roles:
|
||||
if not asset_role_satisfied(role, assets):
|
||||
missing_upstream.extend(ROLE_UPSTREAM_STAGES.get(role, []))
|
||||
return (not missing_upstream), missing_upstream
|
||||
|
||||
|
||||
def upstream_stages_for_roles(satisfied_roles: Iterable[str]) -> set[str]:
|
||||
"""已满足角色对应的上游阶段集合(这些上游可跳过,不补跑)。"""
|
||||
out: set[str] = set()
|
||||
for role in satisfied_roles:
|
||||
out.update(ROLE_UPSTREAM_STAGES.get(role, []))
|
||||
return out
|
||||
@@ -0,0 +1,46 @@
|
||||
"""app/services/bioflow/task_context.py —— 任务绑定上下文(G11)。
|
||||
|
||||
返回 session_id + job_id + plan_revision + run_id + current_stage,供 MCP
|
||||
get_task_context 与 Skill 编排绑定「本次运行」到具体任务与阶段。
|
||||
服务层纪律:纯 dict/stdlib,无 pydantic。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
# 阶段终态(不视为「当前阶段」)
|
||||
_TERMINAL = ("done", "skipped", "archived")
|
||||
|
||||
|
||||
def current_stage(record: Any) -> str | None:
|
||||
"""当前阶段:阶段链中第一个未达终态的阶段;全部终态则取末阶段。"""
|
||||
chain = getattr(record, "stage_chain", None) or []
|
||||
if not chain:
|
||||
return None
|
||||
for stage in chain:
|
||||
st = (record._stage_states.get(stage, {}) or {}).get("status")
|
||||
if st not in _TERMINAL:
|
||||
return stage
|
||||
return chain[-1]
|
||||
|
||||
|
||||
def get_task_context(record: Any | None = None) -> dict[str, Any]:
|
||||
"""G11:任务绑定上下文(session_id/job_id/plan_revision/run_id/当前阶段)。
|
||||
|
||||
record 缺省返回空绑定(5 字段齐全);传 JobRecord 时返回实际绑定。
|
||||
"""
|
||||
if record is None:
|
||||
return {
|
||||
"session_id": "",
|
||||
"job_id": "",
|
||||
"plan_revision": "1",
|
||||
"run_id": "",
|
||||
"current_stage": None,
|
||||
}
|
||||
return {
|
||||
"session_id": getattr(record, "session_id", None) or "",
|
||||
"job_id": getattr(record, "id", None) or "",
|
||||
"plan_revision": getattr(record, "plan_revision", None) or "1",
|
||||
"run_id": getattr(record, "run_id", None) or getattr(record, "id", None) or "",
|
||||
"current_stage": current_stage(record),
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
"""app/services/bioflow/versioning.py —— 版本化修改(G16/G17/G18/G19)。
|
||||
|
||||
- G16 版本化身份:new_revision(based_on_revision) 返回 revision/based_on_revision/
|
||||
asset_id/action_id;make_action_id() 生成唯一 action id。
|
||||
(plan_revision/run_id 已在 JobRecord,见 runner.py + task_context.py)
|
||||
- G17 影响范围:affected_stages(stage_chain, changed_stage) 返回改参数后受影响的
|
||||
下游阶段(依赖图传递闭包,按 stage_chain 序,只重跑受影响节点)。
|
||||
- G18 历史不覆盖:RevisionStore(root) 按 revision 物理隔离存 key→content,
|
||||
新 revision 不覆盖旧 revision。
|
||||
- G19 防重复:ActionDedup().try_start(action_id) 幂等(首次 True,重复 False)。
|
||||
|
||||
服务层纪律:纯 dataclass/dict/stdlib,无 pydantic。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
# ---- G16 版本化身份 ----
|
||||
|
||||
def _next_revision(base: str | None) -> str:
|
||||
"""无 base → r1;base 形如 rN 或 N → r(N+1)。"""
|
||||
if not base:
|
||||
return "r1"
|
||||
try:
|
||||
n = int(str(base).lstrip("r") or 0) + 1
|
||||
except ValueError:
|
||||
n = 1
|
||||
return f"r{n}"
|
||||
|
||||
|
||||
def new_plan_revision(base: str | None = None) -> str:
|
||||
"""新 plan_revision(G11/G16 共用)。"""
|
||||
return _next_revision(base)
|
||||
|
||||
|
||||
def make_action_id() -> str:
|
||||
"""G16:唯一 action id。"""
|
||||
return uuid.uuid4().hex[:16]
|
||||
|
||||
|
||||
def new_run_id() -> str:
|
||||
"""G16:新执行标识。"""
|
||||
return uuid.uuid4().hex[:16]
|
||||
|
||||
|
||||
def new_asset_id() -> str:
|
||||
"""G16:新资产标识。"""
|
||||
return uuid.uuid4().hex[:16]
|
||||
|
||||
|
||||
def new_revision(based_on_revision: str | None = None) -> dict[str, Any]:
|
||||
"""G16:产生一次版本化变更身份。
|
||||
|
||||
返回 {"revision": str, "based_on_revision": str|None,
|
||||
"asset_id": str, "action_id": str}。
|
||||
"""
|
||||
return {
|
||||
"revision": _next_revision(based_on_revision),
|
||||
"based_on_revision": based_on_revision,
|
||||
"asset_id": new_asset_id(),
|
||||
"action_id": make_action_id(),
|
||||
}
|
||||
|
||||
|
||||
# ---- G17 影响范围 ----
|
||||
|
||||
def _downstream_closure(changed_stage: str) -> set[str]:
|
||||
"""依赖图反向传递闭包:changed_stage 的所有下游阶段。"""
|
||||
from .catalog import STAGE_DEPENDENCIES_SNAPSHOT
|
||||
|
||||
rev: dict[str, list[str]] = {}
|
||||
for s, deps in STAGE_DEPENDENCIES_SNAPSHOT.items():
|
||||
for d in deps:
|
||||
rev.setdefault(d, []).append(s)
|
||||
closure: set[str] = set()
|
||||
stack: list[str] = list(rev.get(changed_stage, []))
|
||||
while stack:
|
||||
s = stack.pop()
|
||||
if s in closure:
|
||||
continue
|
||||
closure.add(s)
|
||||
stack.extend(rev.get(s, []))
|
||||
return closure
|
||||
|
||||
|
||||
def affected_stages(stage_chain: list[str], changed_stage: str) -> list[str]:
|
||||
"""G17:改 changed_stage 参数后受影响的下游阶段列表(按 stage_chain 序)。"""
|
||||
closure = _downstream_closure(changed_stage)
|
||||
return [s for s in stage_chain if s in closure]
|
||||
|
||||
|
||||
# ---- G18 历史不覆盖 ----
|
||||
|
||||
class RevisionStore:
|
||||
"""按 revision 物理隔离存储(G18):新 revision 不覆盖旧 revision。"""
|
||||
|
||||
def __init__(self, root: str | Path) -> None:
|
||||
self.root = Path(root)
|
||||
|
||||
def _path(self, revision: str, key: str) -> Path:
|
||||
return self.root / revision / Path(key).name
|
||||
|
||||
def write(self, revision: str, key: str, content: str) -> Path:
|
||||
"""写 key→content 到 revision 独立路径;返回写入路径(不覆盖旧 revision)。"""
|
||||
p = self._path(revision, key)
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(content, encoding="utf-8")
|
||||
return p
|
||||
|
||||
def read(self, revision: str, key: str) -> str | None:
|
||||
"""读 revision 下 key 的内容;不存在返回 None。"""
|
||||
p = self._path(revision, key)
|
||||
if not p.exists():
|
||||
return None
|
||||
return p.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
# ---- G19 防重复 ----
|
||||
|
||||
class ActionDedup:
|
||||
"""action_id 幂等(G19):同一 action_id 重复提交不重复启动。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._started: set[str] = set()
|
||||
|
||||
def try_start(self, action_id: str) -> bool:
|
||||
"""首次 True,重复 False(幂等,不重复启动)。"""
|
||||
if action_id in self._started:
|
||||
return False
|
||||
self._started.add(action_id)
|
||||
return True
|
||||
@@ -0,0 +1,6 @@
|
||||
"""bioflowctl —— 生信业务逻辑 CLI(唯一事实源)。
|
||||
|
||||
无状态、幂等、直连 SQLite + 文件系统。后续 MCP 薄适配与 Skill 接入建立在本 CLI 之上。
|
||||
|
||||
用法:`python -m bioflowctl <子命令> ...`(cwd = bioflow_service/)。
|
||||
"""
|
||||
@@ -0,0 +1,176 @@
|
||||
"""bioflowctl/__main__.py —— CLI 入口。
|
||||
|
||||
用法(cwd = bioflow_service/):
|
||||
python -m bioflowctl source fetch <url|oss|lan> [--job-id JOB_ID] [--cred-user U] [--cred-password P] [--cred-port PORT]
|
||||
python -m bioflowctl file inspect <path>
|
||||
python -m bioflowctl file transform <src> --to <fmt> [--column-map JSON] [--dest PATH]
|
||||
python -m bioflowctl asset register --role <role> <path>
|
||||
python -m bioflowctl asset validate --role <role> <path>
|
||||
python -m bioflowctl report register --path <p>
|
||||
python -m bioflowctl archive <job_id> --name <n> [--description <d>]
|
||||
python -m bioflowctl archive reuse --name <n>
|
||||
|
||||
输出:JSON(stdout);ok=False 时退出码 1。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
from . import commands
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="bioflowctl",
|
||||
description="PlantAI 生信业务逻辑 CLI(唯一事实源;无状态/幂等/直连 SQLite)",
|
||||
)
|
||||
sub = parser.add_subparsers(dest="group", required=True)
|
||||
|
||||
# source fetch
|
||||
g = sub.add_parser("source", help="源文件拉取(web/oss/lan)")
|
||||
ss = g.add_subparsers(dest="action", required=True)
|
||||
p = ss.add_parser("fetch", help="拉取源文件到任务 input 或 _incoming")
|
||||
p.add_argument("source", help="http/https URL、oss://bucket/key 或局域网路径")
|
||||
p.add_argument("--job-id", dest="job_id", default=None, help="目标任务 id(缺省落 _incoming)")
|
||||
p.add_argument("--cred-user", dest="cred_user", default=None, help="局域网 scp 用户名")
|
||||
p.add_argument("--cred-password", dest="cred_password", default=None, help="局域网 scp 密码(仅内存)")
|
||||
p.add_argument("--cred-port", dest="cred_port", default=None, help="局域网 scp 端口")
|
||||
|
||||
# file inspect / transform
|
||||
g = sub.add_parser("file", help="文件探测 / 转换")
|
||||
ss = g.add_subparsers(dest="action", required=True)
|
||||
p = ss.add_parser("inspect", help="探测文件格式/预览/列结构")
|
||||
p.add_argument("path")
|
||||
p = ss.add_parser("transform", help="非标准文件转 fasta/gff3/tsv/samples.yaml")
|
||||
p.add_argument("src")
|
||||
p.add_argument("--to", dest="target", required=True, help="目标格式")
|
||||
p.add_argument("--column-map", dest="column_map", default=None, help='列映射 JSON(如 {"name":"基因名","seq":"序列"})')
|
||||
p.add_argument("--dest", dest="dest", default=None, help="输出路径(缺省自动命名)")
|
||||
|
||||
# asset register / validate
|
||||
g = sub.add_parser("asset", help="角色化资产登记 / 审核")
|
||||
ss = g.add_subparsers(dest="action", required=True)
|
||||
p = ss.add_parser("register", help="角色化登记 + 原子发布 + 同名冲突检测")
|
||||
p.add_argument("--role", dest="role", required=True, help="资产角色(如 pep.fa / gff3 / samples.yaml)")
|
||||
p.add_argument("path")
|
||||
p = ss.add_parser("validate", help="按角色内容审核")
|
||||
p.add_argument("--role", dest="role", required=True)
|
||||
p.add_argument("path")
|
||||
|
||||
# report register
|
||||
g = sub.add_parser("report", help="报告登记")
|
||||
ss = g.add_subparsers(dest="action", required=True)
|
||||
p = ss.add_parser("register", help="登记报告(拒绝空路径/空文件)")
|
||||
p.add_argument("--path", dest="path", required=True)
|
||||
|
||||
# archive / archive reuse
|
||||
g = sub.add_parser("archive", help="任务存档 / 复用")
|
||||
g.add_argument("target", help="任务 id,或字面 'reuse'")
|
||||
g.add_argument("--name", dest="name", required=True, help="归档名")
|
||||
g.add_argument("--description", dest="description", default="")
|
||||
|
||||
# check readiness
|
||||
g = sub.add_parser("check", help="就绪检查")
|
||||
ss = g.add_subparsers(dest="action", required=True)
|
||||
ss.add_parser("readiness", help="四象限就绪检查(环境/参考/磁盘/预算)")
|
||||
|
||||
# change propose / apply
|
||||
g = sub.add_parser("change", help="版本化变更")
|
||||
ss = g.add_subparsers(dest="action", required=True)
|
||||
p = ss.add_parser("propose", help="计算参数变更的影响范围")
|
||||
p.add_argument("--stage", dest="stage", required=True)
|
||||
p = ss.add_parser("apply", help="幂等应用变更")
|
||||
p.add_argument("--action-id", dest="action_id", required=True)
|
||||
|
||||
# plan revision list
|
||||
g = sub.add_parser("plan", help="规划 revision")
|
||||
ss = g.add_subparsers(dest="action", required=True)
|
||||
p = ss.add_parser("revision", help="列出历史 revision")
|
||||
p.add_argument("--root", dest="root", required=True)
|
||||
|
||||
# run node-execute / stop / resume
|
||||
g = sub.add_parser("run", help="执行控制")
|
||||
ss = g.add_subparsers(dest="action", required=True)
|
||||
p = ss.add_parser("node-execute", help="只重跑受影响节点")
|
||||
p.add_argument("--stage", dest="stage", required=True)
|
||||
p = ss.add_parser("stop", help="终止实际执行")
|
||||
p.add_argument("--run-id", dest="run_id", required=True)
|
||||
p.add_argument("--owner", dest="owner", required=True)
|
||||
p = ss.add_parser("resume", help="恢复执行")
|
||||
p.add_argument("--run-id", dest="run_id", required=True)
|
||||
p.add_argument("--owner", dest="owner", required=True)
|
||||
|
||||
# delivery validate / publish
|
||||
g = sub.add_parser("delivery", help="交付清单")
|
||||
ss = g.add_subparsers(dest="action", required=True)
|
||||
p = ss.add_parser("validate", help="校验交付清单完整性")
|
||||
p.add_argument("--manifest", dest="manifest", required=True, help="清单 JSON 文件路径")
|
||||
p = ss.add_parser("publish", help="发布交付清单")
|
||||
p.add_argument("--manifest", dest="manifest", required=True, help="清单 JSON 文件路径")
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def _dispatch(args: argparse.Namespace) -> dict[str, Any]:
|
||||
if args.group == "source" and args.action == "fetch":
|
||||
creds: dict[str, str] = {}
|
||||
if args.cred_user:
|
||||
creds["user"] = args.cred_user
|
||||
if args.cred_password:
|
||||
creds["password"] = args.cred_password
|
||||
if args.cred_port:
|
||||
creds["port"] = args.cred_port
|
||||
return asyncio.run(commands.cmd_source_fetch(
|
||||
args.source, job_id=args.job_id, credentials=creds or None))
|
||||
if args.group == "file" and args.action == "inspect":
|
||||
return commands.cmd_file_inspect(args.path)
|
||||
if args.group == "file" and args.action == "transform":
|
||||
column_map = None
|
||||
if args.column_map:
|
||||
column_map = json.loads(args.column_map)
|
||||
return asyncio.run(commands.cmd_file_transform(
|
||||
args.src, args.target, column_map=column_map, dest=args.dest))
|
||||
if args.group == "asset" and args.action == "register":
|
||||
return commands.register_asset(args.role, args.path)
|
||||
if args.group == "asset" and args.action == "validate":
|
||||
return commands.validate_asset(args.role, args.path)
|
||||
if args.group == "report" and args.action == "register":
|
||||
return commands.cmd_report_register(args.path)
|
||||
if args.group == "archive" and args.target == "reuse":
|
||||
return asyncio.run(commands.cmd_archive_reuse(args.name))
|
||||
if args.group == "archive":
|
||||
return asyncio.run(commands.cmd_archive(args.target, args.name, args.description))
|
||||
if args.group == "check" and args.action == "readiness":
|
||||
return commands.cmd_check_readiness()
|
||||
if args.group == "change" and args.action == "propose":
|
||||
return commands.cmd_change_propose(args.stage)
|
||||
if args.group == "change" and args.action == "apply":
|
||||
return commands.cmd_change_apply(args.action_id)
|
||||
if args.group == "plan" and args.action == "revision":
|
||||
return commands.cmd_plan_revision_list(args.root)
|
||||
if args.group == "run" and args.action == "node-execute":
|
||||
return commands.cmd_run_node_execute(args.stage)
|
||||
if args.group == "run" and args.action == "stop":
|
||||
return commands.cmd_run_stop(args.run_id, args.owner)
|
||||
if args.group == "run" and args.action == "resume":
|
||||
return commands.cmd_run_resume(args.run_id, args.owner)
|
||||
if args.group == "delivery" and args.action == "validate":
|
||||
return commands.cmd_delivery_validate(args.manifest)
|
||||
if args.group == "delivery" and args.action == "publish":
|
||||
return commands.cmd_delivery_publish(args.manifest)
|
||||
return {"ok": False, "code": "USAGE", "message": "未知子命令组合"}
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
result = _dispatch(args)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return 0 if result.get("ok") else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,439 @@
|
||||
"""bioflowctl/commands.py —— 子命令实现(无状态、幂等、直连 SQLite + 文件系统)。
|
||||
|
||||
每个命令返回 dict({"ok": True/False, ...}),由 __main__ 序列化为 JSON 并定退出码。
|
||||
|
||||
复用服务层单一事实源(不复刻逻辑,只做直连编排):
|
||||
- source_fetcher.fetch_source —— web/oss/lan 流式拉取(G5 已流式)
|
||||
- file_transform.inspect_file / transform_file / validate_format
|
||||
- archive.archive_from_job / effective_stage_params / normalize_archive_name
|
||||
- reports.locate_report / extract_figure_refs / validate_report_path
|
||||
- store(aiosqlite 直连 bioflow.db)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.config import get_settings
|
||||
from app.db import Database
|
||||
from app.services.bioflow import store
|
||||
from app.services.bioflow.archive import (
|
||||
archive_from_job,
|
||||
effective_stage_params,
|
||||
normalize_archive_name,
|
||||
)
|
||||
from app.services.bioflow.errors import error_result
|
||||
from app.services.bioflow.file_transform import (
|
||||
TransformError,
|
||||
inspect_file,
|
||||
transform_file,
|
||||
validate_format,
|
||||
)
|
||||
from app.services.bioflow.reports import (
|
||||
extract_figure_refs,
|
||||
locate_report,
|
||||
validate_report_path,
|
||||
)
|
||||
from app.services.bioflow.source_fetcher import FetchError, fetch_source
|
||||
|
||||
_COPY_CHUNK = 1024 * 1024 # 1MB 流式复制缓冲(G5)
|
||||
|
||||
|
||||
def _err(code: str, message: str, *, evidence_refs: list[str] | None = None) -> dict[str, Any]:
|
||||
"""G13:失败返回统一信封(code 顶层 + error 结构化对象)。"""
|
||||
return error_result(code, message, evidence_refs=evidence_refs)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# source fetch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def cmd_source_fetch(
|
||||
source: str,
|
||||
*,
|
||||
job_id: str | None = None,
|
||||
credentials: dict[str, Any] | None = None,
|
||||
fetch_fn: Any | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""拉取源文件(web/oss/lan);job_id 指定时落任务 input,否则落 _incoming。
|
||||
|
||||
fetch_fn 供 MCP 薄适配注入(保持模块级 fetch_source 可 monkeypatch)。
|
||||
"""
|
||||
settings = get_settings()
|
||||
if job_id:
|
||||
dest_dir: Path = Path(settings.jobs_dir_abs) / job_id / "input"
|
||||
else:
|
||||
dest_dir = Path(settings.jobs_dir_abs) / "_incoming" / uuid.uuid4().hex[:8]
|
||||
fetch = fetch_fn or fetch_source
|
||||
try:
|
||||
result = await fetch(source, dest_dir, credentials=credentials)
|
||||
except FetchError as exc:
|
||||
return _err(exc.code, exc.message)
|
||||
result["job_id"] = job_id
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# file inspect / transform
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def cmd_file_inspect(path: str) -> dict[str, Any]:
|
||||
result = inspect_file(path)
|
||||
if not result.get("ok"):
|
||||
return _err(result.get("code") or "READ_ERROR", result.get("message") or "")
|
||||
return result
|
||||
|
||||
|
||||
async def cmd_file_transform(
|
||||
src: str,
|
||||
target: str,
|
||||
*,
|
||||
column_map: dict[str, str] | None = None,
|
||||
dest: str | None = None,
|
||||
llm: Any | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""非标准文件转换(LLM 不注入则启发式 + 显式 column_map;可测、确定性)。"""
|
||||
try:
|
||||
return await transform_file(
|
||||
src, target, column_map=column_map, dest_path=dest, llm=llm,
|
||||
)
|
||||
except TransformError as exc:
|
||||
evidence = [str(exc.details)] if exc.details else None
|
||||
return _err(exc.code, exc.message, evidence_refs=evidence)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# asset register / validate(G2 原子发布 + 同名冲突;G8 角色化资产基础)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _sanitize_role(role: str) -> str:
|
||||
r = (role or "").strip().strip("/")
|
||||
if not r or ".." in r or "/" in r or "\\" in r:
|
||||
return ""
|
||||
return r
|
||||
|
||||
|
||||
def _safe_name(name: str) -> str:
|
||||
return Path(name or "asset").name or "asset"
|
||||
|
||||
|
||||
def _assets_root(assets_root: str | Path | None) -> Path:
|
||||
if assets_root is not None:
|
||||
return Path(assets_root)
|
||||
return Path(get_settings().data_root_abs) / "assets"
|
||||
|
||||
|
||||
def register_asset(
|
||||
role: str, path: str, assets_root: str | Path | None = None
|
||||
) -> dict[str, Any]:
|
||||
"""角色化登记 + 原子发布 + 同名冲突检测(G2/G8 基础)。
|
||||
|
||||
- 流式复制到角色目录(G5:shutil.copyfileobj 1MB 缓冲,不整块进内存)。
|
||||
- 临时文件 os.link 原子发布,目标已存在时显式报错,绝不静默覆盖。
|
||||
"""
|
||||
role = _sanitize_role(role)
|
||||
if not role:
|
||||
return _err("BAD_ROLE", "角色名非法(须非空、不含路径分隔符/..)")
|
||||
src = Path(path).expanduser()
|
||||
if not src.is_file():
|
||||
return _err("NOT_FOUND", f"源文件不存在: {path}")
|
||||
dest_dir = _assets_root(assets_root) / role
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
name = _safe_name(src.name)
|
||||
dest = dest_dir / name
|
||||
if dest.exists() or dest.is_symlink():
|
||||
return _err("ASSET_EXISTS", f"同名资产已存在,拒绝覆盖: {role}/{name}")
|
||||
tmp = dest_dir / f".tmp-{uuid.uuid4().hex[:8]}-{name}"
|
||||
try:
|
||||
with src.open("rb") as fin, tmp.open("wb") as fout:
|
||||
shutil.copyfileobj(fin, fout, length=_COPY_CHUNK)
|
||||
# 原子发布:os.link 目标已存在时抛 FileExistsError(不覆盖)
|
||||
os.link(tmp, dest)
|
||||
except FileExistsError:
|
||||
return _err("ASSET_EXISTS", f"同名资产已存在,拒绝覆盖: {role}/{name}")
|
||||
except OSError as exc:
|
||||
if dest.exists() or dest.is_symlink():
|
||||
return _err("ASSET_EXISTS", f"同名资产已存在,拒绝覆盖: {role}/{name}")
|
||||
return _err("PUBLISH_FAILED", f"资产发布失败: {exc}")
|
||||
finally:
|
||||
tmp.unlink(missing_ok=True)
|
||||
return {"ok": True, "role": role, "name": name, "path": str(dest),
|
||||
"size": dest.stat().st_size}
|
||||
|
||||
|
||||
def _role_to_format(role: str) -> str | None:
|
||||
"""角色名 → 期望格式(校验器键)。无法推断返回 None(走内容嗅探)。"""
|
||||
r = (role or "").lower()
|
||||
if "samples" in r:
|
||||
return "samples.yaml"
|
||||
if "gff" in r or "gtf" in r:
|
||||
return "gff3"
|
||||
if r.endswith(".tsv") or ".tsv" in r:
|
||||
return "tsv"
|
||||
if any(k in r for k in ("fasta", ".fa", ".faa", ".fna", "pep", "cds")):
|
||||
return "fasta"
|
||||
return None
|
||||
|
||||
|
||||
def validate_asset(role: str, path: str) -> dict[str, Any]:
|
||||
"""内容审核:按角色推断期望格式并跑格式校验器。"""
|
||||
p = Path(path).expanduser()
|
||||
if not p.is_file():
|
||||
return _err("NOT_FOUND", f"文件不存在: {path}")
|
||||
fmt = _role_to_format(role)
|
||||
if fmt is None:
|
||||
info = inspect_file(p)
|
||||
guessed = info.get("format") if info.get("ok") else None
|
||||
fmt = guessed if guessed in ("fasta", "gff3", "tsv") else None
|
||||
if fmt is None:
|
||||
return _err("BAD_ROLE",
|
||||
f"无法从角色 {role!r} 推断期望格式(可选 fasta/gff3/tsv/samples.yaml)")
|
||||
validated = validate_format(p, fmt)
|
||||
if not validated.get("ok"):
|
||||
return _err("VALIDATION_FAILED", validated.get("reason") or f"{fmt} 校验失败",
|
||||
evidence_refs=[str(validated)])
|
||||
return {"ok": True, "role": role, "format": fmt, "validated": validated}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# report register(G3:拒绝空路径/空文件)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def cmd_report_register(path: str) -> dict[str, Any]:
|
||||
ok, err = validate_report_path(path)
|
||||
if not ok:
|
||||
return _err("EMPTY_REPORT", err)
|
||||
p = Path(path)
|
||||
figures = extract_figure_refs(p)
|
||||
return {"ok": True, "path": str(p), "size": p.stat().st_size, "figures": figures}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# archive / archive reuse(G1 产物清单 + G6 生效参数)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def cmd_archive(
|
||||
job_id: str,
|
||||
name: str,
|
||||
description: str = "",
|
||||
*,
|
||||
db: Any | None = None,
|
||||
db_path: str | Path | None = None,
|
||||
jobs_dir: str | Path | None = None,
|
||||
archives_dir: str | Path | None = None,
|
||||
include_intermediate: bool = False,
|
||||
archive_fn: Any | None = None,
|
||||
record: Any | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""按 manifest 必需产物清单存档(G1)+ 保存各阶段实际生效参数(G6)。
|
||||
|
||||
- db:已连接 Database(MCP 薄适配注入 ctx.db;缺省自建连接)。
|
||||
- record:内存 JobRecord(MCP 传入;缺省走 DB 查询)。
|
||||
- archive_fn:注入 archive_from_job(供测试 monkeypatch)。
|
||||
"""
|
||||
settings = get_settings()
|
||||
own_db = db is None
|
||||
if db is None:
|
||||
db = Database(db_path)
|
||||
await db.connect()
|
||||
try:
|
||||
archive_name = normalize_archive_name(name)
|
||||
existing = await store.find_archive_by_name(db, archive_name)
|
||||
if existing is not None:
|
||||
return _err("ARCHIVE_EXISTS", f"归档名已存在: {archive_name}")
|
||||
if record is not None:
|
||||
job = {
|
||||
"id": record.id, "session_id": record.session_id,
|
||||
"title": record.title, "stage_chain": record.stage_chain,
|
||||
"params": record.params,
|
||||
}
|
||||
stage_states = getattr(record, "_stage_states", {}) or {}
|
||||
job_dir = record.job_dir
|
||||
else:
|
||||
job = await store.get_job(db, job_id)
|
||||
if job is None:
|
||||
return _err("NOT_FOUND", f"任务不存在: {job_id}")
|
||||
stages = await store.list_stages(db, job_id)
|
||||
stage_states = {s["stage"]: {"params": s["params"]} for s in stages}
|
||||
job_dir = Path(jobs_dir or settings.jobs_dir_abs) / job_id
|
||||
# G6:各阶段实际生效参数(非创建时 params)
|
||||
effective_params = effective_stage_params(job["stage_chain"], stage_states)
|
||||
archives_root = archives_dir or settings.archives_dir_abs
|
||||
report_path = locate_report(job_dir / "workspace")
|
||||
afn = archive_fn or archive_from_job
|
||||
try:
|
||||
result = await afn(
|
||||
{"id": job_id, "session_id": job["session_id"], "title": job["title"],
|
||||
"stage_chain": job["stage_chain"], "params": effective_params},
|
||||
job_dir,
|
||||
archives_root,
|
||||
name=archive_name,
|
||||
description=description,
|
||||
include_intermediate=bool(include_intermediate),
|
||||
report_artifact=str(report_path) if report_path else None,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - 存档失败统一映射 ARCHIVE_FAILED
|
||||
return _err("ARCHIVE_FAILED", f"存档失败: {type(exc).__name__}: {exc}")
|
||||
await store.insert_archive(db, {
|
||||
"id": result.id, "job_id": job_id, "name": archive_name,
|
||||
"description": description, "stage_chain": job["stage_chain"],
|
||||
"params": effective_params,
|
||||
"report_artifact": result.report_artifact,
|
||||
"disk_path": result.disk_path, "size_bytes": result.size_bytes,
|
||||
})
|
||||
return {"ok": True, "data": {
|
||||
"id": result.id, "job_id": job_id, "name": archive_name,
|
||||
"description": description, "disk_path": result.disk_path,
|
||||
"size_bytes": result.size_bytes, "report_artifact": result.report_artifact,
|
||||
}}
|
||||
finally:
|
||||
if own_db:
|
||||
await db.close()
|
||||
|
||||
|
||||
async def cmd_archive_reuse(
|
||||
name: str, *, db: Any | None = None, db_path: str | Path | None = None
|
||||
) -> dict[str, Any]:
|
||||
"""按名称匹配存档,返回阶段链 + 各阶段实际生效参数(G6 读侧)。"""
|
||||
own_db = db is None
|
||||
if db is None:
|
||||
db = Database(db_path)
|
||||
await db.connect()
|
||||
try:
|
||||
arch = await store.find_archive_by_name(db, (name or "").strip())
|
||||
if arch is None:
|
||||
return _err("ARCHIVE_NOT_FOUND", f"未找到匹配存档: {name}")
|
||||
return {"ok": True, "data": {
|
||||
"id": arch["id"], "name": arch["name"],
|
||||
"stage_chain": arch["stage_chain"],
|
||||
"params": arch["params"], "description": arch["description"],
|
||||
}}
|
||||
finally:
|
||||
if own_db:
|
||||
await db.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# check readiness(G10:四象限就绪检查)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def cmd_check_readiness(
|
||||
*, assets_root: str | Path | None = None, stages: list[str] | None = None
|
||||
) -> dict[str, Any]:
|
||||
"""G10:四象限就绪检查;缺项 ready=False + 结构化错误阻断。"""
|
||||
from app.services.bioflow.readiness import check_readiness
|
||||
|
||||
root = assets_root or get_settings().data_root_abs
|
||||
return check_readiness(assets_root=root, stages=stages or [])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# change propose / apply / plan revision list(G17/G18/G19)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def cmd_change_propose(
|
||||
changed_stage: str, stage_chain: list[str] | None = None
|
||||
) -> dict[str, Any]:
|
||||
"""G17:给定参数变更,经依赖图计算受影响的下游阶段列表。"""
|
||||
from app.services.bioflow.versioning import affected_stages
|
||||
chain = list(stage_chain or [])
|
||||
affected = affected_stages(chain + [changed_stage] if changed_stage not in chain else chain,
|
||||
changed_stage)
|
||||
return {"ok": True, "data": {"changed_stage": changed_stage, "affected": affected}}
|
||||
|
||||
|
||||
def cmd_change_apply(action_id: str, *, dedup: Any = None) -> dict[str, Any]:
|
||||
"""G19:change apply 幂等——同一 action_id 重复提交不重复启动。"""
|
||||
from app.services.bioflow.versioning import ActionDedup
|
||||
d = dedup or ActionDedup()
|
||||
started = d.try_start(action_id)
|
||||
return {"ok": True, "started": started, "already_applied": not started,
|
||||
"action_id": action_id}
|
||||
|
||||
|
||||
def cmd_plan_revision_list(root: str | Path, *, store: Any = None) -> dict[str, Any]:
|
||||
"""G18:按 revision 列历史(新 revision 不覆盖旧 revision)。"""
|
||||
from app.services.bioflow.versioning import RevisionStore
|
||||
st = store or RevisionStore(root)
|
||||
revs = sorted(p.name for p in Path(root).iterdir()) if Path(root).is_dir() else []
|
||||
return {"ok": True, "data": {"revisions": revs}}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# run node-execute / stop / resume(G17/G20)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def cmd_run_node_execute(
|
||||
changed_stage: str, stage_chain: list[str] | None = None
|
||||
) -> dict[str, Any]:
|
||||
"""G17:只重跑受影响节点。"""
|
||||
from app.services.bioflow.versioning import affected_stages
|
||||
chain = list(stage_chain or [])
|
||||
affected = affected_stages(chain + [changed_stage] if changed_stage not in chain else chain,
|
||||
changed_stage)
|
||||
return {"ok": True, "data": {"nodes": affected}}
|
||||
|
||||
|
||||
def cmd_run_stop(run_id: str, owner_id: str) -> dict[str, Any]:
|
||||
"""G20:用户停止终止实际执行。"""
|
||||
from app.services.bioflow import run_owner
|
||||
return {"ok": run_owner.stop(run_id, owner_id), "run_id": run_id, "owner_id": owner_id}
|
||||
|
||||
|
||||
def cmd_run_resume(run_id: str, owner_id: str) -> dict[str, Any]:
|
||||
"""G20:恢复不与旧进程重叠。"""
|
||||
from app.services.bioflow import run_owner
|
||||
return {"ok": run_owner.resume(run_id, owner_id), "run_id": run_id, "owner_id": owner_id}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# delivery validate / publish(G21)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _load_manifest(manifest: Any) -> dict[str, Any]:
|
||||
"""接受 dict 或 JSON 文件路径 → manifest dict。"""
|
||||
if isinstance(manifest, dict):
|
||||
return manifest
|
||||
import json
|
||||
return json.loads(Path(manifest).read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def cmd_delivery_validate(manifest: Any) -> dict[str, Any]:
|
||||
"""G21:校验交付清单完整性(八项)。"""
|
||||
from app.services.bioflow.delivery import delivery_validate
|
||||
return delivery_validate(_load_manifest(manifest))
|
||||
|
||||
|
||||
def cmd_delivery_publish(manifest: Any) -> dict[str, Any]:
|
||||
"""G21:发布交付清单(校验通过置 approved)。"""
|
||||
from app.services.bioflow.delivery import publish_delivery
|
||||
return publish_delivery(_load_manifest(manifest))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# plan validate(G12:读单一 schema 校验参数)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def cmd_validate_plan(stage: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""G12:plan validate——读单一参数 schema 校验阶段参数(未知参数/choice 越界拒绝)。"""
|
||||
from app.services.bioflow.catalog import stage_schema
|
||||
|
||||
params = dict(params or {})
|
||||
schema = stage_schema(stage)
|
||||
schema_params = schema["params"]
|
||||
known = {p["name"] for p in schema_params}
|
||||
issues: list[str] = []
|
||||
for k in params:
|
||||
if k not in known:
|
||||
issues.append(f"未知参数: {k}")
|
||||
for p in schema_params:
|
||||
name = p["name"]
|
||||
if name in params and p.get("type") == "choice":
|
||||
if params[name] not in p.get("choices", []):
|
||||
issues.append(f"参数 {name}={params[name]!r} 不在可选 {p['choices']}")
|
||||
if issues:
|
||||
return _err("VALIDATION_FAILED", "; ".join(issues), evidence_refs=issues)
|
||||
return {"ok": True, "data": {"stage": stage, "valid": True}}
|
||||
Executable
+20
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env bash
|
||||
# bioflow_service/install_skill.sh —— 安装 bioflow-analysis skill(G12)。
|
||||
#
|
||||
# 用法:bash install_skill.sh [skills_dir]
|
||||
# - skills_dir 缺省为仓库根 .claude/skills(Claude Code 项目级 skill 目录)
|
||||
# - 复制 bioflow_service/SKILL.md → skills_dir/bioflow-analysis/SKILL.md
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
SKILL_SRC="$HERE/SKILL.md"
|
||||
SKILL_NAME="bioflow-analysis"
|
||||
SKILLS_DIR="${1:-$HERE/../.claude/skills}"
|
||||
|
||||
[ -f "$SKILL_SRC" ] || { echo "缺 SKILL.md: $SKILL_SRC" >&2; exit 2; }
|
||||
|
||||
DEST="$SKILLS_DIR/$SKILL_NAME"
|
||||
mkdir -p "$DEST"
|
||||
cp "$SKILL_SRC" "$DEST/SKILL.md"
|
||||
echo "[install] $SKILL_NAME -> $DEST/SKILL.md"
|
||||
echo "DONE"
|
||||
@@ -0,0 +1,883 @@
|
||||
"""tests/test_bioflowctl_cli.py —— Phase A CLI(bioflowctl)测试用例集。
|
||||
|
||||
覆盖(Phase A 验收):
|
||||
- CLI 命令(bioflowctl/commands.py 子命令 + 其服务层单一事实源):
|
||||
source fetch / file inspect / file transform / asset register / asset validate /
|
||||
report register / archive / archive reuse
|
||||
- M0 六条 gap(G1-G6)正反例。
|
||||
|
||||
命令 → 实现映射:
|
||||
source fetch → source_fetcher.fetch_source(commands.cmd_source_fetch 薄封装)
|
||||
file inspect → file_transform.inspect_file
|
||||
file transform→ file_transform.transform_file
|
||||
asset register→ commands.register_asset(G2 原子发布 + 同名冲突)
|
||||
asset validate→ commands.validate_asset(角色→格式 → 校验器)
|
||||
report register→ commands.cmd_report_register(G3 空路径拒绝)
|
||||
archive → commands.cmd_archive + archive.archive_from_job(G1/G6)
|
||||
archive reuse → commands.cmd_archive_reuse(G6 读生效参数)
|
||||
|
||||
每条用例三段式(前置 / 步骤 / 预期)写在 docstring。分层标记 -m bioflow_unit。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import types as _types
|
||||
from pathlib import Path
|
||||
|
||||
import httpx as _real_httpx
|
||||
import pytest
|
||||
|
||||
from app.db import Database
|
||||
from app.services.bioflow import store
|
||||
from app.services.bioflow.archive import (
|
||||
archive_from_job,
|
||||
effective_stage_params,
|
||||
normalize_archive_name,
|
||||
)
|
||||
from app.services.bioflow.file_transform import (
|
||||
TransformError,
|
||||
inspect_file,
|
||||
transform_file,
|
||||
validate_format,
|
||||
)
|
||||
from app.services.bioflow.params import archive_param_prefill
|
||||
from app.services.bioflow.reports import (
|
||||
extract_figure_refs,
|
||||
locate_report,
|
||||
validate_report_path,
|
||||
)
|
||||
from app.services.bioflow.runner import (
|
||||
STAGE_DONE,
|
||||
STAGE_WAITING_CONFIRM,
|
||||
)
|
||||
from app.services.bioflow.source_fetcher import (
|
||||
FetchError,
|
||||
_resolve_oss_url,
|
||||
check_disk_space,
|
||||
fetch_lan,
|
||||
fetch_source,
|
||||
fetch_web,
|
||||
)
|
||||
from bioflowctl import commands as cli
|
||||
|
||||
pytestmark = pytest.mark.bioflow_unit
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 辅助
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _run(coro):
|
||||
"""同步跑协程(新建 event loop;与 docgen 测试同款可靠模式)。"""
|
||||
return asyncio.new_event_loop().run_until_complete(coro)
|
||||
|
||||
|
||||
def _scenario(coro_factory):
|
||||
"""单 loop 跑完整 runner 场景(runner 内部 create_task 派生任务必须同 loop)。"""
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
return loop.run_until_complete(coro_factory())
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
|
||||
async def _drain(runner, seconds: float = 5.0) -> None:
|
||||
"""等待 runner 派生的后台任务全部结束。"""
|
||||
while runner._tasks:
|
||||
await asyncio.wait_for(asyncio.gather(*list(runner._tasks)), timeout=seconds)
|
||||
await asyncio.sleep(0.01)
|
||||
|
||||
|
||||
def _fake_httpx_stream(chunks, monkeypatch, sf) -> None:
|
||||
"""注入 fake httpx:stream→aiter_bytes 逐 chunk 产出(确定性,不依赖外网)。"""
|
||||
total = sum(len(c) for c in chunks)
|
||||
|
||||
class _Resp:
|
||||
status_code = 200
|
||||
headers = {"content-length": str(total)}
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
async def aiter_bytes(self, chunk_size):
|
||||
for c in chunks:
|
||||
yield c
|
||||
|
||||
class _Stream:
|
||||
async def __aenter__(self):
|
||||
return _Resp()
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
class _Client:
|
||||
def __init__(self, follow_redirects=True):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
def stream(self, method, url, timeout=None):
|
||||
return _Stream()
|
||||
|
||||
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
|
||||
AsyncClient=_Client, Timeout=_real_httpx.Timeout, HTTPError=_real_httpx.HTTPError,
|
||||
))
|
||||
|
||||
|
||||
def _make_job_dir(tmp_path, with_report=True):
|
||||
job_dir = tmp_path / "job" / "j1"
|
||||
(job_dir / "workspace" / "final").mkdir(parents=True)
|
||||
(job_dir / "logs").mkdir(parents=True)
|
||||
(job_dir / "input").mkdir(parents=True)
|
||||
(job_dir / "workspace" / "final" / "Ft.SPL.final_members.id").write_text(
|
||||
"gene1\ngene2\n", encoding="utf-8")
|
||||
(job_dir / "logs" / "identify.log").write_text("log line", encoding="utf-8")
|
||||
(job_dir / "input" / "genome.pep.fa").write_text(">x\nACGT", encoding="utf-8")
|
||||
(job_dir / "events.jsonl").write_text(json.dumps({"type": "job_created"}), encoding="utf-8")
|
||||
if with_report:
|
||||
(job_dir / "workspace" / "final" / "Ft.SPL.gene_family_report.html").write_text(
|
||||
'<html><body><img src="fig://heatmap_1"/><img src="fig://tree_2"/></body></html>',
|
||||
encoding="utf-8")
|
||||
return job_dir
|
||||
|
||||
|
||||
async def _seed_archives(db_path, *archives):
|
||||
"""在独立 db 文件上插入存档行(供 cmd_archive_reuse 注入 db_path)。"""
|
||||
db = Database(db_path)
|
||||
await db.connect()
|
||||
try:
|
||||
for a in archives:
|
||||
await store.insert_archive(db, a)
|
||||
finally:
|
||||
await db.close()
|
||||
|
||||
|
||||
def _archive_row(**over):
|
||||
base = {
|
||||
"id": "a1", "job_id": "j1", "name": "SPL鉴定", "description": "",
|
||||
"stage_chain": ["identify", "phylogeny"],
|
||||
"params": {"identify": {"cpu": "8"}},
|
||||
"report_artifact": None, "disk_path": "/tmp/a1", "size_bytes": 10,
|
||||
}
|
||||
base.update(over)
|
||||
return base
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 1. source fetch(命令: bioflowctl source fetch → source_fetcher.fetch_source)
|
||||
# ===========================================================================
|
||||
|
||||
def test_cli_source_fetch_web_ok(tmp_path, monkeypatch):
|
||||
"""前置:fake 流式 http 响应产出 2 chunk。步骤:fetch_web。
|
||||
预期:ok=True,落盘 + provenance sidecar,size 正确。"""
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
payload = [b">seq1\nACGT\n", b">seq2\nTGCA\n"]
|
||||
_fake_httpx_stream(payload, monkeypatch, sf)
|
||||
r = _run(fetch_web("http://x/data.fa", tmp_path / "out"))
|
||||
assert r["ok"] is True
|
||||
assert r["name"] == "data.fa"
|
||||
assert r["size"] == sum(len(c) for c in payload)
|
||||
assert r["kind"] == "web"
|
||||
assert len(r["provenance"]["sha256"]) == 64
|
||||
assert (tmp_path / "out" / "data.fa.provenance.json").is_file()
|
||||
|
||||
|
||||
def test_cli_source_fetch_bad_scheme(tmp_path):
|
||||
"""前置:ftp:// 源。步骤:fetch_source。预期:FetchError BAD_SCHEME。"""
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_source("ftp://host/x.fa", tmp_path))
|
||||
assert ei.value.code == "BAD_SCHEME"
|
||||
|
||||
|
||||
def test_cli_source_fetch_empty(tmp_path):
|
||||
"""前置:空源。步骤:fetch_source("")。预期:FetchError EMPTY_SOURCE。"""
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_source("", tmp_path))
|
||||
assert ei.value.code == "EMPTY_SOURCE"
|
||||
|
||||
|
||||
def test_cli_source_fetch_lan_local_ok(tmp_path):
|
||||
"""前置:本机存在文件。步骤:fetch_lan(局域网本机路径直读)。
|
||||
预期:ok=True,kind=lan,name/size/provenance 正确。"""
|
||||
f = tmp_path / "genome.fa"
|
||||
f.write_text(">g1\nACGT\n")
|
||||
r = _run(fetch_lan(str(f), tmp_path / "out"))
|
||||
assert r["ok"] is True
|
||||
assert r["kind"] == "lan"
|
||||
assert r["path"] == str(f)
|
||||
assert r["provenance"]["sha256"]
|
||||
|
||||
|
||||
def test_cli_source_fetch_lan_missing(tmp_path):
|
||||
"""前置:路径不存在。步骤:fetch_lan。预期:FetchError PATH_NOT_EXIST。"""
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("/no/such/path.fa", tmp_path))
|
||||
assert ei.value.code == "PATH_NOT_EXIST"
|
||||
|
||||
|
||||
def test_cli_source_fetch_lan_dir(tmp_path):
|
||||
"""前置:目录。步骤:fetch_lan。预期:FetchError IS_DIR(仅单文件)。"""
|
||||
d = tmp_path / "adir"
|
||||
d.mkdir()
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan(str(d), tmp_path / "out"))
|
||||
assert ei.value.code == "IS_DIR"
|
||||
|
||||
|
||||
def test_cli_source_fetch_oss_unconfigured():
|
||||
"""前置:无 BIOFLOW_OSS_ENDPOINT。步骤:_resolve_oss_url(oss://)。
|
||||
预期:FetchError OSS_UNCONFIGURED。"""
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_resolve_oss_url("oss://bucket/key.fa")
|
||||
assert ei.value.code == "OSS_UNCONFIGURED"
|
||||
|
||||
|
||||
def test_cli_source_fetch_disk_full(tmp_path):
|
||||
"""前置:磁盘余量不足(1EB)。步骤:check_disk_space。预期:ok=False。"""
|
||||
ok, free = check_disk_space(tmp_path, 10 ** 18)
|
||||
assert ok is False
|
||||
assert free < 10 ** 18
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 2. file inspect(命令: bioflowctl file inspect → file_transform.inspect_file)
|
||||
# ===========================================================================
|
||||
|
||||
def test_cli_file_inspect_fasta(tmp_path):
|
||||
"""前置:fasta 文件。步骤:inspect_file。预期:ok,format=fasta,size>0。"""
|
||||
f = tmp_path / "proteins.fa"
|
||||
f.write_text(">SPL1\nMKTA\n>SPL2\nMKTB\n")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "fasta"
|
||||
assert r["size"] > 0
|
||||
|
||||
|
||||
def test_cli_file_inspect_not_found(tmp_path):
|
||||
"""前置:文件不存在。步骤:inspect_file。预期:ok=False,code=NOT_FOUND。"""
|
||||
r = inspect_file(tmp_path / "nope.fa")
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_cli_file_inspect_tsv_columns(tmp_path):
|
||||
"""前置:tsv 文件。步骤:inspect_file。预期:format=tsv,列名提取正确。"""
|
||||
f = tmp_path / "samples.tsv"
|
||||
f.write_text("run\tfastq_1\tfastq_2\nSRR1\ta_1.fq\ta_2.fq\n")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "tsv"
|
||||
assert r["columns"] == ["run", "fastq_1", "fastq_2"]
|
||||
|
||||
|
||||
def test_cli_file_inspect_garbled(tmp_path):
|
||||
"""前置:含异常字节的文件。步骤:inspect_file。预期:garbled 比例 > 阈值。"""
|
||||
f = tmp_path / "messy.txt"
|
||||
f.write_bytes(b"\x00\x01\x02\x03\xff\xfe garbage \x00\x00")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["garbled"] > 0.3
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 3. file transform(命令: bioflowctl file transform → file_transform.transform_file)
|
||||
# ===========================================================================
|
||||
|
||||
def test_cli_file_transform_csv_to_fasta(tmp_path):
|
||||
"""前置:csv(gene,seq)。步骤:transform_file(fasta, 显式 column_map)。
|
||||
预期:ok,format=fasta,records=2。"""
|
||||
src = tmp_path / "genes.csv"
|
||||
src.write_text("gene,seq\nSPL1,MKTA\nSPL2,MKTB\n", encoding="utf-8")
|
||||
r = _run(transform_file(src, "fasta", column_map={"name": "gene", "seq": "seq"}))
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "fasta"
|
||||
assert r["records"] == 2
|
||||
|
||||
|
||||
def test_cli_file_transform_tsv_to_samples_yaml(tmp_path):
|
||||
"""前置:tsv(run,r1,r2)。步骤:transform_file(samples.yaml)。
|
||||
预期:ok,format=samples.yaml,records=2。"""
|
||||
src = tmp_path / "samples.tsv"
|
||||
src.write_text("run\tr1\tr2\nSRR1\ta_1.fq\ta_2.fq\nSRR2\tb_1.fq\tb_2.fq\n", encoding="utf-8")
|
||||
r = _run(transform_file(src, "samples.yaml", column_map={"name": "run", "r1": "r1", "r2": "r2"}))
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "samples.yaml"
|
||||
assert r["records"] == 2
|
||||
|
||||
|
||||
def test_cli_file_transform_gff3_unsupported(tmp_path):
|
||||
"""前置:tsv 源。步骤:transform_file(gff3)。预期:TransformError UNSUPPORTED。"""
|
||||
src = tmp_path / "a.tsv"
|
||||
src.write_text("a\tb\tc\n1\t2\t3\n", encoding="utf-8")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "gff3"))
|
||||
assert ei.value.code == "UNSUPPORTED"
|
||||
|
||||
|
||||
def test_cli_file_transform_not_found(tmp_path):
|
||||
"""前置:源不存在。步骤:transform_file。预期:TransformError NOT_FOUND。"""
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(tmp_path / "nope.csv", "fasta"))
|
||||
assert ei.value.code == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_cli_file_transform_bad_format(tmp_path):
|
||||
"""前置:csv 源。步骤:transform_file(bed)。预期:TransformError BAD_FORMAT。"""
|
||||
src = tmp_path / "a.csv"
|
||||
src.write_text("a,b\n1,2\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "bed"))
|
||||
assert ei.value.code == "BAD_FORMAT"
|
||||
|
||||
|
||||
def test_cli_file_transform_empty_table(tmp_path):
|
||||
"""前置:空文件。步骤:transform_file(fasta)。预期:TransformError EMPTY_TABLE。"""
|
||||
src = tmp_path / "empty.csv"
|
||||
src.write_text("", encoding="utf-8")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta"))
|
||||
assert ei.value.code == "EMPTY_TABLE"
|
||||
|
||||
|
||||
def test_cli_file_transform_missing_columns(tmp_path):
|
||||
"""前置:表头无 name/seq 列且无 LLM。步骤:transform_file(fasta)。
|
||||
预期:TransformError MISSING_COLUMNS。"""
|
||||
src = tmp_path / "x.csv"
|
||||
src.write_text("foo,bar\n1,2\n", encoding="utf-8")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta"))
|
||||
assert ei.value.code == "MISSING_COLUMNS"
|
||||
|
||||
|
||||
def test_cli_file_transform_validation_failed(tmp_path):
|
||||
"""前置:seq 列全空(写不出有效 fasta)。步骤:transform_file(fasta)。
|
||||
预期:TransformError VALIDATION_FAILED(产物删除,供追问)。"""
|
||||
src = tmp_path / "genes.csv"
|
||||
src.write_text("gene,seq\nSPL1,\nSPL2,\n", encoding="utf-8")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta", column_map={"name": "gene", "seq": "seq"}))
|
||||
assert ei.value.code == "VALIDATION_FAILED"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 4. asset register(命令: bioflowctl asset register → commands.register_asset)
|
||||
# ===========================================================================
|
||||
|
||||
def test_cli_asset_register_ok(tmp_path):
|
||||
"""前置:本机文件 + 合法角色。步骤:register_asset(role, path, assets_root)。
|
||||
预期:ok=True,落角色目录,role/name/size 正确。"""
|
||||
f = tmp_path / "genome.fa"
|
||||
f.write_text(">g1\nACGT\n")
|
||||
assets = tmp_path / "assets"
|
||||
r = cli.register_asset("pep.fa", str(f), assets_root=assets)
|
||||
assert r["ok"] is True
|
||||
assert r["role"] == "pep.fa"
|
||||
assert r["name"] == "genome.fa"
|
||||
assert (assets / "pep.fa" / "genome.fa").read_text() == ">g1\nACGT\n"
|
||||
|
||||
|
||||
def test_cli_asset_register_missing_source(tmp_path):
|
||||
"""前置:源不存在。步骤:register_asset。预期:ok=False,code=NOT_FOUND。"""
|
||||
r = cli.register_asset("pep.fa", str(tmp_path / "nope.fa"), assets_root=tmp_path / "assets")
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_cli_asset_register_dir_rejected(tmp_path):
|
||||
"""前置:目录。步骤:register_asset。预期:ok=False,code=NOT_FOUND(仅文件)。"""
|
||||
d = tmp_path / "dir"
|
||||
d.mkdir()
|
||||
r = cli.register_asset("pep.fa", str(d), assets_root=tmp_path / "assets")
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_cli_asset_register_bad_role(tmp_path):
|
||||
"""前置:角色名含路径分隔符或 ..。步骤:register_asset。预期:code=BAD_ROLE。"""
|
||||
f = tmp_path / "x.fa"
|
||||
f.write_text(">x\nACGT\n")
|
||||
for bad in ("", "../pep", "a/b", "a\\b"):
|
||||
r = cli.register_asset(bad, str(f), assets_root=tmp_path / "assets")
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "BAD_ROLE"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 5. asset validate(命令: bioflowctl asset validate → commands.validate_asset)
|
||||
# ===========================================================================
|
||||
|
||||
def test_cli_asset_validate_fasta_ok(tmp_path):
|
||||
"""前置:合法 fasta + 角色 pep.fa。步骤:validate_asset。预期:ok=True,format=fasta。"""
|
||||
f = tmp_path / "p.fa"
|
||||
f.write_text(">S1\nACGT\n")
|
||||
r = cli.validate_asset("pep.fa", str(f))
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "fasta"
|
||||
|
||||
|
||||
def test_cli_asset_validate_fasta_invalid(tmp_path):
|
||||
"""前置:无 '>' 头行文件 + 角色 pep.fa。步骤:validate_asset。预期:ok=False。"""
|
||||
f = tmp_path / "p.fa"
|
||||
f.write_text("just text\n")
|
||||
r = cli.validate_asset("pep.fa", str(f))
|
||||
assert r["ok"] is False
|
||||
|
||||
|
||||
def test_cli_asset_validate_samples_yaml_role(tmp_path):
|
||||
"""前置:合法 samples.yaml + 角色 samples.yaml。步骤:validate_asset。
|
||||
预期:ok=True,format=samples.yaml。"""
|
||||
f = tmp_path / "samples.yaml"
|
||||
f.write_text("samples:\n S1:\n r1: a.fq\n r2: b.fq\n")
|
||||
r = cli.validate_asset("samples.yaml", str(f))
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "samples.yaml"
|
||||
|
||||
|
||||
def test_cli_asset_validate_bad_role(tmp_path):
|
||||
"""前置:角色无法推断格式 + txt 文件。步骤:validate_asset。预期:code=BAD_ROLE。"""
|
||||
f = tmp_path / "x.txt"
|
||||
f.write_text("hello\n")
|
||||
r = cli.validate_asset("unknown_role", str(f))
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "BAD_ROLE"
|
||||
|
||||
|
||||
def test_cli_asset_validate_missing_file(tmp_path):
|
||||
"""前置:文件不存在。步骤:validate_asset。预期:code=NOT_FOUND。"""
|
||||
r = cli.validate_asset("pep.fa", str(tmp_path / "nope.fa"))
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "NOT_FOUND"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 6. report register(命令: bioflowctl report register → commands.cmd_report_register)
|
||||
# ===========================================================================
|
||||
|
||||
def test_cli_report_register_ok(tmp_path):
|
||||
"""前置:报告含 fig:// 引用。步骤:cmd_report_register。预期:ok=True,
|
||||
size/figures 正确。"""
|
||||
f = tmp_path / "r.html"
|
||||
f.write_text('<html><img src="fig://a"/></html>')
|
||||
r = cli.cmd_report_register(str(f))
|
||||
assert r["ok"] is True
|
||||
assert r["path"] == str(f)
|
||||
assert r["figures"] == ["a"]
|
||||
|
||||
|
||||
def test_cli_report_register_missing_file(tmp_path):
|
||||
"""前置:路径不存在。步骤:cmd_report_register。预期:ok=False,code=EMPTY_REPORT。"""
|
||||
r = cli.cmd_report_register(str(tmp_path / "nope.html"))
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "EMPTY_REPORT"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 7. archive(命令: bioflowctl archive → commands.cmd_archive + archive.archive_from_job)
|
||||
# ===========================================================================
|
||||
|
||||
def test_cli_archive_pack(tmp_path):
|
||||
"""前置:job_dir 含 final/logs/input/events。步骤:archive_from_job。
|
||||
预期:四类产物入归档;默认不含中间文件。"""
|
||||
job_dir = _make_job_dir(tmp_path)
|
||||
job = {"id": "j1", "title": "t", "stage_chain": ["identify"], "params": {}}
|
||||
result = _run(archive_from_job(job, job_dir, tmp_path / "archives", name="SPL 鉴定"))
|
||||
assert result.name == "SPL 鉴定"
|
||||
dest = tmp_path / "archives" / result.id
|
||||
assert (dest / "final" / "Ft.SPL.final_members.id").exists()
|
||||
assert (dest / "logs" / "identify.log").exists()
|
||||
assert (dest / "input" / "genome.pep.fa").exists()
|
||||
assert (dest / "events.jsonl").exists()
|
||||
|
||||
|
||||
def test_cli_archive_include_intermediate(tmp_path):
|
||||
"""前置:job_dir 含中间 analysis。步骤:archive_from_job(include_intermediate=True)。
|
||||
预期:中间文件入归档。"""
|
||||
job_dir = _make_job_dir(tmp_path)
|
||||
(job_dir / "workspace" / "identify" / "analysis").mkdir(parents=True)
|
||||
(job_dir / "workspace" / "identify" / "analysis" / "domtblout").write_text("intermediate")
|
||||
result = _run(archive_from_job(
|
||||
{"id": "j1", "title": "t", "stage_chain": [], "params": {}},
|
||||
job_dir, tmp_path / "archives", name="X", include_intermediate=True))
|
||||
assert (tmp_path / "archives" / result.id / "intermediate" / "identify" / "analysis" / "domtblout").exists()
|
||||
|
||||
|
||||
def test_cli_archive_normalize_name():
|
||||
"""前置:空/带空白名。步骤:normalize_archive_name。预期:trim / 空→占位。"""
|
||||
assert normalize_archive_name(" SPL 鉴定 ") == "SPL 鉴定"
|
||||
assert normalize_archive_name(" ") == "未命名存档"
|
||||
|
||||
|
||||
def test_cli_archive_cmd_not_found(tmp_path):
|
||||
"""前置:空 db。步骤:cmd_archive(不存在 job)。预期:ok=False,code=NOT_FOUND。"""
|
||||
r = _run(cli.cmd_archive(
|
||||
"nope", "X", db_path=tmp_path / "empty.db",
|
||||
jobs_dir=tmp_path / "jobs", archives_dir=tmp_path / "archives"))
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "NOT_FOUND"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 8. archive reuse(命令: bioflowctl archive reuse → commands.cmd_archive_reuse + store + params)
|
||||
# ===========================================================================
|
||||
|
||||
def test_cli_archive_reuse_find_exact(db):
|
||||
"""前置:插一条存档名 SPL鉴定。步骤:find_archive_by_name(精确)。预期:命中。"""
|
||||
_run(_insert_via(db, _archive_row()))
|
||||
found = _run(store.find_archive_by_name(db, "SPL鉴定"))
|
||||
assert found is not None and found["id"] == "a1"
|
||||
|
||||
|
||||
async def _insert_via(db, archive):
|
||||
await store.insert_archive(db, archive)
|
||||
|
||||
|
||||
def test_cli_archive_reuse_find_fuzzy(db):
|
||||
"""前置:存档名 SPL鉴定。步骤:find_archive_by_name(模糊 SPL)。预期:命中候选。"""
|
||||
_run(_insert_via(db, _archive_row()))
|
||||
found = _run(store.find_archive_by_name(db, "SPL"))
|
||||
assert found is not None and found["id"] == "a1"
|
||||
|
||||
|
||||
def test_cli_archive_reuse_candidates(db):
|
||||
"""前置:两条含 SPL 的存档。步骤:find_archive_candidates。预期:≥2 条。"""
|
||||
_run(_insert_via(db, _archive_row()))
|
||||
_run(_insert_via(db, _archive_row(id="a2", name="SPL另一条")))
|
||||
cands = _run(store.find_archive_candidates(db, "SPL"))
|
||||
assert len(cands) >= 2
|
||||
|
||||
|
||||
def test_cli_archive_reuse_prefill(db):
|
||||
"""前置:存档 params 含 identify。步骤:archive_param_prefill。
|
||||
预期:identify 预填命中;未知阶段空。"""
|
||||
_run(_insert_via(db, _archive_row()))
|
||||
found = _run(store.find_archive_by_name(db, "SPL鉴定"))
|
||||
assert archive_param_prefill(found, "identify") == {"cpu": "8"}
|
||||
assert archive_param_prefill(found, "nope") == {}
|
||||
|
||||
|
||||
def test_cli_archive_reuse_prefill_empty(db):
|
||||
"""前置:存档 params 为空。步骤:archive_param_prefill。预期:空 dict。"""
|
||||
_run(_insert_via(db, _archive_row(params={})))
|
||||
found = _run(store.find_archive_by_name(db, "SPL鉴定"))
|
||||
assert archive_param_prefill(found, "identify") == {}
|
||||
|
||||
|
||||
def test_cli_archive_reuse_cmd(tmp_path):
|
||||
"""前置:seed 一条存档(params 含生效参数 cpu=8)。步骤:cmd_archive_reuse。
|
||||
预期:ok=True,返回 stage_chain 与生效参数。"""
|
||||
db_path = tmp_path / "reuse.db"
|
||||
_run(_seed_archives(db_path, _archive_row(
|
||||
params={"identify": {"cpu": "8", "memory": "16G"}})))
|
||||
r = _run(cli.cmd_archive_reuse("SPL鉴定", db_path=db_path))
|
||||
assert r["ok"] is True
|
||||
assert r["data"]["stage_chain"] == ["identify", "phylogeny"]
|
||||
assert r["data"]["params"] == {"identify": {"cpu": "8", "memory": "16G"}}
|
||||
|
||||
|
||||
def test_cli_archive_reuse_cmd_not_found(tmp_path):
|
||||
"""前置:空 db。步骤:cmd_archive_reuse。预期:ok=False,code=ARCHIVE_NOT_FOUND。"""
|
||||
r = _run(cli.cmd_archive_reuse("不存在", db_path=tmp_path / "empty.db"))
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "ARCHIVE_NOT_FOUND"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# M0 六条 gap(G1-G6)
|
||||
# ===========================================================================
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# G1 归档按 manifest 必需产物复制(正例:非 final 的必需产物也归档;反例:漏归档)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _make_job_with_manifest_required_nonfinal(tmp_path):
|
||||
job_dir = tmp_path / "job" / "j1"
|
||||
(job_dir / "workspace" / "final").mkdir(parents=True)
|
||||
(job_dir / "workspace" / "identify" / "analysis").mkdir(parents=True)
|
||||
(job_dir / "logs").mkdir(parents=True)
|
||||
(job_dir / "input").mkdir(parents=True)
|
||||
(job_dir / "workspace" / "final" / "Ft.SPL.final_members.id").write_text("gene1\n")
|
||||
required = job_dir / "workspace" / "identify" / "analysis" / "domtblout"
|
||||
required.write_text("PF00001\tSPL1\n", encoding="utf-8")
|
||||
manifest = {
|
||||
"manifest_version": "1", "job_id": "j1", "stage": "identify",
|
||||
"status": "success", "prefix": "Ft.SPL",
|
||||
"artifacts": {
|
||||
"final_members": {"artifact": "final_members",
|
||||
"full_path": str(job_dir / "workspace" / "final" / "Ft.SPL.final_members.id"),
|
||||
"type": "table", "size": 10},
|
||||
"domtblout": {"artifact": "domtblout", "full_path": str(required),
|
||||
"type": "table", "size": 5},
|
||||
},
|
||||
"parameters": {}, "metrics": {},
|
||||
}
|
||||
(job_dir / "workspace" / "final" / "Ft.SPL.identify.manifest.json").write_text(
|
||||
json.dumps(manifest), encoding="utf-8")
|
||||
return job_dir, required
|
||||
|
||||
|
||||
def test_g1_archive_copies_manifest_required_nonfinal_artifact(tmp_path):
|
||||
"""G1 正例。前置:manifest 声明必需产物在 final/ 之外(workspace/identify/analysis)。
|
||||
步骤:archive_from_job(include_intermediate=False)。
|
||||
预期:该非 final 必需产物仍被复制进归档(按 manifest,而非仅 final/)。"""
|
||||
job_dir, _required = _make_job_with_manifest_required_nonfinal(tmp_path)
|
||||
job = {"id": "j1", "title": "t", "stage_chain": ["identify"], "params": {}}
|
||||
result = _run(archive_from_job(job, job_dir, tmp_path / "archives", name="X"))
|
||||
dest = tmp_path / "archives" / result.id
|
||||
assert any(p.name == "domtblout" for p in dest.rglob("*"))
|
||||
|
||||
|
||||
def test_g1_archive_does_not_miss_manifest_required_outside_jobdir(tmp_path):
|
||||
"""G1 反例(漏归档)。前置:manifest 必需产物 full_path 指向 job_dir 之外(共享库文件)。
|
||||
步骤:archive_from_job。预期:该必需产物被复制进归档,不遗漏。"""
|
||||
job_dir = tmp_path / "job" / "j1"
|
||||
(job_dir / "workspace" / "final").mkdir(parents=True)
|
||||
(job_dir / "logs").mkdir(parents=True)
|
||||
(job_dir / "input").mkdir(parents=True)
|
||||
(job_dir / "workspace" / "final" / "Ft.SPL.final_members.id").write_text("gene1\n")
|
||||
shared = tmp_path / "shared" / "Pfam-A.hmm"
|
||||
shared.parent.mkdir(parents=True)
|
||||
shared.write_text("HMMER3/f\nNAME Pfam-A\n", encoding="utf-8")
|
||||
manifest = {
|
||||
"manifest_version": "1", "job_id": "j1", "stage": "identify",
|
||||
"status": "success", "prefix": "Ft.SPL",
|
||||
"artifacts": {"hmm_db": {"artifact": "hmm_db", "full_path": str(shared),
|
||||
"type": "db", "size": 20}},
|
||||
"parameters": {}, "metrics": {},
|
||||
}
|
||||
(job_dir / "workspace" / "final" / "Ft.SPL.identify.manifest.json").write_text(
|
||||
json.dumps(manifest), encoding="utf-8")
|
||||
result = _run(archive_from_job(
|
||||
{"id": "j1", "title": "t", "stage_chain": ["identify"], "params": {}},
|
||||
job_dir, tmp_path / "archives", name="X"))
|
||||
dest = tmp_path / "archives" / result.id
|
||||
assert any(p.name == "Pfam-A.hmm" for p in dest.rglob("*"))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# G2 同名上传不覆盖、显式报错(正例:同名冲突报错不覆盖;反例:静默覆盖)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_g2_asset_register_same_name_errors_and_preserves(tmp_path):
|
||||
"""G2 正例。前置:同角色同文件名已发布。步骤:再次 register_asset 同名。
|
||||
预期:显式报错 ASSET_EXISTS,原资产不被覆盖。"""
|
||||
f = tmp_path / "x.fa"
|
||||
f.write_text("V1")
|
||||
assets = tmp_path / "assets"
|
||||
assert cli.register_asset("pep.fa", str(f), assets_root=assets)["ok"] is True
|
||||
r = cli.register_asset("pep.fa", str(f), assets_root=assets)
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "ASSET_EXISTS"
|
||||
assert (assets / "pep.fa" / "x.fa").read_text() == "V1"
|
||||
|
||||
|
||||
def test_g2_asset_register_never_silently_overwrites(tmp_path):
|
||||
"""G2 反例(静默覆盖)。前置:同名资产已存在,第二个同名但内容不同的源。
|
||||
步骤:register_asset。预期:绝不静默覆盖,已发布资产内容仍为第一版。"""
|
||||
d1 = tmp_path / "d1"
|
||||
d2 = tmp_path / "d2"
|
||||
d1.mkdir()
|
||||
d2.mkdir()
|
||||
(d1 / "x.fa").write_text("VERSION1")
|
||||
(d2 / "x.fa").write_text("VERSION2")
|
||||
assets = tmp_path / "assets"
|
||||
assert cli.register_asset("pep.fa", str(d1 / "x.fa"), assets_root=assets)["ok"] is True
|
||||
r = cli.register_asset("pep.fa", str(d2 / "x.fa"), assets_root=assets)
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "ASSET_EXISTS"
|
||||
assert (assets / "pep.fa" / "x.fa").read_text() == "VERSION1"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# G3 空报告路径被拒绝(空串、空文件、None)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_g3_empty_report_path_rejected():
|
||||
"""G3 空串。前置:无。步骤:validate_report_path("")。预期:(False, 报告路径为空)。"""
|
||||
ok, err = validate_report_path("")
|
||||
assert ok is False
|
||||
assert err == "报告路径为空"
|
||||
|
||||
|
||||
def test_g3_empty_file_report_path_rejected(tmp_path):
|
||||
"""G3 空文件。前置:0 字节报告文件。步骤:validate_report_path(empty)。
|
||||
预期:(False, 报告文件为空)。"""
|
||||
empty = tmp_path / "empty.html"
|
||||
empty.write_text("", encoding="utf-8")
|
||||
ok, err = validate_report_path(empty)
|
||||
assert ok is False
|
||||
assert "报告文件为空" in err
|
||||
|
||||
|
||||
def test_g3_none_report_path_rejected():
|
||||
"""G3 None。前置:无。步骤:validate_report_path(None)。预期:(False, 报告路径为空)。"""
|
||||
ok, err = validate_report_path(None)
|
||||
assert ok is False
|
||||
assert err == "报告路径为空"
|
||||
|
||||
|
||||
def test_g3_cli_report_register_empty_rejected():
|
||||
"""G3 CLI 层。前置:无。步骤:cmd_report_register("")。预期:ok=False,code=EMPTY_REPORT。"""
|
||||
r = cli.cmd_report_register("")
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "EMPTY_REPORT"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# G4 partial_success 独立终态(不被当作 success)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_g4_partial_success_independent_terminal_state(db, events, catalog):
|
||||
"""G4。前置:executor 写 status=partial_success 的 manifest 且退出码 0。
|
||||
步骤:run_job(auto_continue=True)。预期:阶段不得被当作 success(done),
|
||||
而是独立终态 partial_success。"""
|
||||
from conftest import FakeExecutor, make_runner
|
||||
executor = FakeExecutor(exit_code=0, manifest_status="partial_success")
|
||||
|
||||
async def scenario():
|
||||
runner = make_runner(db, events, executor, catalog)
|
||||
record = await runner.create_job(
|
||||
"s1", "t", ["identify"], auto_continue=True, auto_start=False)
|
||||
await runner.run_job(record)
|
||||
await _drain(runner)
|
||||
return runner, record
|
||||
|
||||
runner, record = _scenario(scenario)
|
||||
status = runner.stage_status(record, "identify")
|
||||
assert status != STAGE_DONE # 不被当作 success
|
||||
assert status == "partial_success" # 独立终态
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# G5 大文件流式(不整块进内存的观测方式)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_g5_large_download_streams_chunked(tmp_path, monkeypatch):
|
||||
"""G5 正常路径(观测方式)。前置:fake 流式响应产出多 chunk。
|
||||
步骤:fetch_web。预期:观测到 aiter_bytes 以 CHUNK(=1MB) 分块迭代、逐块落盘。"""
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
observed: dict = {}
|
||||
chunks = [b"x" * 1024, b"y" * 2048, b"z" * 512]
|
||||
total = sum(len(c) for c in chunks)
|
||||
|
||||
class _Resp:
|
||||
status_code = 200
|
||||
headers = {"content-length": str(total)}
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
async def aiter_bytes(self, chunk_size):
|
||||
observed["chunk_size"] = chunk_size
|
||||
for c in chunks:
|
||||
yield c
|
||||
|
||||
class _Stream:
|
||||
async def __aenter__(self):
|
||||
return _Resp()
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
class _Client:
|
||||
def __init__(self, follow_redirects=True):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
def stream(self, method, url, timeout=None):
|
||||
return _Stream()
|
||||
|
||||
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
|
||||
AsyncClient=_Client, Timeout=_real_httpx.Timeout, HTTPError=_real_httpx.HTTPError,
|
||||
))
|
||||
r = _run(fetch_web("http://x/big.fa", tmp_path / "out"))
|
||||
assert r["size"] == total
|
||||
assert observed["chunk_size"] == sf.CHUNK # 1MB 分块,非整块
|
||||
assert (tmp_path / "out" / "big.fa").stat().st_size == total
|
||||
|
||||
|
||||
def test_g5_large_file_inspect_uses_preview_not_whole_read(tmp_path, monkeypatch):
|
||||
"""G5 异常路径(观测方式)。前置:>4MB 大文件。步骤:inspect_file(monkeypatch
|
||||
_read_text 使其被调用即失败)。预期:inspect 走 _read_text_preview(只读头)。"""
|
||||
from app.services.bioflow import file_transform as ft
|
||||
big = tmp_path / "big.fa"
|
||||
big.write_bytes(b">s\n" + b"ACGT" * 1_200_000 + b"\n") # ≈4.8MB
|
||||
|
||||
def _boom(*a, **k):
|
||||
raise AssertionError("大文件不应整读 _read_text")
|
||||
|
||||
monkeypatch.setattr(ft, "_read_text", _boom)
|
||||
r = ft.inspect_file(big)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "fasta"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# G6 归档复用读实际生效参数(非创建时 record.params)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_g6_effective_stage_params_returns_effective():
|
||||
"""G6 正例。前置:stage_states 含 confirm/rerun 后的生效参数。
|
||||
步骤:effective_stage_params(stage_chain, stage_states)。
|
||||
预期:返回各阶段生效参数;缺失阶段为空 dict。"""
|
||||
stage_chain = ["identify", "phylogeny", "missing"]
|
||||
stage_states = {
|
||||
"identify": {"params": {"cpu": "8", "memory": "16G"}},
|
||||
"phylogeny": {"params": {"method": "ML"}},
|
||||
}
|
||||
eff = effective_stage_params(stage_chain, stage_states)
|
||||
assert eff == {
|
||||
"identify": {"cpu": "8", "memory": "16G"},
|
||||
"phylogeny": {"method": "ML"},
|
||||
"missing": {},
|
||||
}
|
||||
|
||||
|
||||
def test_g6_archive_reuse_cmd_reads_effective_params(tmp_path):
|
||||
"""G6 正例(读侧端到端)。前置:seed 存档 params=生效参数(cpu=8)。
|
||||
步骤:cmd_archive_reuse。预期:返回生效参数 cpu=8(非创建时 record.params 的 cpu=2)。"""
|
||||
db_path = tmp_path / "reuse.db"
|
||||
_run(_seed_archives(db_path, _archive_row(
|
||||
params={"identify": {"cpu": "8", "memory": "16G"}})))
|
||||
r = _run(cli.cmd_archive_reuse("SPL鉴定", db_path=db_path))
|
||||
assert r["ok"] is True
|
||||
assert r["data"]["params"]["identify"]["cpu"] == "8"
|
||||
|
||||
|
||||
def test_g6_effective_params_diverge_from_creation_record(db, events, catalog):
|
||||
"""G6 反例。前置:创建任务 params cpu=2(record.params),确认时覆盖 cpu=8。
|
||||
步骤:confirm_stage 覆盖后观察两处 params。
|
||||
预期:生效参数(_stage_states)与创建时 record.params 分叉,归档必须快照前者。"""
|
||||
from conftest import FakeExecutor, make_runner
|
||||
executor = FakeExecutor()
|
||||
|
||||
async def scenario():
|
||||
runner = make_runner(db, events, executor, catalog)
|
||||
record = await runner.create_job(
|
||||
"s1", "t", ["identify"], params={"identify": {"cpu": "2"}},
|
||||
auto_continue=False, auto_start=False)
|
||||
await runner.run_job(record) # 手动模式停 waiting_confirm
|
||||
await runner.confirm_stage(record, "identify", {"cpu": "8"})
|
||||
await _drain(runner)
|
||||
return runner, record
|
||||
|
||||
runner, record = _scenario(scenario)
|
||||
effective = record._stage_states["identify"]["params"]
|
||||
assert effective.get("cpu") == "8" # 实际生效参数
|
||||
assert record.params["identify"].get("cpu") == "2" # 创建时 record.params
|
||||
assert effective != record.params["identify"] # 已分叉 → 归档读 effective
|
||||
@@ -0,0 +1,403 @@
|
||||
"""tests/test_file_transform.py —— 非标准文件预处理(探测 + 转换 + 校验)。
|
||||
|
||||
分层标记 -m bioflow_unit。覆盖(S0 验收):
|
||||
- inspect_file:fasta/tsv 探测、乱码检测、列结构提取。
|
||||
- transform_file:xlsx→fasta(显式列映射)、csv→fasta(启发式列映射)、
|
||||
samples.yaml 转换 + 校验。
|
||||
- 校验失败 → TransformError VALIDATION_FAILED(追问循环依据)。
|
||||
- 缺列映射且无 LLM → MISSING_COLUMNS。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.bioflow.file_transform import (
|
||||
TransformError,
|
||||
_garbled_ratio,
|
||||
inspect_file,
|
||||
transform_file,
|
||||
validate_fasta,
|
||||
validate_format,
|
||||
validate_gff3,
|
||||
validate_samples_yaml,
|
||||
validate_tsv,
|
||||
)
|
||||
|
||||
|
||||
def _run(coro):
|
||||
return asyncio.new_event_loop().run_until_complete(coro)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# inspect_file 探测
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_inspect_file_fasta(tmp_path):
|
||||
f = tmp_path / "proteins.fa"
|
||||
f.write_text(">SPL1\nMKTA\n>SPL2\nMKTB\n")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "fasta"
|
||||
assert r["size"] > 0
|
||||
|
||||
|
||||
def test_inspect_file_tsv_columns(tmp_path):
|
||||
f = tmp_path / "samples.tsv"
|
||||
f.write_text("run\tfastq_1\tfastq_2\nSRR1\ta_1.fq\ta_2.fq\n")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "tsv"
|
||||
assert r["columns"] == ["run", "fastq_1", "fastq_2"]
|
||||
|
||||
|
||||
def test_inspect_file_garbled(tmp_path):
|
||||
f = tmp_path / "messy.txt"
|
||||
f.write_bytes(b"\x00\x01\x02\x03\xff\xfe garbage \x00\x00")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["garbled"] > 0.3
|
||||
|
||||
|
||||
def test_garbled_ratio_clean():
|
||||
assert _garbled_ratio("ATCGATCG\n>seq\n") < 0.01
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# transform_file
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_transform_xlsx_to_fasta(tmp_path):
|
||||
import openpyxl
|
||||
src = tmp_path / "spl_list.xlsx"
|
||||
wb = openpyxl.Workbook()
|
||||
ws = wb.active
|
||||
ws.append(["基因名", "序列", "备注"])
|
||||
ws.append(["SPL1", "ATGGCTA", "水稻"])
|
||||
ws.append(["SPL2", "ATGGCTB", "水稻"])
|
||||
wb.save(src)
|
||||
|
||||
r = _run(transform_file(src, "fasta", column_map={"name": "基因名", "seq": "序列"}))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
assert r["validated"]["ok"] is True
|
||||
text = Path(r["path"]).read_text()
|
||||
assert ">SPL1" in text and "ATGGCTA" in text
|
||||
|
||||
|
||||
def test_transform_csv_to_fasta_heuristic(tmp_path):
|
||||
src = tmp_path / "genes.csv"
|
||||
src.write_text("gene_id,protein_sequence\nG1,MKTA\nG2,MKTB\n")
|
||||
r = _run(transform_file(src, "fasta"))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
|
||||
|
||||
def test_transform_csv_to_samples_yaml(tmp_path):
|
||||
src = tmp_path / "design.csv"
|
||||
src.write_text("sample,fastq_1,fastq_2,condition\nA,a_1.fq,a_2.fq,ctrl\nB,b_1.fq,b_2.fq,treated\n")
|
||||
r = _run(transform_file(src, "samples.yaml", column_map={"name": "sample", "r1": "fastq_1", "r2": "fastq_2"}))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
assert r["validated"]["ok"] is True
|
||||
import yaml
|
||||
samples = yaml.safe_load(Path(r["path"]).read_text())["samples"]
|
||||
assert "A" in samples and samples["A"]["r1"] == "a_1.fq"
|
||||
|
||||
|
||||
def test_transform_validation_failed(tmp_path):
|
||||
# seq 列全空 → 产物无序列 → 校验失败
|
||||
src = tmp_path / "bad.csv"
|
||||
src.write_text("name,seq\nG1,\nG2,\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta", column_map={"name": "name", "seq": "seq"}))
|
||||
assert ei.value.code == "VALIDATION_FAILED"
|
||||
|
||||
|
||||
def test_transform_missing_columns(tmp_path):
|
||||
# 无列映射、无 LLM、表头无启发式命中 → MISSING_COLUMNS
|
||||
src = tmp_path / "unknown.csv"
|
||||
src.write_text("col_a,col_b\n1,2\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta"))
|
||||
assert ei.value.code == "MISSING_COLUMNS"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 校验器
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_validate_fasta(tmp_path):
|
||||
f = tmp_path / "ok.fa"
|
||||
f.write_text(">a\nACGT\n")
|
||||
assert validate_fasta(f)["ok"] is True
|
||||
|
||||
|
||||
def test_validate_fasta_empty(tmp_path):
|
||||
f = tmp_path / "bad.fa"
|
||||
f.write_text("no records here\n")
|
||||
assert validate_fasta(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_tsv_inconsistent(tmp_path):
|
||||
f = tmp_path / "bad.tsv"
|
||||
f.write_text("a\tb\tc\n1\t2\n")
|
||||
assert validate_tsv(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_samples_yaml(tmp_path):
|
||||
f = tmp_path / "s.yaml"
|
||||
f.write_text("samples:\n A:\n r1: a.fq\n r2: a2.fq\n")
|
||||
assert validate_samples_yaml(f)["ok"] is True
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强(1)inspect_file:NOT_FOUND / xlsx 列 / csv 格式 / 大文件 / 内容嗅探
|
||||
# ===========================================================================
|
||||
|
||||
def test_inspect_file_not_found():
|
||||
r = inspect_file("/no/such/file.fa")
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_inspect_file_xlsx_columns(tmp_path):
|
||||
import openpyxl
|
||||
f = tmp_path / "spl.xlsx"
|
||||
wb = openpyxl.Workbook()
|
||||
ws = wb.active
|
||||
ws.append(["基因名", "序列"])
|
||||
ws.append(["SPL1", "MKTA"])
|
||||
wb.save(f)
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "xlsx"
|
||||
assert r["columns"] == ["基因名", "序列"]
|
||||
|
||||
|
||||
def test_inspect_file_csv_format(tmp_path):
|
||||
f = tmp_path / "genes.csv"
|
||||
f.write_text("id,seq\nG1,MKTA\n")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "csv"
|
||||
assert r["columns"] == ["id", "seq"]
|
||||
|
||||
|
||||
def test_inspect_file_large_uses_preview(tmp_path):
|
||||
f = tmp_path / "big.fa"
|
||||
f.write_bytes(b">seq\nACGT\n" + b"A" * (4 * 1024 * 1024))
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "fasta"
|
||||
|
||||
|
||||
def test_guess_format_content_sniffing_fasta(tmp_path):
|
||||
from app.services.bioflow.file_transform import _guess_format
|
||||
assert _guess_format("unknown.dat", b">seq1\nACGT\n") == "fasta"
|
||||
|
||||
|
||||
def test_guess_format_content_sniffing_tsv(tmp_path):
|
||||
from app.services.bioflow.file_transform import _guess_format
|
||||
assert _guess_format("unknown.dat", b"a\tb\tc\n1\t2\t3\n") == "tsv"
|
||||
|
||||
|
||||
def test_garbled_ratio_empty():
|
||||
assert _garbled_ratio("") == 0.0
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强(2)transform_file:NOT_FOUND / BAD_FORMAT / EMPTY_TABLE / tsv / gff3 /
|
||||
# samples.yaml 启发式 / LLM 兜底 / LLM 异常兜底
|
||||
# ===========================================================================
|
||||
|
||||
def test_transform_not_found(tmp_path):
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(tmp_path / "nope.csv", "fasta"))
|
||||
assert ei.value.code == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_transform_bad_format(tmp_path):
|
||||
src = tmp_path / "x.csv"
|
||||
src.write_text("a,b\n1,2\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "bogus"))
|
||||
assert ei.value.code == "BAD_FORMAT"
|
||||
|
||||
|
||||
def test_transform_empty_table(tmp_path):
|
||||
src = tmp_path / "empty.csv"
|
||||
src.write_text("")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta"))
|
||||
assert ei.value.code == "EMPTY_TABLE"
|
||||
|
||||
|
||||
def test_transform_to_tsv(tmp_path):
|
||||
src = tmp_path / "genes.csv"
|
||||
src.write_text("gene_id,protein_sequence\nG1,MKTA\nG2,MKTB\n")
|
||||
r = _run(transform_file(src, "tsv"))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
assert r["validated"]["ok"] is True
|
||||
|
||||
|
||||
def test_transform_to_gff3_unsupported(tmp_path):
|
||||
src = tmp_path / "x.csv"
|
||||
src.write_text("a,b\n1,2\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "gff3"))
|
||||
assert ei.value.code == "UNSUPPORTED"
|
||||
|
||||
|
||||
def test_transform_samples_yaml_heuristic(tmp_path):
|
||||
src = tmp_path / "design.csv"
|
||||
src.write_text("name,r1,r2,condition\nA,a_1.fq,a_2.fq,ctrl\nB,b_1.fq,b_2.fq,trt\n")
|
||||
r = _run(transform_file(src, "samples.yaml"))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
|
||||
|
||||
def test_transform_samples_yaml_missing_columns(tmp_path):
|
||||
src = tmp_path / "unknown.csv"
|
||||
src.write_text("col_a,col_b\n1,2\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "samples.yaml"))
|
||||
assert ei.value.code == "MISSING_COLUMNS"
|
||||
|
||||
|
||||
def test_transform_fasta_llm_column_map(tmp_path):
|
||||
src = tmp_path / "candidates.csv"
|
||||
src.write_text("col_a,col_b\nSPL1,ATGGCTA\nSPL2,ATGGCTB\n")
|
||||
|
||||
async def fake_llm(user, system):
|
||||
return {"name": "col_a", "seq": "col_b"}
|
||||
|
||||
r = _run(transform_file(src, "fasta", llm=fake_llm))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
|
||||
|
||||
def test_transform_fasta_llm_exception_falls_back_missing(tmp_path):
|
||||
src = tmp_path / "candidates.csv"
|
||||
src.write_text("col_a,col_b\nSPL1,ATGGCTA\n")
|
||||
|
||||
async def fake_llm(user, system):
|
||||
raise RuntimeError("llm down")
|
||||
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta", llm=fake_llm))
|
||||
assert ei.value.code == "MISSING_COLUMNS"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强(3)校验器:gff3 各失败态 / samples.yaml 缺 r1r2 / YAML 解析失败 /
|
||||
# tsv 空文件 / validate_format 未知格式
|
||||
# ===========================================================================
|
||||
|
||||
def test_validate_gff3_ok(tmp_path):
|
||||
f = tmp_path / "ok.gff3"
|
||||
f.write_text("chr1\tsrc\tgene\t1\t100\t.\t+\t.\tID=x\n")
|
||||
assert validate_gff3(f)["ok"] is True
|
||||
|
||||
|
||||
def test_validate_gff3_no_data(tmp_path):
|
||||
f = tmp_path / "empty.gff3"
|
||||
f.write_text("# just a comment\n")
|
||||
assert validate_gff3(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_gff3_bad_columns(tmp_path):
|
||||
f = tmp_path / "bad.gff3"
|
||||
f.write_text("chr1\tsrc\tgene\t1\t100\n") # 5 列 ≠ 9
|
||||
assert validate_gff3(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_gff3_bad_strand(tmp_path):
|
||||
f = tmp_path / "bad.gff3"
|
||||
f.write_text("chr1\tsrc\tgene\t1\t100\t.\tx\t.\tID=x\n") # 第 7 列非法
|
||||
assert validate_gff3(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_gff3_bad_int(tmp_path):
|
||||
f = tmp_path / "bad.gff3"
|
||||
f.write_text("chr1\tsrc\tgene\tabc\t100\t.\t+\t.\tID=x\n") # 起止非整数
|
||||
assert validate_gff3(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_fasta_no_sequence(tmp_path):
|
||||
f = tmp_path / "bad.fa"
|
||||
f.write_text(">a\n")
|
||||
assert validate_fasta(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_samples_yaml_missing_r1r2(tmp_path):
|
||||
f = tmp_path / "s.yaml"
|
||||
f.write_text("samples:\n A:\n condition: ctrl\n")
|
||||
assert validate_samples_yaml(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_samples_yaml_parse_fail(tmp_path):
|
||||
f = tmp_path / "s.yaml"
|
||||
f.write_text("samples: [unclosed\n")
|
||||
assert validate_samples_yaml(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_tsv_empty(tmp_path):
|
||||
f = tmp_path / "empty.tsv"
|
||||
f.write_text("")
|
||||
assert validate_tsv(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_format_unknown(tmp_path):
|
||||
f = tmp_path / "x.txt"
|
||||
f.write_text("x")
|
||||
r = validate_format(f, "bogus")
|
||||
assert r["ok"] is False
|
||||
assert "不支持" in r["reason"]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强(4)校验器缺失文件 / 格式猜测扩展名 / samples 非 dict / 空行跳过
|
||||
# ===========================================================================
|
||||
|
||||
def test_validators_missing_file(tmp_path):
|
||||
missing = tmp_path / "nope"
|
||||
assert validate_fasta(missing)["ok"] is False
|
||||
assert validate_gff3(missing)["ok"] is False
|
||||
assert validate_tsv(missing)["ok"] is False
|
||||
assert validate_samples_yaml(missing)["ok"] is False
|
||||
|
||||
|
||||
def test_guess_format_all_extensions():
|
||||
from app.services.bioflow.file_transform import _guess_format
|
||||
assert _guess_format("a.gff3", b"") == "gff3"
|
||||
assert _guess_format("a.gff", b"") == "gff3"
|
||||
assert _guess_format("a.gtf", b"") == "gtf"
|
||||
assert _guess_format("a.vcf", b"") == "vcf"
|
||||
assert _guess_format("a.vcf.gz", b"") == "vcf"
|
||||
assert _guess_format("a.yaml", b"") == "yaml"
|
||||
assert _guess_format("a.yml", b"") == "yaml"
|
||||
assert _guess_format("a.xls", b"") == "xlsx"
|
||||
assert _guess_format("a.faa", b"") == "fasta"
|
||||
assert _guess_format("a.fna", b"") == "fasta"
|
||||
assert _guess_format("a.pep", b"") == "fasta"
|
||||
assert _guess_format("a.cds", b"") == "fasta"
|
||||
assert _guess_format("a.dat", b"no marker") == "txt"
|
||||
|
||||
|
||||
def test_validate_samples_yaml_not_dict(tmp_path):
|
||||
f = tmp_path / "s.yaml"
|
||||
f.write_text("samples: just_a_string\n")
|
||||
assert validate_samples_yaml(f)["ok"] is False
|
||||
|
||||
|
||||
def test_transform_samples_yaml_skips_empty_rows(tmp_path):
|
||||
src = tmp_path / "d.csv"
|
||||
src.write_text("name,r1,r2\n,,\nA,a.fq,a2.fq\n")
|
||||
r = _run(transform_file(src, "samples.yaml"))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 1
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,414 @@
|
||||
"""tests/test_phase_b_error.py —— Phase B 测试用例集。
|
||||
|
||||
覆盖(Phase B 验收):
|
||||
1. G13 结构化错误对象:CLI 命令 + MCP 工具失败时返回统一错误对象,含 8 个字段
|
||||
(error_id / category / code / summary / evidence_refs / diagnosis_status /
|
||||
retryable_without_change / suggested_actions),而非裸字符串 message。
|
||||
2. MCP 薄适配行为回归:fetch_source_file / inspect_file / transform_file /
|
||||
archive_job / reuse_archive / register_job_files 委托 CLI/service 后行为与原实现一致。
|
||||
3. reports.register_report 的 httpx.Timeout 修复:不再抛 ValueError。
|
||||
|
||||
## G13 契约(本文件断言的结构)
|
||||
失败返回为 dict,且 `ok == False`,并携带结构化错误对象:
|
||||
- 结构 A:`{"ok": False, "code": "...", "error": {8 字段}}`
|
||||
- 结构 B:`{"ok": False, "code": "...", "error_id": ..., "category": ..., ...}`(字段平铺)
|
||||
两种结构均被 `_error_of` 识别;`code` 须可经顶层 `result["code"]` 或 `error["code"]`
|
||||
取到(与 Phase A 测试 `r["code"]` 断言兼容)。
|
||||
|
||||
字段类型契约:
|
||||
- error_id: 非空 str(错误唯一标识)
|
||||
- category: 非空 str(错误分类,如 validation/not_found/io/network)
|
||||
- code: 非空 str(既有错误码,如 NOT_FOUND / ASSET_EXISTS / BAD_ROLE)
|
||||
- summary: 非空 str(人读摘要,等价原 message)
|
||||
- evidence_refs: list(证据引用,可为空)
|
||||
- diagnosis_status: 非空 str(如 diagnosed)
|
||||
- retryable_without_change: bool(不改动重试是否可能成功)
|
||||
- suggested_actions: list[str](建议动作,可为空)
|
||||
|
||||
每条用例三段式(前置/步骤/预期)写在 docstring。分层标记 -m bioflow_unit。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import types as _types
|
||||
from pathlib import Path
|
||||
|
||||
import httpx as _real_httpx
|
||||
import pytest
|
||||
|
||||
import app.mcp_server as mcp
|
||||
from app.services.bioflow import store
|
||||
from app.services.bioflow.reports import register_report
|
||||
from bioflowctl import commands as cli
|
||||
|
||||
pytestmark = pytest.mark.bioflow_unit
|
||||
|
||||
G13_FIELDS = (
|
||||
"error_id", "category", "code", "summary", "evidence_refs",
|
||||
"diagnosis_status", "retryable_without_change", "suggested_actions",
|
||||
)
|
||||
|
||||
|
||||
def _run(coro):
|
||||
return asyncio.new_event_loop().run_until_complete(coro)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# G13 结构化错误对象断言助手
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _error_of(result: dict) -> dict:
|
||||
"""从失败返回中取出结构化错误对象(兼容 `error` 键与字段平铺两种结构)。"""
|
||||
if isinstance(result.get("error"), dict):
|
||||
return result["error"]
|
||||
if "error_id" in result:
|
||||
return result
|
||||
raise AssertionError(f"失败返回缺结构化错误对象: keys={sorted(result)}")
|
||||
|
||||
|
||||
def _code_of(result: dict) -> str | None:
|
||||
return result.get("code") or _error_of(result).get("code")
|
||||
|
||||
|
||||
def _assert_g13_error(result, expected_code: str) -> dict:
|
||||
"""断言 result 为 G13 失败返回,且 code 匹配;返回错误对象。"""
|
||||
assert isinstance(result, dict), f"失败返回非 dict: {result!r}"
|
||||
assert result.get("ok") is False, f"预期失败但 ok=True: {result!r}"
|
||||
err = _error_of(result)
|
||||
missing = [f for f in G13_FIELDS if f not in err]
|
||||
assert not missing, f"G13 错误对象缺字段 {missing}: {err}"
|
||||
assert isinstance(err["error_id"], str) and err["error_id"], "error_id 须非空 str"
|
||||
assert isinstance(err["category"], str) and err["category"], "category 须非空 str"
|
||||
assert isinstance(err["code"], str) and err["code"], "code 须非空 str"
|
||||
assert isinstance(err["summary"], str) and err["summary"], "summary 须非空 str"
|
||||
assert isinstance(err["evidence_refs"], list), "evidence_refs 须 list"
|
||||
assert isinstance(err["diagnosis_status"], str) and err["diagnosis_status"], "diagnosis_status 须非空 str"
|
||||
assert isinstance(err["retryable_without_change"], bool), "retryable_without_change 须 bool"
|
||||
assert isinstance(err["suggested_actions"], list), "suggested_actions 须 list"
|
||||
assert _code_of(result) == expected_code, f"code 期望 {expected_code},实际 {_code_of(result)}"
|
||||
return err
|
||||
|
||||
|
||||
async def _seed_archive_into(db, archive: dict) -> None:
|
||||
await store.insert_archive(db, archive)
|
||||
|
||||
|
||||
def _archive_row(**over) -> dict:
|
||||
base = {
|
||||
"id": "a1", "job_id": "j1", "name": "SPL鉴定", "description": "",
|
||||
"stage_chain": ["identify", "phylogeny"],
|
||||
"params": {"identify": {"cpu": "8"}},
|
||||
"report_artifact": None, "disk_path": "/tmp/a1", "size_bytes": 10,
|
||||
}
|
||||
base.update(over)
|
||||
return base
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 1. G13 结构化错误对象 —— 形状契约(冒烟)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g13_error_object_has_all_fields(tmp_path):
|
||||
"""前置:空 db。步骤:cmd_archive_reuse(不存在的存档)。
|
||||
预期:ok=False 且含 8 字段结构化错误对象,code=ARCHIVE_NOT_FOUND。"""
|
||||
r = _run(cli.cmd_archive_reuse("不存在", db_path=tmp_path / "empty.db"))
|
||||
_assert_g13_error(r, "ARCHIVE_NOT_FOUND")
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 2. CLI 命令失败路径 → G13(覆盖各命令的错误码)
|
||||
# ===========================================================================
|
||||
|
||||
# ---- source fetch ----
|
||||
|
||||
def test_g13_source_fetch_empty(tmp_path):
|
||||
"""前置:空源。步骤:cmd_source_fetch("")。预期:G13 错误,code=EMPTY_SOURCE。"""
|
||||
r = _run(cli.cmd_source_fetch(""))
|
||||
_assert_g13_error(r, "EMPTY_SOURCE")
|
||||
|
||||
|
||||
def test_g13_source_fetch_bad_scheme(tmp_path):
|
||||
"""前置:ftp:// 源。步骤:cmd_source_fetch。预期:G13 错误,code=BAD_SCHEME。"""
|
||||
r = _run(cli.cmd_source_fetch("ftp://host/x.fa"))
|
||||
_assert_g13_error(r, "BAD_SCHEME")
|
||||
|
||||
|
||||
def test_g13_source_fetch_lan_missing(tmp_path):
|
||||
"""前置:局域网路径不存在。步骤:cmd_source_fetch。预期:G13 错误,code=PATH_NOT_EXIST。"""
|
||||
r = _run(cli.cmd_source_fetch("/no/such/path.fa"))
|
||||
_assert_g13_error(r, "PATH_NOT_EXIST")
|
||||
|
||||
|
||||
# ---- file inspect ----
|
||||
|
||||
def test_g13_file_inspect_not_found(tmp_path):
|
||||
"""前置:文件不存在。步骤:cmd_file_inspect。预期:G13 错误,code=NOT_FOUND。"""
|
||||
r = cli.cmd_file_inspect(str(tmp_path / "nope.fa"))
|
||||
_assert_g13_error(r, "NOT_FOUND")
|
||||
|
||||
|
||||
# ---- file transform ----
|
||||
|
||||
def test_g13_file_transform_not_found(tmp_path):
|
||||
"""前置:源不存在。步骤:cmd_file_transform。预期:G13 错误,code=NOT_FOUND。"""
|
||||
r = _run(cli.cmd_file_transform(str(tmp_path / "nope.csv"), "fasta"))
|
||||
_assert_g13_error(r, "NOT_FOUND")
|
||||
|
||||
|
||||
def test_g13_file_transform_bad_format(tmp_path):
|
||||
"""前置:csv 源,目标 bed。步骤:cmd_file_transform。预期:G13 错误,code=BAD_FORMAT。"""
|
||||
src = tmp_path / "a.csv"
|
||||
src.write_text("a,b\n1,2\n")
|
||||
r = _run(cli.cmd_file_transform(str(src), "bed"))
|
||||
_assert_g13_error(r, "BAD_FORMAT")
|
||||
|
||||
|
||||
def test_g13_file_transform_empty_table(tmp_path):
|
||||
"""前置:空文件。步骤:cmd_file_transform(fasta)。预期:G13 错误,code=EMPTY_TABLE。"""
|
||||
src = tmp_path / "empty.csv"
|
||||
src.write_text("", encoding="utf-8")
|
||||
r = _run(cli.cmd_file_transform(str(src), "fasta"))
|
||||
_assert_g13_error(r, "EMPTY_TABLE")
|
||||
|
||||
|
||||
def test_g13_file_transform_missing_columns(tmp_path):
|
||||
"""前置:表头无 name/seq。步骤:cmd_file_transform(fasta)。预期:G13 错误,code=MISSING_COLUMNS。"""
|
||||
src = tmp_path / "x.csv"
|
||||
src.write_text("foo,bar\n1,2\n")
|
||||
r = _run(cli.cmd_file_transform(str(src), "fasta"))
|
||||
_assert_g13_error(r, "MISSING_COLUMNS")
|
||||
|
||||
|
||||
def test_g13_file_transform_validation_failed(tmp_path):
|
||||
"""前置:seq 全空。步骤:cmd_file_transform(fasta)。预期:G13 错误,code=VALIDATION_FAILED。"""
|
||||
src = tmp_path / "genes.csv"
|
||||
src.write_text("gene,seq\nSPL1,\nSPL2,\n")
|
||||
r = _run(cli.cmd_file_transform(str(src), "fasta", column_map={"name": "gene", "seq": "seq"}))
|
||||
_assert_g13_error(r, "VALIDATION_FAILED")
|
||||
|
||||
|
||||
def test_g13_file_transform_gff3_unsupported(tmp_path):
|
||||
"""前置:tsv 源。步骤:cmd_file_transform(gff3)。预期:G13 错误,code=UNSUPPORTED。"""
|
||||
src = tmp_path / "a.tsv"
|
||||
src.write_text("a\tb\tc\n1\t2\t3\n")
|
||||
r = _run(cli.cmd_file_transform(str(src), "gff3"))
|
||||
_assert_g13_error(r, "UNSUPPORTED")
|
||||
|
||||
|
||||
# ---- asset register ----
|
||||
|
||||
def test_g13_asset_register_not_found(tmp_path):
|
||||
"""前置:源不存在。步骤:register_asset。预期:G13 错误,code=NOT_FOUND。"""
|
||||
r = cli.register_asset("pep.fa", str(tmp_path / "nope.fa"), assets_root=tmp_path / "assets")
|
||||
_assert_g13_error(r, "NOT_FOUND")
|
||||
|
||||
|
||||
def test_g13_asset_register_bad_role(tmp_path):
|
||||
"""前置:角色名非法。步骤:register_asset。预期:G13 错误,code=BAD_ROLE。"""
|
||||
f = tmp_path / "x.fa"
|
||||
f.write_text(">x\nACGT\n")
|
||||
r = cli.register_asset("../pep", str(f), assets_root=tmp_path / "assets")
|
||||
_assert_g13_error(r, "BAD_ROLE")
|
||||
|
||||
|
||||
def test_g13_asset_register_exists(tmp_path):
|
||||
"""前置:同名资产已发布。步骤:register_asset 再次同名。预期:G13 错误,code=ASSET_EXISTS。"""
|
||||
f = tmp_path / "x.fa"
|
||||
f.write_text("V1")
|
||||
assets = tmp_path / "assets"
|
||||
assert cli.register_asset("pep.fa", str(f), assets_root=assets)["ok"] is True
|
||||
r = cli.register_asset("pep.fa", str(f), assets_root=assets)
|
||||
_assert_g13_error(r, "ASSET_EXISTS")
|
||||
|
||||
|
||||
# ---- asset validate ----
|
||||
|
||||
def test_g13_asset_validate_not_found(tmp_path):
|
||||
"""前置:文件不存在。步骤:validate_asset。预期:G13 错误,code=NOT_FOUND。"""
|
||||
r = cli.validate_asset("pep.fa", str(tmp_path / "nope.fa"))
|
||||
_assert_g13_error(r, "NOT_FOUND")
|
||||
|
||||
|
||||
def test_g13_asset_validate_bad_role(tmp_path):
|
||||
"""前置:角色无法推断格式 + txt。步骤:validate_asset。预期:G13 错误,code=BAD_ROLE。"""
|
||||
f = tmp_path / "x.txt"
|
||||
f.write_text("hello\n")
|
||||
r = cli.validate_asset("unknown_role", str(f))
|
||||
_assert_g13_error(r, "BAD_ROLE")
|
||||
|
||||
|
||||
# ---- report register ----
|
||||
|
||||
def test_g13_report_register_empty():
|
||||
"""前置:空路径。步骤:cmd_report_register("")。预期:G13 错误,code=EMPTY_REPORT。"""
|
||||
r = cli.cmd_report_register("")
|
||||
_assert_g13_error(r, "EMPTY_REPORT")
|
||||
|
||||
|
||||
# ---- archive ----
|
||||
|
||||
def test_g13_archive_not_found(tmp_path):
|
||||
"""前置:空 db。步骤:cmd_archive(不存在 job)。预期:G13 错误,code=NOT_FOUND。"""
|
||||
r = _run(cli.cmd_archive(
|
||||
"nope", "X", db_path=tmp_path / "empty.db",
|
||||
jobs_dir=tmp_path / "jobs", archives_dir=tmp_path / "archives"))
|
||||
_assert_g13_error(r, "NOT_FOUND")
|
||||
|
||||
|
||||
# ---- archive reuse ----
|
||||
|
||||
def test_g13_archive_reuse_not_found(tmp_path):
|
||||
"""前置:空 db。步骤:cmd_archive_reuse。预期:G13 错误,code=ARCHIVE_NOT_FOUND。"""
|
||||
r = _run(cli.cmd_archive_reuse("不存在", db_path=tmp_path / "empty.db"))
|
||||
_assert_g13_error(r, "ARCHIVE_NOT_FOUND")
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 3. MCP 薄适配 —— 行为回归 + G13(6 个工具)
|
||||
# ===========================================================================
|
||||
|
||||
# ---- inspect_file ----
|
||||
|
||||
def test_mcp_inspect_file_ok(tmp_path):
|
||||
"""回归。前置:fasta 文件。步骤:mcp.inspect_file。预期:ok=True,format=fasta。"""
|
||||
f = tmp_path / "p.fa"
|
||||
f.write_text(">S1\nMKTA\n")
|
||||
r = _run(mcp.inspect_file(str(f)))
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "fasta"
|
||||
|
||||
|
||||
def test_mcp_inspect_file_not_found(tmp_path):
|
||||
"""G13。前置:文件不存在。步骤:mcp.inspect_file。预期:G13 错误,code=NOT_FOUND。"""
|
||||
r = _run(mcp.inspect_file(str(tmp_path / "nope.fa")))
|
||||
_assert_g13_error(r, "NOT_FOUND")
|
||||
|
||||
|
||||
# ---- transform_file ----
|
||||
|
||||
def test_mcp_transform_file_ok(tmp_path):
|
||||
"""回归。前置:csv(gene,seq)。步骤:mcp.transform_file(fasta, 显式 column_map)。
|
||||
预期:ok=True,format=fasta,records=2。"""
|
||||
src = tmp_path / "genes.csv"
|
||||
src.write_text("gene,seq\nSPL1,MKTA\nSPL2,MKTB\n")
|
||||
r = _run(mcp.transform_file(str(src), "fasta", {"name": "gene", "seq": "seq"}))
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "fasta"
|
||||
assert r["records"] == 2
|
||||
|
||||
|
||||
def test_mcp_transform_file_failure(tmp_path):
|
||||
"""G13。前置:源不存在。步骤:mcp.transform_file。预期:G13 错误,code=NOT_FOUND。"""
|
||||
r = _run(mcp.transform_file(str(tmp_path / "nope.csv"), "fasta"))
|
||||
_assert_g13_error(r, "NOT_FOUND")
|
||||
|
||||
|
||||
# ---- fetch_source_file ----
|
||||
|
||||
def test_mcp_fetch_source_file_lan_ok(tmp_path):
|
||||
"""回归。前置:本机文件。步骤:mcp.fetch_source_file(局域网路径, job_id=None)。
|
||||
预期:ok=True,kind=lan。"""
|
||||
f = tmp_path / "genome.fa"
|
||||
f.write_text(">g1\nACGT\n")
|
||||
r = _run(mcp.fetch_source_file(str(f), job_id=None))
|
||||
assert r["ok"] is True
|
||||
assert r["kind"] == "lan"
|
||||
|
||||
|
||||
def test_mcp_fetch_source_file_failure(tmp_path):
|
||||
"""G13。前置:空源。步骤:mcp.fetch_source_file("")。预期:G13 错误,code=EMPTY_SOURCE。"""
|
||||
r = _run(mcp.fetch_source_file("", job_id=None))
|
||||
_assert_g13_error(r, "EMPTY_SOURCE")
|
||||
|
||||
|
||||
# ---- archive_job ----
|
||||
|
||||
def test_mcp_archive_job_not_found(api_context):
|
||||
"""G13。前置:无该任务。步骤:mcp.archive_job(不存在)。预期:G13 错误,code=NOT_FOUND。"""
|
||||
r = _run(mcp.archive_job("nope", "X"))
|
||||
_assert_g13_error(r, "NOT_FOUND")
|
||||
|
||||
|
||||
# ---- reuse_archive ----
|
||||
|
||||
def test_mcp_reuse_archive_not_found(api_context):
|
||||
"""G13。前置:无匹配存档。步骤:mcp.reuse_archive。预期:G13 错误,code=ARCHIVE_NOT_FOUND。"""
|
||||
r = _run(mcp.reuse_archive("不存在"))
|
||||
_assert_g13_error(r, "ARCHIVE_NOT_FOUND")
|
||||
|
||||
|
||||
def test_mcp_reuse_archive_ok(api_context):
|
||||
"""回归。前置:seed 存档。步骤:mcp.reuse_archive。预期:ok=True,返回 stage_chain+params。"""
|
||||
_run(_seed_archive_into(api_context.db, _archive_row(
|
||||
params={"identify": {"cpu": "8", "memory": "16G"}})))
|
||||
r = _run(mcp.reuse_archive("SPL鉴定"))
|
||||
assert r["ok"] is True
|
||||
assert r["data"]["stage_chain"] == ["identify", "phylogeny"]
|
||||
assert r["data"]["params"]["identify"]["cpu"] == "8"
|
||||
|
||||
|
||||
# ---- register_job_files ----
|
||||
|
||||
def test_mcp_register_job_files_not_found(api_context):
|
||||
"""G13。前置:无该任务。步骤:mcp.register_job_files(不存在, [路径])。预期:G13 错误,code=NOT_FOUND。"""
|
||||
r = _run(mcp.register_job_files("nope", ["/tmp/x.fa"]))
|
||||
_assert_g13_error(r, "NOT_FOUND")
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 4. reports.register_report 的 httpx.Timeout 修复(不再抛 ValueError)
|
||||
# ===========================================================================
|
||||
|
||||
def _fake_httpx_client_for_report(monkeypatch, rep, resp_factory):
|
||||
"""注入 fake httpx:Timeout 用真实类(校验修复后传参合法),AsyncClient 用 resp_factory。"""
|
||||
captured: dict = {}
|
||||
|
||||
class _Client:
|
||||
def __init__(self, timeout=None):
|
||||
captured["timeout"] = timeout
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
async def post(self, url, json=None):
|
||||
return resp_factory()
|
||||
|
||||
monkeypatch.setattr(rep, "httpx", _types.SimpleNamespace(
|
||||
AsyncClient=_Client, Timeout=_real_httpx.Timeout,
|
||||
HTTPError=_real_httpx.HTTPError,
|
||||
))
|
||||
return captured
|
||||
|
||||
|
||||
def test_register_report_unreachable_no_value_error(monkeypatch):
|
||||
"""修复。前置:登记端点不可达(ConnectError)。步骤:register_report(register_url)。
|
||||
预期:不抛 ValueError,返回 degraded=True, reason=unreachable。"""
|
||||
from app.services.bioflow import reports as rep
|
||||
|
||||
def _resp():
|
||||
raise _real_httpx.ConnectError("unreachable")
|
||||
|
||||
_fake_httpx_client_for_report(monkeypatch, rep, _resp)
|
||||
r = _run(rep.register_report({"job_id": "j1"}, register_url="http://127.0.0.1:1/x"))
|
||||
assert r == {"ok": False, "degraded": True, "reason": "unreachable"}
|
||||
|
||||
|
||||
def test_register_report_http_error_degraded(monkeypatch):
|
||||
"""修复。前置:登记端点返回 500。步骤:register_report(register_url)。
|
||||
预期:不抛 ValueError,返回 degraded=True, reason=http_500。"""
|
||||
from app.services.bioflow import reports as rep
|
||||
|
||||
class _Resp:
|
||||
status_code = 500
|
||||
|
||||
async def json(self):
|
||||
return {}
|
||||
|
||||
_fake_httpx_client_for_report(monkeypatch, rep, lambda: _Resp())
|
||||
r = _run(rep.register_report({"job_id": "j1"}, register_url="http://127.0.0.1:1/x"))
|
||||
assert r["ok"] is False
|
||||
assert r["degraded"] is True
|
||||
assert r["reason"] == "http_500"
|
||||
@@ -0,0 +1,79 @@
|
||||
"""tests/test_phase_c.py —— Phase C(G12 Skill 接入 + 11 工具)补充测试。
|
||||
|
||||
契约测试见 test_phase_c_skill.py;本文件补:MCP 注册、cmd_validate_plan(CLI 参数校验)。
|
||||
分层标记 -m bioflow_unit。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.bioflow.skill_tools import (
|
||||
TOOL_NAMES,
|
||||
control_execution,
|
||||
validate_plan,
|
||||
)
|
||||
from bioflowctl.commands import cmd_validate_plan
|
||||
|
||||
pytestmark = pytest.mark.bioflow_unit
|
||||
|
||||
|
||||
def _run(coro):
|
||||
return asyncio.new_event_loop().run_until_complete(coro)
|
||||
|
||||
|
||||
def test_g12_eleven_tools_registered_on_mcp():
|
||||
from app.mcp_server import build_mcp_servers
|
||||
gf, rn, _ext = build_mcp_servers()
|
||||
gf_names = {t.name for t in _run(gf.list_tools())}
|
||||
rn_names = {t.name for t in _run(rn.list_tools())}
|
||||
for name in TOOL_NAMES:
|
||||
assert name in gf_names, f"基因家族端点缺工具 {name}"
|
||||
assert name in rn_names, f"转录组端点缺工具 {name}"
|
||||
|
||||
|
||||
def test_g12_validate_plan_known():
|
||||
r = validate_plan(["identify", "phylogeny"])
|
||||
assert r["ok"] is True
|
||||
assert r["result"]["valid"] is True
|
||||
|
||||
|
||||
def test_g12_validate_plan_unknown():
|
||||
r = validate_plan(["bogus_stage"])
|
||||
assert r["ok"] is False
|
||||
assert r["error"]["code"] == "UNKNOWN_STAGE"
|
||||
|
||||
|
||||
def test_g12_control_execution_unknown_command():
|
||||
r = control_execution("", "bogus")
|
||||
assert r["ok"] is False
|
||||
assert r["error"]["code"] == "VALIDATION"
|
||||
|
||||
|
||||
def test_g12_control_execution_stop_resume():
|
||||
# 独立 run_id 避免模块级状态跨用例污染
|
||||
r1 = control_execution("run-ctx-9", "start")
|
||||
assert r1["ok"] is True and r1["result"]["done"] is True # claim
|
||||
r2 = control_execution("run-ctx-9", "stop")
|
||||
assert r2["ok"] is True and r2["result"]["done"] is True # 所有权匹配 → 终止
|
||||
|
||||
|
||||
# ---- cmd_validate_plan(CLI 层,读单一 schema 校验参数)----
|
||||
|
||||
def test_g12_cmd_validate_plan_valid():
|
||||
r = cmd_validate_plan("identify", {"cpu": "8", "memory": "32g"})
|
||||
assert r["ok"] is True
|
||||
|
||||
|
||||
def test_g12_cmd_validate_plan_unknown_param():
|
||||
r = cmd_validate_plan("identify", {"bogus": "x"})
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "VALIDATION_FAILED"
|
||||
assert "bogus" in r["message"]
|
||||
|
||||
|
||||
def test_g12_cmd_validate_plan_choice_out_of_range():
|
||||
r = cmd_validate_plan("identify", {"cpu": "99"})
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "VALIDATION_FAILED"
|
||||
@@ -0,0 +1,201 @@
|
||||
"""tests/test_phase_c_skill.py —— Phase C(Skill 接入 + 11 工具,G12)测试用例集。
|
||||
|
||||
覆盖(Phase C 验收):
|
||||
1. 11 个分析控制工具:get_task_context / inspect_asset / validate_plan /
|
||||
check_readiness / get_run_evidence / propose_change / apply_change /
|
||||
execute_nodes / control_execution / validate_delivery / publish_delivery
|
||||
—— 每个工具正确委托底层 CLI 命令,返回统一结构化结果(ok+result 或 G13 错误对象)。
|
||||
2. SKILL.md 结构:frontmatter 齐全(name/description/触发词)、11 工具白名单、
|
||||
对话剧本、人工边界(subfamily_confirm/rnaseq_de requires_human)、验收清单。
|
||||
3. 装卸脚本:install/uninstall 正确安装/移除 skill。
|
||||
|
||||
## 契约说明(开发对齐用)
|
||||
- 11 工具集中 `app.services.bioflow.skill_tools`,每个工具为模块级函数,签名见
|
||||
SKILL_TOOL_SIGNATURES;返回统一 `{"ok": bool, "result"?: ..., "error"?: G13}`。
|
||||
- SKILL.md 位于 `bioflow_service/SKILL.md`(相对仓库根)。
|
||||
- 装卸脚本:`bioflow_service/install_skill.sh` 与 `bioflow_service/uninstall_skill.sh`。
|
||||
|
||||
未实现函数用 `_phase_c_fn` 动态导入,缺失则 skip(开发完成后须全部存在、无 skip)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.bioflow_unit
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
|
||||
TOOL_NAMES = [
|
||||
"get_task_context", "inspect_asset", "validate_plan", "check_readiness",
|
||||
"get_run_evidence", "propose_change", "apply_change", "execute_nodes",
|
||||
"control_execution", "validate_delivery", "publish_delivery",
|
||||
]
|
||||
|
||||
# 每个工具的最小调用参数(无参/空参 → 应返回统一结构,失败即 G13)
|
||||
SKILL_TOOL_SIGNATURES = {
|
||||
"get_task_context": dict(job_id=None),
|
||||
"inspect_asset": dict(role="pep.fa", path=""),
|
||||
"validate_plan": dict(stages=[]),
|
||||
"check_readiness": dict(assets_root="", stages=[]),
|
||||
"get_run_evidence": dict(run_id=""),
|
||||
"propose_change": dict(job_id="", stage="", params={}),
|
||||
"apply_change": dict(action_id="", change={}),
|
||||
"execute_nodes": dict(job_id="", nodes=[]),
|
||||
"control_execution": dict(run_id="", command=""),
|
||||
"validate_delivery": dict(manifest={}),
|
||||
"publish_delivery": dict(manifest={}),
|
||||
}
|
||||
|
||||
|
||||
def _phase_c_fn(module: str, name: str):
|
||||
try:
|
||||
mod = importlib.import_module(module)
|
||||
return getattr(mod, name)
|
||||
except (ImportError, AttributeError) as exc:
|
||||
pytest.skip(f"Phase C 未实现 {module}.{name}: {exc}")
|
||||
|
||||
|
||||
def _skill_md_path() -> Path | None:
|
||||
"""定位 SKILL.md(候选路径:bioflow_service/SKILL.md)。"""
|
||||
candidates = [
|
||||
REPO_ROOT / "bioflow_service" / "SKILL.md",
|
||||
REPO_ROOT / "bioflow_service" / "bioflowctl" / "SKILL.md",
|
||||
REPO_ROOT / "SKILL.md",
|
||||
]
|
||||
for p in candidates:
|
||||
if p.is_file():
|
||||
return p
|
||||
return None
|
||||
|
||||
|
||||
def _skill_md_text() -> str | None:
|
||||
p = _skill_md_path()
|
||||
if p is None:
|
||||
return None
|
||||
return p.read_text(encoding="utf-8", errors="replace")
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G12-1:11 个分析控制工具
|
||||
# ===========================================================================
|
||||
|
||||
def test_g12_skill_exposes_eleven_tools():
|
||||
"""G12。前置:skill_tools 模块。步骤:读 TOOL_NAMES/模块属性。预期:恰 11 工具。"""
|
||||
try:
|
||||
mod = importlib.import_module("app.services.bioflow.skill_tools")
|
||||
except ImportError:
|
||||
pytest.skip("Phase C 未实现 app.services.bioflow.skill_tools")
|
||||
names = getattr(mod, "TOOL_NAMES", None)
|
||||
if names is None:
|
||||
# 逐个属性探测
|
||||
names = [n for n in TOOL_NAMES if hasattr(mod, n)]
|
||||
assert set(names) == set(TOOL_NAMES), f"工具集不一致: {names}"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tool", TOOL_NAMES)
|
||||
def test_g12_tool_returns_unified_structure(tool):
|
||||
"""G12。前置:工具可调用。步骤:以最小参数调用。预期:返回 dict 含 ok;
|
||||
失败时携带 G13 错误对象(非裸字符串)。"""
|
||||
fn = _phase_c_fn("app.services.bioflow.skill_tools", tool)
|
||||
result = fn(**SKILL_TOOL_SIGNATURES[tool])
|
||||
assert isinstance(result, dict), f"{tool} 未返回 dict: {result!r}"
|
||||
assert "ok" in result, f"{tool} 返回缺 ok 键"
|
||||
if result["ok"] is False:
|
||||
err = result.get("error") if isinstance(result.get("error"), dict) else result
|
||||
assert "error_id" in err or "code" in err, f"{tool} 失败返回缺 G13 错误对象"
|
||||
|
||||
|
||||
def test_g12_get_task_context_delegates():
|
||||
"""G12。前置:get_task_context 工具。步骤:调用。预期:返回 5 字段任务上下文。"""
|
||||
fn = _phase_c_fn("app.services.bioflow.skill_tools", "get_task_context")
|
||||
r = fn(job_id=None)
|
||||
if r.get("ok") is True:
|
||||
ctx = r.get("result") or r.get("data") or {}
|
||||
for k in ("session_id", "job_id", "plan_revision", "run_id", "current_stage"):
|
||||
assert k in ctx, f"缺字段 {k}"
|
||||
|
||||
|
||||
def test_g12_validate_delivery_delegates():
|
||||
"""G12。前置:validate_delivery 工具。步骤:不完整 manifest。预期:ok=False + G13。"""
|
||||
fn = _phase_c_fn("app.services.bioflow.skill_tools", "validate_delivery")
|
||||
r = fn(manifest={})
|
||||
assert isinstance(r, dict) and "ok" in r
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G12-2:SKILL.md 结构
|
||||
# ===========================================================================
|
||||
|
||||
def test_skill_md_exists():
|
||||
"""SKILL.md。前置:文件存在。步骤:定位。预期:找到 SKILL.md。"""
|
||||
assert _skill_md_path() is not None, "未找到 SKILL.md"
|
||||
|
||||
|
||||
def test_skill_md_frontmatter():
|
||||
"""SKILL.md。前置:frontmatter。步骤:解析 name/description/触发词。
|
||||
预期:name/description 非空,含触发词。"""
|
||||
text = _skill_md_text()
|
||||
assert text is not None
|
||||
m = re.match(r"^---\s*\n(.*?)\n---", text, flags=re.S)
|
||||
assert m is not None, "缺 frontmatter"
|
||||
fm = m.group(1)
|
||||
assert "name:" in fm, "frontmatter 缺 name"
|
||||
assert "description:" in fm, "frontmatter 缺 description"
|
||||
# 触发词(name 或 description 或 triggers 字段)
|
||||
assert any(k in fm for k in ("triggers", "触发", "生物", "基因家族", "生信")), "缺触发词"
|
||||
|
||||
|
||||
def test_skill_md_eleven_tools_whitelist():
|
||||
"""SKILL.md。前置:正文。步骤:检查 11 工具白名单。预期:11 工具名全部列出。"""
|
||||
text = _skill_md_text()
|
||||
assert text is not None
|
||||
for tool in TOOL_NAMES:
|
||||
assert tool in text, f"SKILL.md 缺工具 {tool}"
|
||||
|
||||
|
||||
def test_skill_md_human_boundaries():
|
||||
"""SKILL.md。前置:正文。步骤:检查人工边界。预期:subfamily_confirm/rnaseq_de 标 requires_human。"""
|
||||
text = _skill_md_text()
|
||||
assert text is not None
|
||||
assert "subfamily_confirm" in text, "缺 subfamily_confirm 人工边界"
|
||||
assert "rnaseq_de" in text, "缺 rnaseq_de 人工边界"
|
||||
assert "requires_human" in text, "缺 requires_human 标注"
|
||||
|
||||
|
||||
def test_skill_md_acceptance_checklist():
|
||||
"""SKILL.md。前置:正文。步骤:检查验收清单。预期:含验收/检查清单。"""
|
||||
text = _skill_md_text()
|
||||
assert text is not None
|
||||
assert any(k in text for k in ("验收", "acceptance", "检查清单", "checklist")), "缺验收清单"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G12-3:装卸脚本
|
||||
# ===========================================================================
|
||||
|
||||
def test_skill_install_script_exists():
|
||||
"""装卸。前置:install 脚本。步骤:定位。预期:存在 install_skill.sh。"""
|
||||
p = REPO_ROOT / "bioflow_service" / "install_skill.sh"
|
||||
assert p.is_file(), f"未找到 install 脚本: {p}"
|
||||
|
||||
|
||||
def test_skill_uninstall_script_exists():
|
||||
"""装卸。前置:uninstall 脚本。步骤:定位。预期:存在 uninstall_skill.sh。"""
|
||||
p = REPO_ROOT / "bioflow_service" / "uninstall_skill.sh"
|
||||
assert p.is_file(), f"未找到 uninstall 脚本: {p}"
|
||||
|
||||
|
||||
def test_skill_install_uninstall_scripts_content():
|
||||
"""装卸。前置:两脚本。步骤:读内容。预期:install 含安装、uninstall 含移除逻辑。"""
|
||||
install = REPO_ROOT / "bioflow_service" / "install_skill.sh"
|
||||
uninstall = REPO_ROOT / "bioflow_service" / "uninstall_skill.sh"
|
||||
assert install.is_file(), f"未找到 install 脚本: {install}"
|
||||
assert uninstall.is_file(), f"未找到 uninstall 脚本: {uninstall}"
|
||||
itxt = install.read_text(encoding="utf-8", errors="replace")
|
||||
utxt = uninstall.read_text(encoding="utf-8", errors="replace")
|
||||
assert "SKILL" in itxt or "skill" in itxt
|
||||
assert "rm" in utxt or "remove" in utxt.lower() or "uninstall" in utxt.lower()
|
||||
@@ -0,0 +1,228 @@
|
||||
"""tests/test_phase_d.py —— Phase D(G7-G11 + G14 + G15)补充测试(契约测试见 test_phase_d_contract.py)。
|
||||
|
||||
覆盖契约未直接断言的实现细节:
|
||||
- classify_reason(runner 失败原因 + 退出码/日志证据定类)。
|
||||
- auto_fill_prerequisites 的 satisfied_roles 跳过上游(G7 端到端)。
|
||||
- stage_schema 单一 schema 访问点(G9)。
|
||||
- RecoveryBudget.decide 的 runner 集成语义 + runner 自动重试(G15)。
|
||||
- task_context.current_stage。
|
||||
分层标记 -m bioflow_unit。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.bioflow.catalog import build_catalog, stage_schema
|
||||
from app.services.bioflow.errors import classify_reason
|
||||
from app.services.bioflow.kernel import ImageMap
|
||||
from app.services.bioflow.readiness import check_readiness
|
||||
from app.services.bioflow.recovery import RecoveryBudget
|
||||
from app.services.bioflow.runner import JOB_FAILED, STAGE_FAILED, JobRecord
|
||||
from app.services.bioflow.stage_assets import (
|
||||
downstream_entry_satisfied,
|
||||
stage_input_roles,
|
||||
upstream_stages_for_roles,
|
||||
)
|
||||
from app.services.bioflow.task_context import current_stage, get_task_context
|
||||
from conftest import FakeTimedOutExecutor, make_runner
|
||||
|
||||
pytestmark = pytest.mark.bioflow_unit
|
||||
|
||||
|
||||
def _scenario(coro_factory):
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
return loop.run_until_complete(coro_factory())
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
|
||||
def _rnaseq_catalog():
|
||||
im = ImageMap(stages={
|
||||
"rnaseq_qc": {"image": "bioflow-rnaseq", "module": "cli.run_rnaseq_qc", "executable": True},
|
||||
"rnaseq_align": {"image": "bioflow-rnaseq", "module": "cli.run_rnaseq_align", "executable": True},
|
||||
"rnaseq_quant": {"image": "bioflow-rnaseq", "module": "cli.run_rnaseq_quant", "executable": True},
|
||||
"rnaseq_sample_qc": {"image": "bioflow-rnaseq", "module": "cli.run_rnaseq_sample_qc", "executable": True},
|
||||
"rnaseq_de": {"image": "bioflow-rnaseq", "module": "cli.run_rnaseq_de", "executable": True},
|
||||
})
|
||||
return build_catalog(im)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G14 classify_reason(runner 失败原因 + 证据)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g14_reason_timeout_transient():
|
||||
cat, diag, retry, _ = classify_reason("timeout")
|
||||
assert cat == "transient" and retry is True
|
||||
|
||||
|
||||
def test_g14_reason_exit_code_undiagnosed():
|
||||
cat, diag, retry, _ = classify_reason("exit_code")
|
||||
assert cat == "adapter" and diag == "undiagnosed" and retry is False
|
||||
|
||||
|
||||
def test_g14_reason_no_manifest_artifact():
|
||||
cat, *_ = classify_reason("no_manifest")
|
||||
assert cat == "artifact"
|
||||
|
||||
|
||||
def test_g14_exit_code_137_resource():
|
||||
cat, *_ = classify_reason("exit_code", exit_code=137)
|
||||
assert cat == "resource"
|
||||
|
||||
|
||||
def test_g14_log_oom_resource():
|
||||
cat, *_ = classify_reason("exit_code", log_tail="Error: out of memory in container")
|
||||
assert cat == "resource"
|
||||
|
||||
|
||||
def test_g14_log_image_environment():
|
||||
cat, *_ = classify_reason("exit_code", log_tail="docker: no such image")
|
||||
assert cat == "environment"
|
||||
|
||||
|
||||
def test_g14_log_scientific_precondition():
|
||||
cat, *_ = classify_reason("exit_code", log_tail="no significant DEGs found")
|
||||
assert cat == "scientific_precondition"
|
||||
|
||||
|
||||
def test_g14_log_reference():
|
||||
cat, *_ = classify_reason("exit_code", log_tail="reference genome index not found")
|
||||
assert cat == "reference"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G7/G8 角色化资产 + 下游直入
|
||||
# ===========================================================================
|
||||
|
||||
def test_g8_stage_input_roles_counts_only():
|
||||
assert stage_input_roles("rnaseq_de") == ["counts"]
|
||||
assert stage_input_roles("unknown") == []
|
||||
|
||||
|
||||
def test_g7_downstream_entry_satisfied_tuple():
|
||||
ok, missing = downstream_entry_satisfied("rnaseq_de", {"counts": {"validated": {"ok": True}}})
|
||||
assert ok is True and missing == []
|
||||
|
||||
|
||||
def test_g7_auto_fill_skips_counts_upstream():
|
||||
catalog = _rnaseq_catalog()
|
||||
ordered, _ = catalog.auto_fill_prerequisites(["rnaseq_de"])
|
||||
assert set(["rnaseq_qc", "rnaseq_align", "rnaseq_quant", "rnaseq_sample_qc"]) <= set(ordered)
|
||||
ordered2, added2 = catalog.auto_fill_prerequisites(["rnaseq_de"], satisfied_roles=["counts"])
|
||||
assert "rnaseq_qc" not in ordered2 and "rnaseq_align" not in ordered2
|
||||
assert "rnaseq_quant" not in ordered2 and "rnaseq_sample_qc" not in ordered2
|
||||
assert ordered2 == ["rnaseq_de"] and added2 == []
|
||||
|
||||
|
||||
def test_g7_upstream_stages_for_roles():
|
||||
up = upstream_stages_for_roles(["counts"])
|
||||
assert set(["rnaseq_qc", "rnaseq_align", "rnaseq_quant", "rnaseq_sample_qc"]) <= up
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G9 单一参数 schema
|
||||
# ===========================================================================
|
||||
|
||||
def test_g9_stage_schema_single_source():
|
||||
schema = stage_schema("identify")
|
||||
assert schema["name"] == "基因家族鉴定"
|
||||
assert isinstance(schema["params"], list) and schema["params"]
|
||||
assert schema["dependencies"] == []
|
||||
assert schema["input_roles"] == ["hmm", "protein_fasta"]
|
||||
|
||||
|
||||
def test_g9_stage_schema_unknown_fallback():
|
||||
schema = stage_schema("no_such_stage")
|
||||
assert schema["name"] == "no_such_stage"
|
||||
assert schema["params"] and schema["params"][0]["name"] == "cpu"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G10 就绪检查(四象限)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g10_check_readiness_all_ready(tmp_path):
|
||||
r = check_readiness(assets_root=tmp_path, stages=[])
|
||||
assert r["ready"] is True
|
||||
for q in ("environment", "reference", "space", "budget"):
|
||||
assert q in r and r[q]["ok"] is True
|
||||
|
||||
|
||||
def test_g10_check_readiness_missing_reference(tmp_path):
|
||||
r = check_readiness(assets_root=tmp_path, stages=["identify"])
|
||||
assert r["ready"] is False
|
||||
assert r["reference"]["ok"] is False
|
||||
assert r["reference"]["error"]["category"] == "reference"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G11 任务绑定
|
||||
# ===========================================================================
|
||||
|
||||
def test_g11_get_task_context_empty_binding():
|
||||
ctx = get_task_context()
|
||||
for k in ("session_id", "job_id", "plan_revision", "run_id", "current_stage"):
|
||||
assert k in ctx
|
||||
|
||||
|
||||
def test_g11_get_task_context_with_record():
|
||||
rec = JobRecord(id="j1", session_id="s1", stage_chain=["identify", "phylogeny"])
|
||||
rec._stage_states = {"identify": {"status": "done"}, "phylogeny": {"status": "running"}}
|
||||
ctx = get_task_context(rec)
|
||||
assert ctx["session_id"] == "s1" and ctx["job_id"] == "j1"
|
||||
assert ctx["current_stage"] == "phylogeny"
|
||||
|
||||
|
||||
def test_g11_current_stage():
|
||||
rec = JobRecord(id="j1", stage_chain=["a", "b", "c"])
|
||||
rec._stage_states = {"a": {"status": "done"}, "b": {"status": "done"}, "c": {"status": "running"}}
|
||||
assert current_stage(rec) == "c"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G15 恢复预算 + runner 自动重试
|
||||
# ===========================================================================
|
||||
|
||||
def test_g15_recovery_can_retry_semantics():
|
||||
b = RecoveryBudget(max_same_cause_retries=2)
|
||||
assert b.can_retry("x") is True
|
||||
b.record_retry("x")
|
||||
assert b.can_retry("x") is True
|
||||
b.record_retry("x")
|
||||
assert b.can_retry("x") is False
|
||||
|
||||
|
||||
def test_g15_recovery_decide_non_transient():
|
||||
b = RecoveryBudget()
|
||||
assert b.decide("s", "exit_code") is False
|
||||
assert b.decide("s", "no_manifest") is False
|
||||
|
||||
|
||||
def test_g15_recovery_decide_user_stopped():
|
||||
b = RecoveryBudget()
|
||||
assert b.decide("s", "timeout", user_stopped=True) is False
|
||||
|
||||
|
||||
def test_g15_recovery_decide_resource_gate():
|
||||
b = RecoveryBudget()
|
||||
assert b.decide("s", "exit_code", log_tail="out of memory") is False
|
||||
|
||||
|
||||
def test_g15_runner_auto_retries_transient(db, events, catalog):
|
||||
executor = FakeTimedOutExecutor()
|
||||
|
||||
async def scenario():
|
||||
runner = make_runner(db, events, executor, catalog)
|
||||
record = await runner.create_job(
|
||||
"s1", "t", ["identify"], auto_continue=True, auto_start=False)
|
||||
await runner.run_job(record)
|
||||
return runner, record
|
||||
|
||||
runner, record = _scenario(scenario)
|
||||
assert runner.stage_status(record, "identify") == STAGE_FAILED
|
||||
assert record.status == JOB_FAILED
|
||||
assert len(executor.calls) == 3 # 1 初试 + 2 自动重试
|
||||
@@ -0,0 +1,264 @@
|
||||
"""tests/test_phase_d_contract.py —— Phase D(契约与就绪 + 诊断)测试用例集。
|
||||
|
||||
覆盖(Phase D 验收):G7 / G8 / G9 / G10 / G11 / G14 / G15。
|
||||
|
||||
## 契约说明(开发对齐用)
|
||||
- G14 错误分类:`app.services.bioflow.errors.CATEGORIES`(8 类)+ `classify(code, summary)`。
|
||||
8 类确定性分类:input / scientific_precondition / reference / environment /
|
||||
transient / resource / adapter / artifact。OOM→resource、镜像缺→environment、
|
||||
科学前提→scientific_precondition、参考缺失→reference。
|
||||
- G10 就绪检查:`check_readiness` 四象限(environment/reference/space/budget),
|
||||
缺项返回结构化错误并阻断(阻断 = ready=False)。
|
||||
- G11 任务绑定:`get_task_context` 返回 session_id+job_id+plan_revision+run_id+当前阶段。
|
||||
- G7/G8:`stage_input_roles(stage)` 返回阶段依赖输入角色;`asset_role_satisfied` 判断
|
||||
角色是否有 validated 资产;下游直入(counts→DE)不补跑上游 QC/align/quant。
|
||||
- G9 单一参数 schema:所有入口读 `catalog.get(stage).params`(无多份不一致)。
|
||||
- G15 恢复预算:同因重试 ≤2;收敛条件(scientific_precondition / 无新证据 /
|
||||
用户停止 / 资源门禁)终止重试。
|
||||
|
||||
未实现函数用 `_phase_d_fn` 动态导入,缺失则 skip(开发完成后须全部可导入、无 skip)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.bioflow import errors
|
||||
from app.services.bioflow.catalog import build_catalog
|
||||
|
||||
pytestmark = pytest.mark.bioflow_unit
|
||||
|
||||
|
||||
def _phase_d_fn(module: str, name: str):
|
||||
"""动态取 Phase D 函数;未实现则 skip(开发完成后须全部存在)。"""
|
||||
try:
|
||||
mod = importlib.import_module(module)
|
||||
return getattr(mod, name)
|
||||
except (ImportError, AttributeError) as exc:
|
||||
pytest.skip(f"Phase D 未实现 {module}.{name}: {exc}")
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G14 错误分类(8 类确定性分类)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g14_categories_exactly_eight():
|
||||
"""前置:errors 模块。步骤:读 CATEGORIES。预期:恰 8 类。"""
|
||||
assert set(errors.CATEGORIES) == {
|
||||
"input", "scientific_precondition", "reference", "environment",
|
||||
"transient", "resource", "adapter", "artifact",
|
||||
}
|
||||
|
||||
|
||||
def test_g14_classify_resource():
|
||||
"""前置:DISK_FULL。步骤:classify。预期:category=resource,不可重试。"""
|
||||
cat, diag, retry, actions = errors.classify("DISK_FULL", "")
|
||||
assert cat == "resource"
|
||||
assert diag == "diagnosed"
|
||||
assert retry is False
|
||||
assert isinstance(actions, list)
|
||||
|
||||
|
||||
def test_g14_classify_environment():
|
||||
"""前置:OSS_UNCONFIGURED。步骤:classify。预期:category=environment。"""
|
||||
cat, *_ = errors.classify("OSS_UNCONFIGURED", "")
|
||||
assert cat == "environment"
|
||||
|
||||
|
||||
def test_g14_classify_transient_retryable():
|
||||
"""前置:NETWORK_ERROR。步骤:classify。预期:category=transient,可重试。"""
|
||||
cat, diag, retry, _ = errors.classify("NETWORK_ERROR", "")
|
||||
assert cat == "transient"
|
||||
assert retry is True
|
||||
|
||||
|
||||
def test_g14_classify_input():
|
||||
"""前置:NOT_FOUND。步骤:classify。预期:category=input。"""
|
||||
cat, *_ = errors.classify("NOT_FOUND", "")
|
||||
assert cat == "input"
|
||||
|
||||
|
||||
def test_g14_classify_adapter():
|
||||
"""前置:UNSUPPORTED。步骤:classify。预期:category=adapter。"""
|
||||
cat, *_ = errors.classify("UNSUPPORTED", "")
|
||||
assert cat == "adapter"
|
||||
|
||||
|
||||
def test_g14_classify_artifact():
|
||||
"""前置:ARCHIVE_EXISTS。步骤:classify。预期:category=artifact。"""
|
||||
cat, *_ = errors.classify("ARCHIVE_EXISTS", "")
|
||||
assert cat == "artifact"
|
||||
|
||||
|
||||
def test_g14_classify_oom_keyword_to_resource():
|
||||
"""G14 确定性:OOM→resource。前置:无枚举码、摘要含 out of memory。
|
||||
步骤:classify(None, "container out of memory")。预期:category=resource。"""
|
||||
cat, *_ = errors.classify(None, "container out of memory")
|
||||
assert cat == "resource"
|
||||
|
||||
|
||||
def test_g14_classify_image_keyword_to_environment():
|
||||
"""G14 确定性:镜像缺→environment。前置:摘要含 image。步骤:classify。
|
||||
预期:category=environment。"""
|
||||
cat, *_ = errors.classify(None, "docker image not found")
|
||||
assert cat == "environment"
|
||||
|
||||
|
||||
def test_g14_classify_scientific_precondition():
|
||||
"""G14 确定性:科学前提→scientific_precondition。前置:摘要含科学前提语义。
|
||||
步骤:classify。预期:category=scientific_precondition(开发须补齐该映射)。"""
|
||||
cat, *_ = errors.classify(None, "上游科学前提未满足,无法继续")
|
||||
assert cat == "scientific_precondition"
|
||||
|
||||
|
||||
def test_g14_classify_reference():
|
||||
"""G14 确定性:参考缺失→reference。前置:摘要含参考基因组缺失。
|
||||
步骤:classify。预期:category=reference(开发须补齐该映射)。"""
|
||||
cat, *_ = errors.classify(None, "参考基因组缺失")
|
||||
assert cat == "reference"
|
||||
|
||||
|
||||
def test_g14_classify_category_invariant():
|
||||
"""G14 不变量:任意输入分类结果 category 必在 8 类内。"""
|
||||
samples = [
|
||||
("DISK_FULL", ""), ("NOT_FOUND", ""), (None, "timeout"), (None, "unknown_xyz"),
|
||||
("BAD_SCHEME", ""), ("ASSET_EXISTS", ""),
|
||||
]
|
||||
for code, summary in samples:
|
||||
cat, *_ = errors.classify(code, summary)
|
||||
assert cat in errors.CATEGORIES, f"分类结果 {cat!r} 不在 8 类内"
|
||||
|
||||
|
||||
def test_g14_error_object_carries_category():
|
||||
"""G14 集成:error_object 携带 classify 的 category。"""
|
||||
obj = errors.error_object("DISK_FULL", "磁盘不足")
|
||||
assert obj["category"] == "resource"
|
||||
assert obj["retryable_without_change"] is False
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G9 单一参数 schema(所有入口读 catalog.get(stage).params)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g9_catalog_is_single_param_source(image_map):
|
||||
"""G9。前置:image_map。步骤:两次 build_catalog。预期:参数定义一致(单一来源)。"""
|
||||
c1 = build_catalog(image_map)
|
||||
c2 = build_catalog(image_map)
|
||||
assert c1.get("identify").params == c2.get("identify").params
|
||||
|
||||
|
||||
def test_g9_catalog_params_consistent_shape(image_map):
|
||||
"""G9。前置:image_map。步骤:读 identify 参数。预期:参数为 list[dict],含 name。"""
|
||||
entry = build_catalog(image_map).get("identify")
|
||||
assert isinstance(entry.params, list)
|
||||
if entry.params:
|
||||
assert all("name" in p for p in entry.params)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G10 就绪检查(check_readiness 四象限:环境/参考/空间/预算)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g10_check_readiness_all_ready(tmp_path):
|
||||
"""G10。前置:四象限齐备。步骤:check_readiness。预期:ready=True,四象限均 ok。"""
|
||||
check_readiness = _phase_d_fn("app.services.bioflow.readiness", "check_readiness")
|
||||
r = check_readiness(assets_root=tmp_path, stages=[])
|
||||
assert r["ready"] is True
|
||||
for q in ("environment", "reference", "space", "budget"):
|
||||
assert q in r, f"缺象限 {q}"
|
||||
assert r[q].get("ok") is True
|
||||
|
||||
|
||||
def test_g10_check_readiness_missing_reference(tmp_path):
|
||||
"""G10。前置:参考缺失。步骤:check_readiness。预期:ready=False,结构化错误+阻断。"""
|
||||
check_readiness = _phase_d_fn("app.services.bioflow.readiness", "check_readiness")
|
||||
r = check_readiness(assets_root=tmp_path, stages=["identify"])
|
||||
assert r["ready"] is False
|
||||
# 缺项返回结构化错误(G13 对象)
|
||||
err = r.get("error") or (r.get("reference") or {}).get("error")
|
||||
assert err is not None, "缺项应返回结构化错误对象"
|
||||
for f in ("error_id", "category", "code", "summary", "diagnosis_status"):
|
||||
assert f in err, f"缺字段 {f}"
|
||||
|
||||
|
||||
def test_g10_check_readiness_four_quadrants_keys(tmp_path):
|
||||
"""G10。前置:任意。步骤:check_readiness。预期:返回值含四象限键。"""
|
||||
check_readiness = _phase_d_fn("app.services.bioflow.readiness", "check_readiness")
|
||||
r = check_readiness(assets_root=tmp_path, stages=[])
|
||||
for q in ("environment", "reference", "space", "budget"):
|
||||
assert q in r, f"缺象限键 {q}"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G11 任务绑定(get_task_context)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g11_get_task_context_fields():
|
||||
"""G11。前置:绑定态任务。步骤:get_task_context。预期:返回 5 字段。"""
|
||||
get_task_context = _phase_d_fn("app.services.bioflow.task_context", "get_task_context")
|
||||
ctx = get_task_context()
|
||||
for k in ("session_id", "job_id", "plan_revision", "run_id", "current_stage"):
|
||||
assert k in ctx, f"缺字段 {k}"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G7/G8 角色化资产 + 下游直入
|
||||
# ===========================================================================
|
||||
|
||||
def test_g8_stage_input_roles():
|
||||
"""G8。前置:阶段 rnaseq_de。步骤:stage_input_roles。预期:返回 counts 等输入角色。"""
|
||||
stage_input_roles = _phase_d_fn("app.services.bioflow.stage_assets", "stage_input_roles")
|
||||
roles = stage_input_roles("rnaseq_de")
|
||||
assert isinstance(roles, list)
|
||||
assert roles, "rnaseq_de 应声明输入角色(如 counts)"
|
||||
|
||||
|
||||
def test_g8_role_satisfied_by_validated_asset():
|
||||
"""G8。前置:validated 资产。步骤:asset_role_satisfied。预期:有 validated 资产即满足。"""
|
||||
asset_role_satisfied = _phase_d_fn("app.services.bioflow.stage_assets", "asset_role_satisfied")
|
||||
assert asset_role_satisfied("counts", {"role": "counts", "validated": {"ok": True}}) is True
|
||||
assert asset_role_satisfied("counts", {"role": "counts", "validated": {"ok": False}}) is False
|
||||
|
||||
|
||||
def test_g7_downstream_direct_entry_counts():
|
||||
"""G7。前置:提供 counts 资产做 DE。步骤:下游直入判断。
|
||||
预期:不补跑上游 QC/align/quant(上游角色由 counts 资产满足)。"""
|
||||
downstream_entry_satisfied = _phase_d_fn(
|
||||
"app.services.bioflow.stage_assets", "downstream_entry_satisfied")
|
||||
ok, missing_upstream = downstream_entry_satisfied(
|
||||
"rnaseq_de", {"counts": {"validated": {"ok": True}}})
|
||||
assert ok is True
|
||||
assert missing_upstream == []
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G15 恢复预算(同因重试≤2 + 收敛条件)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g15_same_cause_retry_le_2():
|
||||
"""G15。前置:恢复预算。步骤:同因连续重试。预期:≤2 次后拒绝。"""
|
||||
RecoveryBudget = _phase_d_fn("app.services.bioflow.recovery", "RecoveryBudget")
|
||||
b = RecoveryBudget(max_same_cause_retries=2)
|
||||
assert b.can_retry("cause_x") is True # 第 1 次
|
||||
b.record_retry("cause_x")
|
||||
assert b.can_retry("cause_x") is True # 第 2 次
|
||||
b.record_retry("cause_x")
|
||||
assert b.can_retry("cause_x") is False # 第 3 次拒绝(≤2)
|
||||
|
||||
|
||||
def test_g15_convergence_scientific_precondition():
|
||||
"""G15。前置:scientific_precondition 分类。步骤:判断是否可重试。
|
||||
预期:科学前提类不重试(收敛)。"""
|
||||
cat, diag, retry, _ = errors.classify(None, "科学前提未满足")
|
||||
assert cat == "scientific_precondition"
|
||||
assert retry is False
|
||||
|
||||
|
||||
def test_g15_convergence_resource_gate():
|
||||
"""G15。前置:resource 门禁(如 DISK_FULL)。步骤:classify。
|
||||
预期:资源类不可重试(资源门禁收敛)。"""
|
||||
cat, diag, retry, _ = errors.classify("DISK_FULL", "")
|
||||
assert cat == "resource"
|
||||
assert retry is False
|
||||
@@ -0,0 +1,184 @@
|
||||
"""tests/test_phase_e_versioning.py —— Phase E(版本化修改)测试用例集。
|
||||
|
||||
覆盖(Phase E 验收):G16 / G17 / G18 / G19 / G20。
|
||||
|
||||
## 契约说明(开发对齐用,预期模块 + 函数签名)
|
||||
- G16 版本化身份字段:plan_revision / run_id(已存在于 JobRecord + task_context.get_task_context);
|
||||
新增 asset_id / based_on_revision / action_id,落在
|
||||
`app.services.bioflow.versioning`:
|
||||
new_revision(based_on_revision: str = None) -> dict
|
||||
# 返回 {"revision": str, "based_on_revision": str|None,
|
||||
# "asset_id": str, "action_id": str}
|
||||
make_action_id() -> str # 唯一 action id
|
||||
- G17 影响范围:`versioning.affected_stages(stage_chain: list[str], changed_stage: str) -> list[str]`
|
||||
返回改参数后受影响的下游阶段(只重跑受影响节点;含 changed_stage 之后的所有下游)。
|
||||
- G18 历史不覆盖:`versioning.RevisionStore(root: str|Path)`
|
||||
write(revision: str, key: str, content: str) -> Path # 写到 revision 独立路径,不覆盖旧 revision
|
||||
read(revision: str, key: str) -> str | None
|
||||
- G19 防重复:`versioning.ActionDedup()`
|
||||
try_start(action_id: str) -> bool # 首次 True,重复 False(幂等,不重复启动)
|
||||
- G20 执行所有权:`app.services.bioflow.run_owner`:
|
||||
claim(run_id: str, owner_id: str) -> bool # 首次 True(取得所有权),重复/他人 False
|
||||
release(run_id: str, owner_id: str) -> bool
|
||||
stop(run_id: str, owner_id: str) -> bool # 终止实际执行(非仅置状态)
|
||||
resume(run_id: str, owner_id: str) -> bool # 已运行则 False(不重叠)
|
||||
|
||||
未实现函数用 `_phase_e_fn` 动态导入,缺失则 skip(开发完成后须全部存在、无 skip)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.bioflow.runner import JobRecord
|
||||
from app.services.bioflow.task_context import get_task_context
|
||||
|
||||
pytestmark = pytest.mark.bioflow_unit
|
||||
|
||||
|
||||
def _phase_e_fn(module: str, name: str):
|
||||
"""动态取 Phase E 函数/类;未实现则 skip(开发完成后须全部存在)。"""
|
||||
try:
|
||||
mod = importlib.import_module(module)
|
||||
return getattr(mod, name)
|
||||
except (ImportError, AttributeError) as exc:
|
||||
pytest.skip(f"Phase E 未实现 {module}.{name}: {exc}")
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G16 版本化身份(plan_revision / run_id / asset_id / based_on_revision / action_id)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g16_job_record_revision_fields():
|
||||
"""G16。前置:JobRecord。步骤:构造。预期:含 plan_revision + run_id 字段。"""
|
||||
rec = JobRecord(id="j1")
|
||||
assert rec.plan_revision == "1"
|
||||
assert rec.run_id is None
|
||||
|
||||
|
||||
def test_g16_task_context_revision_fields():
|
||||
"""G16。前置:无 record。步骤:get_task_context()。预期:返回 plan_revision + run_id。"""
|
||||
ctx = get_task_context()
|
||||
assert "plan_revision" in ctx
|
||||
assert "run_id" in ctx
|
||||
|
||||
|
||||
def test_g16_new_revision_identity_fields():
|
||||
"""G16。前置:versioning.new_revision。步骤:调用。预期:返回 4 个身份字段。"""
|
||||
new_revision = _phase_e_fn("app.services.bioflow.versioning", "new_revision")
|
||||
rev = new_revision(based_on_revision="1")
|
||||
for k in ("revision", "based_on_revision", "asset_id", "action_id"):
|
||||
assert k in rev, f"缺字段 {k}"
|
||||
assert rev["based_on_revision"] == "1"
|
||||
|
||||
|
||||
def test_g16_action_id_unique():
|
||||
"""G16。前置:versioning.make_action_id。步骤:两次调用。预期:action_id 唯一。"""
|
||||
make_action_id = _phase_e_fn("app.services.bioflow.versioning", "make_action_id")
|
||||
a = make_action_id()
|
||||
b = make_action_id()
|
||||
assert a and b
|
||||
assert a != b
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G17 影响范围(改参数 → 依赖图 → 受影响下游阶段,只重跑受影响节点)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g17_affected_stages_downstream():
|
||||
"""G17。前置:链 identify→phylogeny→motif。步骤:affected_stages(changed=identify)。
|
||||
预期:返回下游 phylogeny/motif(不含已完成的 identify 之外无关节点)。"""
|
||||
affected_stages = _phase_e_fn("app.services.bioflow.versioning", "affected_stages")
|
||||
out = affected_stages(["identify", "phylogeny", "motif"], "identify")
|
||||
assert out == ["phylogeny", "motif"]
|
||||
|
||||
|
||||
def test_g17_affected_stages_no_downstream():
|
||||
"""G17。前置:改末阶段 motif。步骤:affected_stages。预期:无下游,空列表。"""
|
||||
affected_stages = _phase_e_fn("app.services.bioflow.versioning", "affected_stages")
|
||||
out = affected_stages(["identify", "phylogeny", "motif"], "motif")
|
||||
assert out == []
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G18 历史不覆盖(新 revision 不覆盖旧 revision 的 manifest/产物/参数)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g18_revision_store_preserves_old(tmp_path):
|
||||
"""G18。前置:RevisionStore。步骤:rev1 写 key,rev2 写同名 key。
|
||||
预期:rev1 内容不被 rev2 覆盖。"""
|
||||
RevisionStore = _phase_e_fn("app.services.bioflow.versioning", "RevisionStore")
|
||||
store = RevisionStore(tmp_path)
|
||||
store.write("rev1", "params.json", '{"cpu": "2"}')
|
||||
store.write("rev2", "params.json", '{"cpu": "8"}')
|
||||
assert store.read("rev1", "params.json") == '{"cpu": "2"}'
|
||||
assert store.read("rev2", "params.json") == '{"cpu": "8"}'
|
||||
|
||||
|
||||
def test_g18_revision_paths_isolated(tmp_path):
|
||||
"""G18。前置:RevisionStore。步骤:rev1/rev2 各写产物。
|
||||
预期:落不同路径(物理隔离,不互覆)。"""
|
||||
RevisionStore = _phase_e_fn("app.services.bioflow.versioning", "RevisionStore")
|
||||
store = RevisionStore(tmp_path)
|
||||
p1 = store.write("rev1", "final.txt", "v1")
|
||||
p2 = store.write("rev2", "final.txt", "v2")
|
||||
assert str(p1) != str(p2)
|
||||
assert p1.read_text() == "v1"
|
||||
assert p2.read_text() == "v2"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G19 防重复(action_id 幂等,重复提交不重复启动)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g19_action_dedup_idempotent():
|
||||
"""G19。前置:ActionDedup。步骤:同 action_id 两次 try_start。
|
||||
预期:首次 True,第二次 False(幂等)。"""
|
||||
ActionDedup = _phase_e_fn("app.services.bioflow.versioning", "ActionDedup")
|
||||
dedup = ActionDedup()
|
||||
assert dedup.try_start("act-1") is True
|
||||
assert dedup.try_start("act-1") is False
|
||||
|
||||
|
||||
def test_g19_duplicate_action_not_restarted():
|
||||
"""G19。前置:ActionDedup。步骤:两个不同 action_id 各自启动,重复一个。
|
||||
预期:不同 id 各自 True;重复 id 不重复启动。"""
|
||||
ActionDedup = _phase_e_fn("app.services.bioflow.versioning", "ActionDedup")
|
||||
dedup = ActionDedup()
|
||||
assert dedup.try_start("act-1") is True
|
||||
assert dedup.try_start("act-2") is True
|
||||
assert dedup.try_start("act-1") is False # 不重复启动
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G20 执行所有权(stop 终止实际执行 / resume 不重叠 / 重启不重复接管)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g20_claim_single_owner():
|
||||
"""G20。前置:run_owner.claim。步骤:owner A 取得 run,owner B 再 claim。
|
||||
预期:A True,B False(单一所有权,不重复接管)。"""
|
||||
claim = _phase_e_fn("app.services.bioflow.run_owner", "claim")
|
||||
assert claim("run-1", "owner-A") is True
|
||||
assert claim("run-1", "owner-B") is False
|
||||
|
||||
|
||||
def test_g20_resume_no_overlap():
|
||||
"""G20。前置:run_owner.resume。步骤:已运行 run 再 resume。
|
||||
预期:False(不重叠执行)。"""
|
||||
claim = _phase_e_fn("app.services.bioflow.run_owner", "claim")
|
||||
resume = _phase_e_fn("app.services.bioflow.run_owner", "resume")
|
||||
assert claim("run-2", "owner-A") is True
|
||||
assert resume("run-2", "owner-A") is False
|
||||
|
||||
|
||||
def test_g20_stop_terminates_execution():
|
||||
"""G20。前置:run_owner.claim + stop。步骤:claim 后 stop。
|
||||
预期:stop 返回 True(实际终止),随后 release 允许重新 claim。"""
|
||||
claim = _phase_e_fn("app.services.bioflow.run_owner", "claim")
|
||||
stop = _phase_e_fn("app.services.bioflow.run_owner", "stop")
|
||||
release = _phase_e_fn("app.services.bioflow.run_owner", "release")
|
||||
assert claim("run-3", "owner-A") is True
|
||||
assert stop("run-3", "owner-A") is True
|
||||
assert release("run-3", "owner-A") is True
|
||||
assert claim("run-3", "owner-B") is True # 释放后可被重新接管
|
||||
@@ -0,0 +1,97 @@
|
||||
"""tests/test_phase_f.py —— Phase F(交付)测试用例集。
|
||||
|
||||
覆盖:G21 交付清单(delivery_validate/publish)、G22 终报告范围(classify_report_scope)、
|
||||
G23 资源预算门禁(check_download_budget / check_disk_available)。
|
||||
契约测试见 test_phase_f_delivery.py;本文件补契约未直接断言的点(publish / build_report)。
|
||||
分层标记 -m bioflow_unit。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.bioflow.delivery import (
|
||||
build_report,
|
||||
check_disk_available,
|
||||
check_download_budget,
|
||||
classify_report_scope,
|
||||
delivery_validate,
|
||||
publish_delivery,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.bioflow_unit
|
||||
|
||||
|
||||
def _complete() -> dict:
|
||||
return {
|
||||
"selected_stages": ["identify", "phylogeny"],
|
||||
"run_id": "run-1",
|
||||
"input_sources": ["/data/genome.pep.fa"],
|
||||
"output_sources": ["/data/final/Ft.SPL.final_members.id"],
|
||||
"config": {"cpu": "8"},
|
||||
"required_results": ["final_members.id"],
|
||||
"validation_values": {"final_members": 24},
|
||||
"review_status": "approved",
|
||||
}
|
||||
|
||||
|
||||
# G21 补充:publish 幂等/阻断
|
||||
|
||||
def test_g21_publish_valid():
|
||||
m = _complete()
|
||||
r = publish_delivery(m)
|
||||
assert r["ok"] is True and r["published"] is True
|
||||
assert m["review_status"] == "approved"
|
||||
|
||||
|
||||
def test_g21_publish_invalid_blocked():
|
||||
m = _complete()
|
||||
m["run_id"] = ""
|
||||
r = publish_delivery(m)
|
||||
assert r["ok"] is False and r["published"] is False
|
||||
|
||||
|
||||
def test_g21_validate_checks_eight_keys():
|
||||
r = delivery_validate(_complete())
|
||||
assert set(r["checks"]) == {
|
||||
"selected_stages", "run_id", "input_sources", "output_sources",
|
||||
"config", "required_results", "validation_values", "review_status",
|
||||
}
|
||||
|
||||
|
||||
# G22 补充:五分类边界
|
||||
|
||||
def test_g22_classify_five_categories():
|
||||
assert classify_report_scope("done", "user") == "user_provided"
|
||||
assert classify_report_scope("done", "reuse") == "reused"
|
||||
assert classify_report_scope("done", None) == "user_provided"
|
||||
assert classify_report_scope("waiting", None) == "not_executed"
|
||||
assert classify_report_scope("queued", None) == "not_executed"
|
||||
assert classify_report_scope("skipped", None) == "skipped"
|
||||
assert classify_report_scope("failed", None) == "failed"
|
||||
assert classify_report_scope("partial_success", None) == "not_executed"
|
||||
|
||||
|
||||
# G22 补充:终报告结构
|
||||
|
||||
def test_g22_build_report_structure():
|
||||
m = _complete()
|
||||
report = build_report(m, stage_sources={"identify": "reused",
|
||||
"phylogeny": "failed"})
|
||||
for k in ("execution_scope", "data_sources", "qc", "scientific_preconditions",
|
||||
"failed_skipped_uncovered", "artifacts"):
|
||||
assert k in report, f"缺报告段 {k}"
|
||||
assert report["failed_skipped_uncovered"]["failed"] == ["phylogeny"]
|
||||
assert report["artifacts"] == ["final_members.id"]
|
||||
|
||||
|
||||
# G23 补充
|
||||
|
||||
def test_g23_download_budget_boundary():
|
||||
assert check_download_budget(100, 100)["ok"] is True # 恰好等于不超
|
||||
assert check_download_budget(101, 100)["ok"] is False
|
||||
|
||||
|
||||
def test_g23_disk_available_ok(tmp_path):
|
||||
r = check_disk_available(tmp_path, 1)
|
||||
assert r["ok"] is True
|
||||
assert r["free_bytes"] > 0
|
||||
@@ -0,0 +1,187 @@
|
||||
"""tests/test_phase_f_delivery.py —— Phase F(交付)测试用例集。
|
||||
|
||||
覆盖(Phase F 验收):G21 / G22 / G23。
|
||||
|
||||
## 契约说明(开发对齐用,预期模块 + 函数签名)
|
||||
模块:`app.services.bioflow.delivery`
|
||||
|
||||
- G21 交付清单完整性:
|
||||
delivery_validate(manifest: dict) -> dict
|
||||
manifest 八项:selected_stages / run_id / input_sources / output_sources /
|
||||
config / required_results / validation_values / review_status。
|
||||
返回 {"ok": bool, "checks": {item: bool},
|
||||
"incomplete": [{"item": str, "reason": str, "error": {G13 对象}}]}
|
||||
任一项缺失 → ok=False + incomplete 记录 item/reason/结构化错误(未完成原因)。
|
||||
- G22 终报告范围:
|
||||
classify_report_scope(status: str, source: str | None = None) -> str
|
||||
status 终态分类:failed→"failed"、skipped→"skipped"、
|
||||
非终态(waiting/queued/created/await_*)→"not_executed";
|
||||
done + source="reuse"→"reused";done + source="user"(或空)→"user_provided"。
|
||||
- G23 资源预算:
|
||||
check_download_budget(size_bytes: int, budget_bytes: int) -> dict
|
||||
# 超限 → {"ok": False, "code": "DOWNLOAD_BUDGET_EXCEEDED", "error": {...}}
|
||||
check_disk_available(dest_dir, needed_bytes: int) -> dict
|
||||
# 不足 → {"ok": False, "code": "DISK_FULL", "error": {...}}
|
||||
两者阻断(ok=False)并返回结构化错误。
|
||||
|
||||
未实现函数用 `_phase_f_fn` 动态导入,缺失则 skip(开发完成后须全部存在、无 skip)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.bioflow_unit
|
||||
|
||||
|
||||
def _phase_f_fn(module: str, name: str):
|
||||
"""动态取 Phase F 函数;未实现则 skip(开发完成后须全部存在)。"""
|
||||
try:
|
||||
mod = importlib.import_module(module)
|
||||
return getattr(mod, name)
|
||||
except (ImportError, AttributeError) as exc:
|
||||
pytest.skip(f"Phase F 未实现 {module}.{name}: {exc}")
|
||||
|
||||
|
||||
def _complete_manifest() -> dict:
|
||||
return {
|
||||
"selected_stages": ["identify", "phylogeny"],
|
||||
"run_id": "run-1",
|
||||
"input_sources": ["/data/genome.pep.fa"],
|
||||
"output_sources": ["/data/final/Ft.SPL.final_members.id"],
|
||||
"config": {"cpu": "8"},
|
||||
"required_results": ["final_members.id"],
|
||||
"validation_values": {"final_members": 24},
|
||||
"review_status": "approved",
|
||||
}
|
||||
|
||||
|
||||
def _g13_fields():
|
||||
return ("error_id", "category", "code", "summary", "diagnosis_status")
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G21 交付清单完整性(delivery validate)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g21_delivery_validate_complete():
|
||||
"""G21。前置:八项齐备 manifest。步骤:delivery_validate。预期:ok=True,incomplete 空。"""
|
||||
delivery_validate = _phase_f_fn("app.services.bioflow.delivery", "delivery_validate")
|
||||
r = delivery_validate(_complete_manifest())
|
||||
assert r["ok"] is True
|
||||
assert r.get("incomplete") == []
|
||||
|
||||
|
||||
def test_g21_delivery_validate_missing_run_id():
|
||||
"""G21。前置:缺 run_id。步骤:delivery_validate。预期:ok=False,incomplete 含 run_id + 结构化错误。"""
|
||||
delivery_validate = _phase_f_fn("app.services.bioflow.delivery", "delivery_validate")
|
||||
m = _complete_manifest()
|
||||
m["run_id"] = ""
|
||||
r = delivery_validate(m)
|
||||
assert r["ok"] is False
|
||||
items = {i["item"]: i for i in r["incomplete"]}
|
||||
assert "run_id" in items
|
||||
err = items["run_id"].get("error")
|
||||
assert err is not None
|
||||
for f in _g13_fields():
|
||||
assert f in err, f"缺字段 {f}"
|
||||
|
||||
|
||||
def test_g21_delivery_validate_missing_required_results():
|
||||
"""G21。前置:缺 required_results。步骤:delivery_validate。预期:ok=False,未完成原因含 required_results。"""
|
||||
delivery_validate = _phase_f_fn("app.services.bioflow.delivery", "delivery_validate")
|
||||
m = _complete_manifest()
|
||||
m["required_results"] = []
|
||||
r = delivery_validate(m)
|
||||
assert r["ok"] is False
|
||||
items = {i["item"]: i for i in r["incomplete"]}
|
||||
assert "required_results" in items
|
||||
assert items["required_results"].get("reason")
|
||||
|
||||
|
||||
def test_g21_delivery_validate_checks_eight_items():
|
||||
"""G21。前置:完整 manifest。步骤:delivery_validate。预期:checks 覆盖八项。"""
|
||||
delivery_validate = _phase_f_fn("app.services.bioflow.delivery", "delivery_validate")
|
||||
r = delivery_validate(_complete_manifest())
|
||||
checks = r.get("checks")
|
||||
assert checks is not None, "应返回 checks 覆盖八项"
|
||||
for k in ("selected_stages", "run_id", "input_sources", "output_sources",
|
||||
"config", "required_results", "validation_values", "review_status"):
|
||||
assert k in checks, f"checks 缺项 {k}"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G22 终报告范围(用户提供 / 历史复用 / 未执行 / 跳过 / 失败)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g22_report_scope_user_provided():
|
||||
"""G22。前置:done + source=user。步骤:classify_report_scope。预期:user_provided。"""
|
||||
fn = _phase_f_fn("app.services.bioflow.delivery", "classify_report_scope")
|
||||
assert fn("done", source="user") == "user_provided"
|
||||
|
||||
|
||||
def test_g22_report_scope_reused():
|
||||
"""G22。前置:done + source=reuse。步骤:classify_report_scope。预期:reused。"""
|
||||
fn = _phase_f_fn("app.services.bioflow.delivery", "classify_report_scope")
|
||||
assert fn("done", source="reuse") == "reused"
|
||||
|
||||
|
||||
def test_g22_report_scope_not_executed():
|
||||
"""G22。前置:非终态 waiting。步骤:classify_report_scope。预期:not_executed。"""
|
||||
fn = _phase_f_fn("app.services.bioflow.delivery", "classify_report_scope")
|
||||
assert fn("waiting", source=None) == "not_executed"
|
||||
|
||||
|
||||
def test_g22_report_scope_skipped():
|
||||
"""G22。前置:skipped。步骤:classify_report_scope。预期:skipped。"""
|
||||
fn = _phase_f_fn("app.services.bioflow.delivery", "classify_report_scope")
|
||||
assert fn("skipped", source=None) == "skipped"
|
||||
|
||||
|
||||
def test_g22_report_scope_failed():
|
||||
"""G22。前置:failed。步骤:classify_report_scope。预期:failed。"""
|
||||
fn = _phase_f_fn("app.services.bioflow.delivery", "classify_report_scope")
|
||||
assert fn("failed", source=None) == "failed"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# G23 资源预算(下载预算超限 / 磁盘不足阻断,结构化错误)
|
||||
# ===========================================================================
|
||||
|
||||
def test_g23_download_budget_exceeded():
|
||||
"""G23。前置:下载超预算。步骤:check_download_budget。预期:ok=False + 结构化错误阻断。"""
|
||||
fn = _phase_f_fn("app.services.bioflow.delivery", "check_download_budget")
|
||||
r = fn(size_bytes=1024 * 1024 * 1024, budget_bytes=100)
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "DOWNLOAD_BUDGET_EXCEEDED"
|
||||
err = r.get("error")
|
||||
assert err is not None
|
||||
for f in _g13_fields():
|
||||
assert f in err, f"缺字段 {f}"
|
||||
|
||||
|
||||
def test_g23_download_budget_ok():
|
||||
"""G23。前置:下载未超预算。步骤:check_download_budget。预期:ok=True。"""
|
||||
fn = _phase_f_fn("app.services.bioflow.delivery", "check_download_budget")
|
||||
r = fn(size_bytes=100, budget_bytes=1024)
|
||||
assert r["ok"] is True
|
||||
|
||||
|
||||
def test_g23_disk_insufficient_blocks(tmp_path):
|
||||
"""G23。前置:磁盘不足(1EB)。步骤:check_disk_available。预期:ok=False + 结构化错误。"""
|
||||
fn = _phase_f_fn("app.services.bioflow.delivery", "check_disk_available")
|
||||
r = fn(tmp_path, needed_bytes=10 ** 18)
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "DISK_FULL"
|
||||
err = r.get("error")
|
||||
assert err is not None
|
||||
for f in _g13_fields():
|
||||
assert f in err, f"缺字段 {f}"
|
||||
|
||||
|
||||
def test_g23_disk_sufficient_ok(tmp_path):
|
||||
"""G23。前置:磁盘充足(1 字节)。步骤:check_disk_available。预期:ok=True。"""
|
||||
fn = _phase_f_fn("app.services.bioflow.delivery", "check_disk_available")
|
||||
r = fn(tmp_path, needed_bytes=1)
|
||||
assert r["ok"] is True
|
||||
@@ -0,0 +1,455 @@
|
||||
"""tests/test_source_fetcher.py —— 源文件拉取(web/OSS/局域网)+ 磁盘余量 + provenance。
|
||||
|
||||
分层标记 -m bioflow_unit。覆盖(S0 验收):
|
||||
- web http 流式下载(本地 http.server 实测 httpx 通道)+ provenance 落盘。
|
||||
- 协议白名单(ftp:// 拒绝 BAD_SCHEME)。
|
||||
- oss:// 无 endpoint 拒绝 / 预签名 URL 直通 web / endpoint 解析。
|
||||
- 局域网本机路径直读;远端 host:/path 解析。
|
||||
- 磁盘余量不足 DISK_FULL(check_disk_space 纯函数)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import http.server
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.bioflow.source_fetcher import (
|
||||
FetchError,
|
||||
_parse_lan_remote,
|
||||
_resolve_oss_url,
|
||||
check_disk_space,
|
||||
fetch_lan,
|
||||
fetch_source,
|
||||
fetch_web,
|
||||
)
|
||||
|
||||
|
||||
def _run(coro):
|
||||
return asyncio.new_event_loop().run_until_complete(coro)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 本地 http 服务器(测 httpx 通道,不依赖外网)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.fixture()
|
||||
def http_url(tmp_path):
|
||||
content = b">seq1\nACGTACGT\n>seq2\nTGCATGCA\n"
|
||||
(tmp_path / "test.fa").write_bytes(content)
|
||||
handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=str(tmp_path))
|
||||
server = http.server.HTTPServer(("127.0.0.1", 0), handler)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
yield f"http://127.0.0.1:{server.server_address[1]}/test.fa"
|
||||
server.shutdown()
|
||||
|
||||
|
||||
def test_fetch_web_download_and_provenance(http_url, tmp_path):
|
||||
out = tmp_path / "out"
|
||||
r = _run(fetch_web(http_url, out))
|
||||
assert r["ok"] is True
|
||||
assert r["name"] == "test.fa"
|
||||
assert r["size"] > 0
|
||||
assert r["kind"] == "web"
|
||||
assert r["provenance"]["source"] == http_url
|
||||
assert len(r["provenance"]["sha256"]) == 64
|
||||
assert (out / "test.fa").read_text().startswith(">seq1")
|
||||
# sidecar provenance
|
||||
assert (out / "test.fa.provenance.json").is_file()
|
||||
|
||||
|
||||
def test_fetch_web_http_error(http_url, tmp_path):
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_web(http_url + "/missing", tmp_path / "out"))
|
||||
assert ei.value.code == "HTTP_ERROR"
|
||||
|
||||
|
||||
def test_fetch_source_bad_scheme(tmp_path):
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_source("ftp://host/x.fa", tmp_path))
|
||||
assert ei.value.code == "BAD_SCHEME"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# OSS
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_oss_url_no_endpoint_rejected():
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_resolve_oss_url("oss://bucket/key.fa")
|
||||
assert ei.value.code == "OSS_UNCONFIGURED"
|
||||
|
||||
|
||||
def test_oss_url_endpoint_resolve(monkeypatch):
|
||||
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-cn-hangzhou.aliyuncs.com")
|
||||
url = _resolve_oss_url("oss://seq-bucket/data/x.fa")
|
||||
assert url == "https://seq-bucket.oss-cn-hangzhou.aliyuncs.com/data/x.fa"
|
||||
|
||||
|
||||
def test_oss_presigned_passthrough():
|
||||
url = _resolve_oss_url("https://bucket.oss-cn-hangzhou.aliyuncs.com/x.fa?sig=abc")
|
||||
assert url.startswith("https://")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 局域网 / 本机路径
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_parse_lan_remote():
|
||||
assert _parse_lan_remote("192.168.1.5:/data/x.fa") == ("192.168.1.5", "/data/x.fa")
|
||||
assert _parse_lan_remote("xzq@192.168.1.5:/data/x.fa") == ("xzq@192.168.1.5", "/data/x.fa")
|
||||
assert _parse_lan_remote("/local/path/x.fa") is None
|
||||
assert _parse_lan_remote("http://x/y") is None
|
||||
|
||||
|
||||
def test_fetch_lan_local_path(tmp_path):
|
||||
f = tmp_path / "genome.fa"
|
||||
f.write_text(">g1\nACGT\n")
|
||||
r = _run(fetch_lan(str(f), tmp_path / "out"))
|
||||
assert r["ok"] is True
|
||||
assert r["kind"] == "lan"
|
||||
assert r["path"] == str(f)
|
||||
assert r["provenance"]["sha256"]
|
||||
|
||||
|
||||
def test_fetch_lan_missing_path(tmp_path):
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("/no/such/path.fa", tmp_path))
|
||||
assert ei.value.code == "PATH_NOT_EXIST"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 磁盘余量
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_check_disk_space_insufficient(tmp_path):
|
||||
ok, free = check_disk_space(tmp_path, 10 ** 18) # 1EB,必不足
|
||||
assert ok is False
|
||||
assert free < 10 ** 18
|
||||
|
||||
|
||||
def test_check_disk_space_sufficient(tmp_path):
|
||||
ok, _free = check_disk_space(tmp_path, 1) # 1 字节
|
||||
assert ok is True
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强(1)web:BAD_SCHEME / 磁盘满 DISK_FULL / 断点 TRUNCATED / NETWORK_ERROR
|
||||
# ===========================================================================
|
||||
|
||||
def test_fetch_web_bad_scheme(tmp_path):
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_web("ftp://host/x.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "BAD_SCHEME"
|
||||
|
||||
|
||||
def test_fetch_web_disk_full(http_url, tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
monkeypatch.setattr(sf, "check_disk_space", lambda dest_dir, needed: (False, 0))
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_web(http_url, tmp_path / "out"))
|
||||
assert ei.value.code == "DISK_FULL"
|
||||
|
||||
|
||||
def test_fetch_web_truncated(tmp_path, monkeypatch):
|
||||
"""断点/不完整下载:aiter_bytes 实收字节 < Content-Length → TRUNCATED。
|
||||
|
||||
用 fake httpx 确定性触发(真实服务器下 httpx 会因短读抛 HTTPError)。
|
||||
"""
|
||||
import types as _types
|
||||
|
||||
import httpx as real_httpx
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
|
||||
class FakeResp:
|
||||
status_code = 200
|
||||
headers = {"content-length": "100"}
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
async def aiter_bytes(self, chunk_size):
|
||||
yield b"short"
|
||||
|
||||
class FakeStream:
|
||||
async def __aenter__(self):
|
||||
return FakeResp()
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, follow_redirects=True):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
def stream(self, method, url, timeout=None):
|
||||
return FakeStream()
|
||||
|
||||
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
|
||||
AsyncClient=FakeClient, Timeout=real_httpx.Timeout, HTTPError=real_httpx.HTTPError,
|
||||
))
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_web("http://x/y.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "TRUNCATED"
|
||||
|
||||
|
||||
def test_fetch_web_network_error(tmp_path, monkeypatch):
|
||||
"""网络错误:httpx.HTTPError → NETWORK_ERROR(fake 确定性触发)。"""
|
||||
import types as _types
|
||||
|
||||
import httpx as real_httpx
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
|
||||
class FailingStream:
|
||||
async def __aenter__(self):
|
||||
raise real_httpx.HTTPError("connection reset")
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
class FailingClient:
|
||||
def __init__(self, follow_redirects=True):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
def stream(self, method, url, timeout=None):
|
||||
return FailingStream()
|
||||
|
||||
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
|
||||
AsyncClient=FailingClient, Timeout=real_httpx.Timeout, HTTPError=real_httpx.HTTPError,
|
||||
))
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_web("http://x/y.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "NETWORK_ERROR"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强(2)OSS:BAD_OSS / endpoint 解析(内网 http)/ fetch_oss 走 web + kind 修正
|
||||
# ===========================================================================
|
||||
|
||||
def test_oss_url_bad_format():
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_resolve_oss_url("oss://onlybucket")
|
||||
assert ei.value.code == "BAD_OSS"
|
||||
|
||||
|
||||
def test_oss_url_intranet_endpoint_http(monkeypatch):
|
||||
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-internal.example.com")
|
||||
url = _resolve_oss_url("oss://seq-bucket/data/x.fa")
|
||||
assert url == "http://seq-bucket.oss-internal.example.com/data/x.fa"
|
||||
|
||||
|
||||
def test_fetch_oss_marks_kind_and_provenance(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
from pathlib import Path as _Path
|
||||
|
||||
async def fake_web(url, dest_dir, timeout=None):
|
||||
dest = _Path(dest_dir) / "x.fa"
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(">x\nACGT\n")
|
||||
return {"ok": True, "path": str(dest), "name": "x.fa", "size": 4,
|
||||
"kind": "web", "provenance": {"source": url, "kind": "web",
|
||||
"sha256": "0" * 64, "fetched_at": "t"}}
|
||||
|
||||
monkeypatch.setattr(sf, "fetch_web", fake_web)
|
||||
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-cn-hangzhou.aliyuncs.com")
|
||||
r = _run(sf.fetch_oss("oss://b/k.fa", tmp_path / "out"))
|
||||
assert r["ok"] is True
|
||||
assert r["kind"] == "oss"
|
||||
assert r["provenance"]["kind"] == "oss"
|
||||
assert r["provenance"]["oss_spec"] == "oss://b/k.fa"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强(3)fetch_source:EMPTY_SOURCE / oss 路由 / aliyuncs 预签名路由
|
||||
# ===========================================================================
|
||||
|
||||
def test_fetch_source_empty(tmp_path):
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_source("", tmp_path))
|
||||
assert ei.value.code == "EMPTY_SOURCE"
|
||||
|
||||
|
||||
def test_fetch_source_oss_routing(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
calls = []
|
||||
|
||||
async def fake_oss(spec, dest_dir, timeout=None):
|
||||
calls.append(spec)
|
||||
return {"ok": True, "kind": "oss"}
|
||||
|
||||
monkeypatch.setattr(sf, "fetch_oss", fake_oss)
|
||||
r = _run(sf.fetch_source("oss://bucket/k.fa", tmp_path))
|
||||
assert r["kind"] == "oss"
|
||||
assert calls == ["oss://bucket/k.fa"]
|
||||
|
||||
|
||||
def test_fetch_source_aliyuncs_presigned_routes_to_oss(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
calls = []
|
||||
|
||||
async def fake_oss(spec, dest_dir, timeout=None):
|
||||
calls.append(spec)
|
||||
return {"ok": True, "kind": "oss"}
|
||||
|
||||
monkeypatch.setattr(sf, "fetch_oss", fake_oss)
|
||||
presigned = "https://bucket.aliyuncs.com/x.fa?sig=abc"
|
||||
r = _run(sf.fetch_source(presigned, tmp_path))
|
||||
assert r["kind"] == "oss"
|
||||
assert calls == [presigned]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强(4)局域网:目录 IS_DIR / scp 拉取 / sshpass 缺失 / scp 失败 / 解析边界
|
||||
# ===========================================================================
|
||||
|
||||
def test_fetch_lan_dir_rejected(tmp_path):
|
||||
d = tmp_path / "adir"
|
||||
d.mkdir()
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan(str(d), tmp_path / "out"))
|
||||
assert ei.value.code == "IS_DIR"
|
||||
|
||||
|
||||
def test_parse_lan_remote_relative_rejected():
|
||||
assert _parse_lan_remote("host:relative/path") is None
|
||||
|
||||
|
||||
def test_fetch_lan_scp_success(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
from pathlib import Path as _Path
|
||||
|
||||
async def fake_scp(*args, **kwargs):
|
||||
dest = _Path(args[-1])
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(">g\nACGT\n")
|
||||
|
||||
class P:
|
||||
returncode = 0
|
||||
|
||||
async def communicate(self):
|
||||
return (b"", b"")
|
||||
|
||||
return P()
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
|
||||
r = _run(fetch_lan("xzq@192.168.1.5:/data/genome.fa", tmp_path / "out",
|
||||
credentials={"user": "xzq"}))
|
||||
assert r["ok"] is True
|
||||
assert r["kind"] == "lan"
|
||||
assert r["name"] == "genome.fa"
|
||||
|
||||
|
||||
def test_fetch_lan_scp_sshpass_missing(tmp_path, monkeypatch):
|
||||
import shutil
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
monkeypatch.setattr(shutil, "which", lambda name: None)
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("host:/data/x.fa", tmp_path / "out", credentials={"password": "p"}))
|
||||
assert ei.value.code == "SSHPASS_MISSING"
|
||||
|
||||
|
||||
def test_fetch_lan_scp_failed(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
|
||||
async def fake_scp(*args, **kwargs):
|
||||
class P:
|
||||
returncode = 1
|
||||
|
||||
async def communicate(self):
|
||||
return (b"", b"scp: no such file")
|
||||
|
||||
return P()
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "LAN_ERROR"
|
||||
|
||||
|
||||
def test_scp_fetch_remote_uses_port_and_user(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
from pathlib import Path as _Path
|
||||
captured = {}
|
||||
|
||||
async def fake_scp(*args, **kwargs):
|
||||
captured["args"] = list(args)
|
||||
dest = _Path(args[-1])
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(">g\nACGT\n")
|
||||
|
||||
class P:
|
||||
returncode = 0
|
||||
|
||||
async def communicate(self):
|
||||
return (b"", b"")
|
||||
|
||||
return P()
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
|
||||
_run(fetch_lan("host:/data/x.fa", tmp_path / "out",
|
||||
credentials={"user": "u", "port": "2222"}))
|
||||
# scp -P 2222 u@host:/data/x.fa <dest>
|
||||
assert captured["args"][1] == "-P"
|
||||
assert captured["args"][2] == "2222"
|
||||
assert captured["args"][-2] == "u@host:/data/x.fa"
|
||||
|
||||
|
||||
def test_fetch_lan_scp_timeout(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
|
||||
async def fake_scp(*args, **kwargs):
|
||||
class P:
|
||||
returncode = 0
|
||||
|
||||
async def communicate(self):
|
||||
return (b"", b"")
|
||||
|
||||
return P()
|
||||
|
||||
async def fake_wait_for(coro, timeout=None):
|
||||
coro.close() # 未 await 的 communicate 协程关闭,避免 never-awaited 警告
|
||||
raise asyncio.TimeoutError()
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
|
||||
monkeypatch.setattr(asyncio, "wait_for", fake_wait_for)
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "LAN_TIMEOUT"
|
||||
|
||||
|
||||
def test_fetch_lan_scp_no_file(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
|
||||
async def fake_scp(*args, **kwargs):
|
||||
# returncode 0 但未产出目标文件
|
||||
class P:
|
||||
returncode = 0
|
||||
|
||||
async def communicate(self):
|
||||
return (b"", b"")
|
||||
|
||||
return P()
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "LAN_ERROR"
|
||||
Executable
+19
@@ -0,0 +1,19 @@
|
||||
#!/usr/bin/env bash
|
||||
# bioflow_service/uninstall_skill.sh —— 卸载 bioflow-analysis skill(G12)。
|
||||
#
|
||||
# 用法:bash uninstall_skill.sh [skills_dir]
|
||||
# - 移除 skills_dir/bioflow-analysis/
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
SKILL_NAME="bioflow-analysis"
|
||||
SKILLS_DIR="${1:-$HERE/../.claude/skills}"
|
||||
|
||||
DEST="$SKILLS_DIR/$SKILL_NAME"
|
||||
if [ -d "$DEST" ]; then
|
||||
rm -rf "$DEST"
|
||||
echo "[uninstall] removed $DEST"
|
||||
else
|
||||
echo "[uninstall] $DEST 不存在(已卸载)"
|
||||
fi
|
||||
echo "DONE"
|
||||
Executable
+91
@@ -0,0 +1,91 @@
|
||||
#!/bin/bash
|
||||
# deploy/bioflow_switch_restart.sh —— 切分支后重启生信相关服务并验证
|
||||
#
|
||||
# 背景:QA(8091)/前端(5180)/bioflow_service(8098) 都是「启动时刻」快照,
|
||||
# 切分支后不重启就会跑旧分支的代码/配置(曾导致「设置面板无生信开关」+
|
||||
# 「规划降级」两个问题)。本脚本把「切分支→重启→验证」固化,避免再踩。
|
||||
#
|
||||
# 用法:bash deploy/bioflow_switch_restart.sh [分支名]
|
||||
# 分支名缺省 feature/bioflow-skill-orchestration
|
||||
#
|
||||
# 重启:QA(8091) + bioflow_service(8098) + 前端 vite(5180)
|
||||
# BFF(8090) 不重启:/api/capabilities 每次请求动态拉 QA registry,QA 重启后自动生效
|
||||
# 验证:服务健康 + QA registry 生信能力 + 前端 SettingsPanel 生信三开关
|
||||
#
|
||||
# 关键点(勿删):
|
||||
# - QA 需 source agent/qa/env/prod.env + 强制 QA_V1_ENABLE_BIOFLOW_AGENT=1(熔断开)
|
||||
# - 8098 裸启动即可:main.py 已内置 load_dotenv,自动加载 bioflow_service/.env
|
||||
# - 前端 pkill 用「zh/PlantAI-rag-new/ui/」精确匹配,避免误杀 5181(wjq 仓库) 的 vite
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="/home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new"
|
||||
VENV="/home/xzq/5090/teamWorkspace/zh/plantai-rag-new-venv/bin/python"
|
||||
BRANCH="${1:-feature/bioflow-skill-orchestration}"
|
||||
PASS=0; FAIL=0
|
||||
|
||||
ok() { PASS=$((PASS+1)); echo " ✅ $1"; }
|
||||
bad() { FAIL=$((FAIL+1)); echo " ❌ $1"; }
|
||||
|
||||
echo "======== 1. 切分支 ========"
|
||||
cd "$ROOT" || { echo "仓库目录不存在: $ROOT"; exit 1; }
|
||||
git checkout "$BRANCH" || { echo "切分支失败: $BRANCH"; exit 1; }
|
||||
echo "当前分支: $(git branch --show-current)"
|
||||
|
||||
echo ""
|
||||
echo "======== 2. 重启 QA (8091) ========"
|
||||
pkill -f "scripts.qa_v1_1_73_server" 2>/dev/null; sleep 2
|
||||
cd "$ROOT/agent/qa"
|
||||
set -a; source env/prod.env; set +a
|
||||
export QA_V1_ENABLE_BIOFLOW_AGENT=1 # 生信对话熔断开关,切生信分支必须开
|
||||
nohup "$VENV" -m uvicorn scripts.qa_v1_1_73_server:app --host 127.0.0.1 --port 8091 \
|
||||
> /tmp/plantai-qa-8091.log 2>&1 < /dev/null &
|
||||
disown
|
||||
|
||||
echo ""
|
||||
echo "======== 3. 重启 bioflow_service (8098) ========"
|
||||
pkill -f "app.main:app.*port 8098" 2>/dev/null; sleep 2
|
||||
cd "$ROOT/bioflow_service"
|
||||
nohup "$VENV" -m uvicorn app.main:app --host 127.0.0.1 --port 8098 \
|
||||
> /tmp/plantai-bioflow-8098.log 2>&1 < /dev/null &
|
||||
disown
|
||||
|
||||
echo ""
|
||||
echo "======== 4. 重启前端 vite (5180) ========"
|
||||
pkill -f "zh/PlantAI-rag-new/ui/node_modules/.bin/vite" 2>/dev/null; sleep 2
|
||||
cd "$ROOT/ui"
|
||||
VITE_BFF_TARGET="http://127.0.0.1:8090" setsid npm run dev \
|
||||
> /tmp/plantai-vite.log 2>&1 < /dev/null &
|
||||
disown
|
||||
|
||||
echo ""
|
||||
echo "等待服务启动 (12s)..."
|
||||
sleep 12
|
||||
|
||||
echo ""
|
||||
echo "======== 5. 验证 ========"
|
||||
|
||||
echo "[健康检查]"
|
||||
c=$(curl -s -m 3 -o /dev/null -w "%{http_code}" http://127.0.0.1:8098/docs 2>/dev/null)
|
||||
[ "$c" = "200" ] && ok "8098 bioflow_service /docs=$c" || bad "8098 bioflow_service /docs=$c(见 /tmp/plantai-bioflow-8098.log)"
|
||||
c=$(curl -s -m 3 -o /dev/null -w "%{http_code}" http://127.0.0.1:8091/api/v1/qa-v1-1-73/capabilities/registry 2>/dev/null)
|
||||
[ "$c" = "200" ] && ok "8091 QA registry=$c" || bad "8091 QA registry=$c(见 /tmp/plantai-qa-8091.log)"
|
||||
c=$(curl -s -m 3 -o /dev/null -w "%{http_code}" http://127.0.0.1:5180/ 2>/dev/null)
|
||||
[ "$c" = "200" ] && ok "5180 前端 =$c" || bad "5180 前端 =$c(见 /tmp/plantai-vite.log)"
|
||||
|
||||
echo "[生信能力注册]"
|
||||
REG=$(curl -s -m 5 http://127.0.0.1:8091/api/v1/qa-v1-1-73/capabilities/registry 2>/dev/null)
|
||||
BIO=$(echo "$REG" | grep -o '"name":"bioflow-[a-z-]*"' | wc -l)
|
||||
[ "$BIO" -ge 3 ] && ok "QA registry 生信能力 $BIO 个(期望 3)" || bad "QA registry 生信能力仅 $BIO 个(期望 3)"
|
||||
|
||||
echo "[前端生信三开关]"
|
||||
SW=$(curl -s -m 5 -H "Accept: */*" "http://127.0.0.1:5180/src/components/settings/SettingsPanel.vue" 2>/dev/null | grep -c "settingsBioflow\|bioflowCaps")
|
||||
[ "$SW" -ge 2 ] && ok "前端 SettingsPanel 含「生信分析能力」三开关" || bad "前端 SettingsPanel 无生信三开关(可能旧代码,需硬刷新浏览器)"
|
||||
|
||||
echo ""
|
||||
echo "======== 结果: $PASS 通过 / $FAIL 失败 ========"
|
||||
if [ "$FAIL" -eq 0 ]; then
|
||||
echo "✅ 全部就绪。浏览器请 Ctrl+Shift+R 硬刷新(清缓存),再测生信对话。"
|
||||
else
|
||||
echo "❌ 有失败项,请查看 /tmp/plantai-qa-8091.log /tmp/plantai-bioflow-8098.log /tmp/plantai-vite.log"
|
||||
fi
|
||||
exit "$FAIL"
|
||||
Executable
+141
@@ -0,0 +1,141 @@
|
||||
#!/usr/bin/env bash
|
||||
# deploy/toggle_bioflow.sh —— 生信能力按需装卸(热,不重启 QA)+ bioflow_service 启停 + 注册状态自检。
|
||||
#
|
||||
# 用法:
|
||||
# bash deploy/toggle_bioflow.sh <gene-family|rnaseq|extension|all> <on|off>
|
||||
#
|
||||
# 原理:
|
||||
# - 能力装卸经 BFF POST /api/capabilities/install(透传 QA install_manager,热加载生效不重启 QA)
|
||||
# - on = register(manifest 校验 + 原子写 capabilities.yaml + registry 差分刷新)
|
||||
# - off = unregister(按 name 移除能力段,其余能力不受影响)
|
||||
# - 自检 = GET /api/capabilities 的 installed 数组是否含对应能力
|
||||
# - bioflow_service 启停:全部能力关闭时停 8098;任一开启时启动 8098
|
||||
# - 卸载活跃任务:三表已持久化,服务恢复后 runner.recover 续跑;QA 侧按「生信服务未启动」降级话术
|
||||
set -uo pipefail
|
||||
|
||||
BFF="http://127.0.0.1:8090"
|
||||
BIOFLOW_SERVICE_DIR="/home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new/bioflow_service"
|
||||
PY="/home/xzq/5090/teamWorkspace/zh/plantai-rag-new-venv/bin/python"
|
||||
BIOFLOW_PORT="${BIOFLOW_PORT:-8098}"
|
||||
|
||||
CAP="${1:-}"
|
||||
ACTION="${2:-}"
|
||||
|
||||
CAPS=(gene-family rnaseq extension)
|
||||
|
||||
usage() {
|
||||
echo "用法: $0 <gene-family|rnaseq|extension|all> <on|off>" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
# capability key -> registry 能力名
|
||||
cap_name() {
|
||||
case "$1" in
|
||||
gene-family) echo "bioflow-gene-family" ;;
|
||||
rnaseq) echo "bioflow-rnaseq" ;;
|
||||
extension) echo "bioflow-extension" ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# capability key -> register manifest YAML(与 agent/qa/configs/capabilities.yaml 一致)
|
||||
manifest() {
|
||||
case "$1" in
|
||||
gene-family) cat <<'YAML'
|
||||
capabilities:
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 基因家族分析(路径规划/建任务/参数确认/文件获取与预处理/状态查询),自然语言触发
|
||||
session_default: true
|
||||
nl_triggers: [基因家族, 家族成员, 亚家族, 系统发育, 建树, 进化树, 保守基序, motif, 结构域, 基因结构, 共线性, Ka/Ks, 选择压力, 组合证据]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/gene-family/
|
||||
prompt_pack:
|
||||
file: bioflow_gene_family_agent_rules.md
|
||||
YAML
|
||||
;;
|
||||
rnaseq) cat <<'YAML'
|
||||
capabilities:
|
||||
- name: bioflow-rnaseq
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 转录组分析(实验设计确认/样本配置/路径规划/建任务/差异表达/富集),自然语言触发
|
||||
session_default: true
|
||||
nl_triggers: [转录组, RNA-seq, RNAseq, rnaseq, 差异表达, 表达定量, 富集, WGCNA, GSEA, 实验设计, fastq]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/rnaseq/
|
||||
prompt_pack:
|
||||
file: bioflow_rnaseq_agent_rules.md
|
||||
YAML
|
||||
;;
|
||||
extension) cat <<'YAML'
|
||||
capabilities:
|
||||
- name: bioflow-extension
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 用户自定义扩展分析(创建/转换/试运行/校验/发布/运行),自然语言触发
|
||||
session_default: true
|
||||
nl_triggers: [扩展, 拓展分析, 扩展分析, 自定义分析, 分析模板, 分析脚本, 分析工具]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/extension/
|
||||
prompt_pack:
|
||||
file: bioflow_extension_agent_rules.md
|
||||
YAML
|
||||
;;
|
||||
*) usage ;;
|
||||
esac
|
||||
}
|
||||
|
||||
[ -n "$CAP" ] && [ -n "$ACTION" ] || usage
|
||||
[ "$ACTION" = "on" ] || [ "$ACTION" = "off" ] || usage
|
||||
if [ "$CAP" != "all" ]; then
|
||||
cap_name "$CAP" >/dev/null 2>&1 || usage
|
||||
fi
|
||||
|
||||
# 1. 逐能力 register/unregister(热加载不重启 QA)
|
||||
targets=("$CAP")
|
||||
[ "$CAP" = "all" ] && targets=("${CAPS[@]}")
|
||||
for key in "${targets[@]}"; do
|
||||
name="$(cap_name "$key")"
|
||||
if [ "$ACTION" = "on" ]; then
|
||||
body="$(manifest "$key")"
|
||||
resp=$(curl -s -m 30 -X POST "$BFF/api/capabilities/install" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "$(python3 -c 'import json,sys; print(json.dumps({"action":"register","manifest":sys.stdin.read()}))' <<<"$body")")
|
||||
else
|
||||
resp=$(curl -s -m 30 -X POST "$BFF/api/capabilities/install" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"action\":\"unregister\",\"name\":\"$name\"}")
|
||||
fi
|
||||
echo "[$ACTION] $name -> $(echo "$resp" | head -c 300)"
|
||||
done
|
||||
|
||||
# 2. 自检:GET /api/capabilities 的 installed 数组
|
||||
installed=$(curl -s -m 10 "$BFF/api/capabilities")
|
||||
echo "注册状态: $(echo "$installed" | python3 -c 'import json,sys; d=json.load(sys.stdin); print([i.get("name") for i in d.get("installed") or []])' 2>/dev/null || echo 'parse-fail')"
|
||||
|
||||
# 3. bioflow_service 按需启停:任一能力开启则启动,全部关闭则停止
|
||||
any_on=false
|
||||
for key in "${CAPS[@]}"; do
|
||||
name="$(cap_name "$key")"
|
||||
if echo "$installed" | grep -q "\"$name\""; then
|
||||
any_on=true
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$any_on" = true ]; then
|
||||
if ! lsof -ti tcp:$BIOFLOW_PORT -sTCP:LISTEN >/dev/null 2>&1; then
|
||||
echo "启动 bioflow_service :$BIOFLOW_PORT"
|
||||
( cd "$BIOFLOW_SERVICE_DIR" && setsid "$PY" -m uvicorn app.main:app --host 127.0.0.1 --port "$BIOFLOW_PORT" \
|
||||
> /tmp/plantai-bioflow.log 2>&1 < /dev/null & ) && disown -a
|
||||
sleep 2
|
||||
fi
|
||||
curl -s -m 8 "http://127.0.0.1:$BIOFLOW_PORT/api/bioflow/health" | head -c 150
|
||||
echo
|
||||
else
|
||||
echo "全部能力已卸载:停止 bioflow_service :$BIOFLOW_PORT"
|
||||
kill $(lsof -ti tcp:$BIOFLOW_PORT -sTCP:LISTEN) 2>/dev/null || true
|
||||
fi
|
||||
|
||||
echo "DONE"
|
||||
@@ -0,0 +1,177 @@
|
||||
# 生信流程 · 对话式手工测试手册(S6 更新版)
|
||||
|
||||
> 供用户逐项实测。生信模块已从「卡片式交互」改造为「自然语言对话式交互 + MCP 外挂」,
|
||||
> 本手册以**对话剧本**形式给出每个场景的:对话示例 → 系统应如何回应 → 通过判据。
|
||||
> 环境:内网 http://192.168.0.175:8090(BFF);生信服务 8098、QA 8091 已部署;
|
||||
> 三能力(基因家族/转录组/扩展)按 §0 装卸。
|
||||
|
||||
---
|
||||
|
||||
## 0. 前置条件
|
||||
|
||||
1. 浏览器打开 http://192.168.0.175:8090。
|
||||
2. **能力安装检查**:设置面板(SettingsPanel)应能看到三个生信开关(基因家族/转录组/扩展)。
|
||||
打开任意一个后,工作区出现「生信任务」「生信档案」TAB;全关则 TAB 消失(**TAB 显隐 = 能力注册状态**,非服务健康)。
|
||||
3. 服务器 docker 已有 8 个 bioflow 镜像;金标数据在 `bioflow_kernel/assets/examples/test_input/`(Ft 四件套 pep/cds/genome/gff3)。
|
||||
4. 熔断开关:`QA_V1_ENABLE_BIOFLOW_AGENT=1`(默认 0,关时生信句走普通 QA)。
|
||||
|
||||
---
|
||||
|
||||
## 1. 多配置开关回归(S6)
|
||||
|
||||
| 配置 | 操作 | 期望 | 判据 |
|
||||
|---|---|---|---|
|
||||
| 全关(熔断=0 + 三能力卸载) | 发「帮我做基因家族分析」 | 普通 QA 礼貌回答(说明未安装生信能力),**无**路径卡 | 无生信分流 |
|
||||
| 只开基因家族 | 发「帮我做基因家族分析」 | 进入生信对话(列路径+挂路径卡) | 命中 |
|
||||
| 只开基因家族 | 发「帮我做转录组差异表达」 | 走普通 QA(转录组能力未装) | 不响应 |
|
||||
| 只开转录组 | 发「帮我做转录组」 | 进入生信对话 | 命中 |
|
||||
| 只开转录组 | 发「帮我做基因家族分析」 | 走普通 QA | 不响应 |
|
||||
| 全开但非生信提问 | 发「光合作用的电子传递链是什么」 | 走普通 QA(非生信意图不调工具) | 不调工具 |
|
||||
|
||||
---
|
||||
|
||||
## 2. 对话剧本 A:基因家族分析(卡片辅助通道)
|
||||
|
||||
**第 1 轮 —— 需求与路径**
|
||||
> 用户:帮我做水稻 SPL 基因家族分析
|
||||
> 系统(自然语言):列「核心流程(必选)+ 可选分析」,说明每个可选项生物学意义;
|
||||
> **同帧挂出路径选择卡**。
|
||||
> 判据:回复含路径列表 + 路径卡出现(卡片不替代自然语言)。
|
||||
|
||||
**第 2 轮 —— 卡片操作 → 系统复述**
|
||||
> 用户(路径卡勾选「核心流程 + Ka/Ks + 启动子」,点确认;打字补充文件位置)
|
||||
> 系统:先自然语言**复述**用户选择("您选择了核心流程 + Ka/Ks + 启动子分析…"),再请确认。
|
||||
> 判据:系统复述内容与卡片勾选一致;建任务前必有一轮复述确认(不直接建任务)。
|
||||
|
||||
**第 3 轮 —— 网络文件 + 非标准文件预处理**
|
||||
> 用户:文件在 `https://…/rice.fa` 与 `oss://seq-bucket/rice.gff3`,另传 `spl_list.xlsx`
|
||||
> 系统:拉取 web/OSS 文件(**下载前检查磁盘余量、记录 provenance**);对 `spl_list.xlsx`
|
||||
> 做内容嗅探后**自然语言说明列结构并提问**("推测第二列是蛋白序列,备注列忽略?")。
|
||||
> 判据:文件获取成功;Excel 触发预处理提问(列表方式,可卡片点选也可打字回答)。
|
||||
|
||||
**第 4 轮 —— 建任务执行**
|
||||
> 用户:都对,备注列不要;确认,开始吧
|
||||
> 系统:复述整理结果("已整理为 FASTA 3 条序列")→ 创建任务 → 消息内出现**只读精简进度块**,
|
||||
> 右侧「生信任务」TAB 完整滚动日志实时跟随。
|
||||
> 判据:进度块只读(无按钮);TAB 日志自动滚底。
|
||||
|
||||
**执行中 —— 进度转述 + 随时提问**
|
||||
> 用户:什么是组合证据?
|
||||
> 系统:经流程知识库自然语言回答(带引用),**不打断任务**。
|
||||
> 判据:提问得到引用回答,任务继续推进。
|
||||
|
||||
**人工确认边界(不可跳过)**
|
||||
> 系统:第 8 步亚家族预测完成(6 个亚家族)。⚠ 需您人工确认,我不能代您确认。
|
||||
> 用户:把 SPL3 从亚家族 A 移到 B → 系统复述修改 → 用户「确认」。
|
||||
> 判据:subfamily_confirm 必停、必复述、绝不代确认。
|
||||
|
||||
**收尾 —— 存档**
|
||||
> 系统:全部完成(19 成员 6 亚家族),同帧挂出存档卡。
|
||||
> 用户(存档卡填名称点「存档」或打字「存档,叫 SPL 水稻鉴定」)
|
||||
> 系统:实际执行 `archive_job` 存档(复用 REST 语义),复述存档三要素(名称/位置/大小)。
|
||||
> 判据:存档成功(重名提示 ARCHIVE_EXISTS)。
|
||||
|
||||
---
|
||||
|
||||
## 3. 对话剧本 B:转录组分析(纯自然语言通道,全程不点卡片)
|
||||
|
||||
**第 1 轮 —— 必确认信息**
|
||||
> 用户:帮我做转录组,比较干旱处理和对照
|
||||
> 系统:列主流程 + 可选;**必须确认**四项(物种/参考基因组版本、FASTQ 单双端、
|
||||
> 对照处理组重复数、链特异性),列表展示。
|
||||
> 判据:实验设计确认边界被明确列出,未确认前不得建差异表达任务。
|
||||
|
||||
**第 2 轮 —— OSS 文件 + 样本信息(纯打字,不点卡)**
|
||||
> 用户:数据在 oss://seq-bucket/drought/,水稻日本晴,干旱 3 重复对照 3 重复,双端非链特异
|
||||
> 系统:从 OSS 拉取 12 个 FASTQ 并核对成对;复述样本设计("干旱 3 vs 对照 3,双端,非链特异"),
|
||||
> 请确认基线(对照为基线、比较方向「干旱 vs 对照」)。
|
||||
> 判据:纯自然语言路径走通(**卡片与打字两条通道服务端语义一致**)。
|
||||
|
||||
**第 3 轮 —— 执行 + 结果转述**
|
||||
> 用户:对 → 系统执行 QC→比对→定量→样本质控→差异表达→富集,逐段转述(接头率/Q30/唯一比对率/差异基因数/PCA 分离)。
|
||||
> 判据:每步完成后自然语言转述关键指标;样本质控 PCA 结论有转述。
|
||||
|
||||
---
|
||||
|
||||
## 4. 三种网络文件来源用例
|
||||
|
||||
| 来源 | 示例 | 期望 |
|
||||
|---|---|---|
|
||||
| web | 「文件在 https://…/x.fa」 | httpx 流式下载,不设大小上限;下载前查磁盘余量;记录 provenance |
|
||||
| OSS | 「数据在 oss://bucket/key 或预签名 URL」 | 经 BIOFLOW_OSS_ENDPOINT 解析或预签名直通;无 endpoint 时提示改用预签名 |
|
||||
| 局域网 | 「文件在 192.168.x.x:/data/x.fa,账号 xx 密码 xx」 | ssh/scp 拉取(凭据单次会话内存用,不落盘、审计不记明文) |
|
||||
|
||||
判据:文件落 `bioflow_data/jobs/{job}/input/`,先内容嗅探后登记,provenance 记录来源/哈希/时间。
|
||||
|
||||
---
|
||||
|
||||
## 5. 非标准文件预处理(Excel / 乱码文本)闭环
|
||||
|
||||
1. 上传 `xxx.xlsx` / 乱码 txt → 系统 `inspect_file` 探测(列结构/格式猜测/乱码检测)。
|
||||
2. 系统自然语言说明 + 列表提问(哪列是序列、是否去重、是否去表头)。
|
||||
3. 用户回答 → `transform_file` 转换(LLM 定规则 + 脚本执行,大文件抽样定规则)。
|
||||
4. 结果过格式验证器(fasta 可解析/gff3 9 列/tsv 列一致/samples.yaml schema);
|
||||
失败说明问题并追问,循环 ≤3 次后建议人工整理。
|
||||
5. 转换后 `register_job_files` 登记,provenance 记「原始→规则→产物」链。
|
||||
|
||||
判据:全程自然语言引导;转换产物格式校验通过;原文件永不覆盖。
|
||||
|
||||
---
|
||||
|
||||
## 6. 卡片辅助 vs 纯自然语言双通道等价
|
||||
|
||||
| 场景 | 卡片通道 | 纯 NL 通道 | 等价判据 |
|
||||
|---|---|---|---|
|
||||
| 路径选择 | 路径卡勾选 | 打字「核心流程 + Ka/Ks」 | 两者复述的确认内容一致,建任务入参一致 |
|
||||
| 参数确认 | 参数卡填值 | 打字「用 8 核、padj 用 0.01」 | confirm_stage 入参一致 |
|
||||
| 文件提供 | 文件卡拖拽/填路径 | 打字「文件在 …」 | register_job_files / fetch_source_file 入参一致 |
|
||||
| 存档 | 存档卡点存档 | 打字「存档,叫 X」 | archive_job 入参一致 |
|
||||
|
||||
判据:卡片操作产生的是 `card_result` 结构化消息进 QA 流(**不直连 bioflow 控制面**),
|
||||
与打字在服务端语义完全一致(QA 对话状态机是唯一决策点)。
|
||||
|
||||
---
|
||||
|
||||
## 7. 旧会话兼容(Legacy 块只读)
|
||||
|
||||
1. 打开一个改造前创建、含旧生信块(bioflow_qa / 旧 bioflow_* 块)的会话。
|
||||
2. 期望:旧块以只读形式降级渲染(标题 + 只读引用/提示),**不报错、无可交互控件**。
|
||||
|
||||
---
|
||||
|
||||
## 8. 扩展子系统对话化(第三能力单元)
|
||||
|
||||
1. 打开「扩展」能力开关。
|
||||
2. 对话创建扩展:描述分析脚本(代码文件 + 运行命令)→ `create_extension`(draft)。
|
||||
3. `build` → `trial_run` → `validate` → `confirm` → `publish` → `run` 逐步走(卡片或打字等价)。
|
||||
4. 试运行失败 → 「AI 修复」触发 `repair_extension`(修复上限 20 轮);校验失败/状态不足时系统给出确定性提示。
|
||||
|
||||
判据:扩展生命周期全程对话化;非法状态返回清晰提示(EXT_STATUS),不报 500。
|
||||
|
||||
---
|
||||
|
||||
## 9. 后端金标验证(服务器命令行,S6 回归)
|
||||
|
||||
```bash
|
||||
cd /home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new
|
||||
BIOFLOW_DATA=/home/xzq/5090/teamWorkspace/zh/bioflow-data \
|
||||
bash bioflow_kernel/tests/smoke_golden.sh bioflow-gf:dev
|
||||
```
|
||||
|
||||
期望:4 步全绿——python 3.11.14 / ldd 0 缺库 / 无 conda / 金标 identify 24 ID diff 空(`GOLDEN_DIFF_OK 24 IDs`)。
|
||||
|
||||
---
|
||||
|
||||
## 10. 端到端主线(一次性走全流程,双通道各一遍)
|
||||
|
||||
1. 卡片通道:§2 全程(路径卡→文件卡→参数卡→进度块→存档卡)。
|
||||
2. 纯 NL 通道:§3 全程(不点任何卡片)。
|
||||
3. 含一次 Excel 预处理、一次局域网/OSS 文件拉取。
|
||||
4. 任务完成后存档 → 生信档案 TAB 回看 → 复用发起。
|
||||
|
||||
期望:两条通道均走通,7 项引导(路径/文件/参数/报告/存档/复用/过程展示)逐项达成,
|
||||
进度块实时流转、右侧日志滚底。
|
||||
|
||||
---
|
||||
|
||||
*本手册为 S6 对话式改造更新版,替代旧卡片式 `MANUAL_TEST_BIOFLOW.md` 的生信流程部分。*
|
||||
@@ -1,42 +0,0 @@
|
||||
# feat(review): journal_directory 启动自动 seed(部署即运行)
|
||||
|
||||
## 背景
|
||||
|
||||
source_v4.xlsx(1056 条目录)此前需手动跑 import_journals.py 入库,部署后忘跑即出现"目录数据未导入"。本次实现启动时自动检测 + 自动导入。线上当前环境已由人工完成导入(review.db 内 1056 行),本功能消除的是新环境部署/重建库时的手动步骤风险。
|
||||
|
||||
## 修改内容
|
||||
|
||||
| 文件 | 变更 |
|
||||
|---|---|
|
||||
| `app/services/journal_seed.py`(新增) | 共享核心:schema/清洗/解析/upsert 从 CLI 原样迁入;`extract_rows_from_xlsx()`(口径 779/277 与 id 冲突闸门,不符不写库)、`upsert_to_db()`、`ensure_journal_directory_seeded()`(空表才导,四键 dict 契约,绝不抛异常) |
|
||||
| `scripts/import_journals.py` | 瘦身为薄 CLI(-431/+92 行):对账报告/退出码 0/1/2/3 全保留,核心复用 journal_seed |
|
||||
| `app/config.py` | +`seed_on_startup=True`(REVIEW_SEED_ON_STARTUP 可关)、+`journal_directory_seed_path` |
|
||||
| `app/main.py` | lifespan:开关门控 → `asyncio.to_thread` 导入(不堵 event loop)→ try/except 不阻塞启动;路径走 `db.path` 注入链 |
|
||||
| `tests/conftest.py`(新增) | 全局关 seed,防止单测触碰真库/拖慢套件 |
|
||||
| `tests/test_seed.py`(新增) | 6 用例(见下) |
|
||||
|
||||
不动:source_v4.xlsx、journal_directory 路由/表 schema、db.py、前端。
|
||||
|
||||
## 测试用例与结果(pytest 78 passed,新增 6 条)
|
||||
|
||||
1. 空库 → seed 成功,rows=1056(en=779/zh=277)✓
|
||||
2. 非空库(1 行)→ skip,count 不变 ✓
|
||||
3. xlsx 不存在 → 返回 error,不抛异常,不写库 ✓
|
||||
4. 幂等:连调两次,第二次 skip ✓
|
||||
5. 开关 off(lifespan)→ 表不创建 ✓
|
||||
6. 开关 on(lifespan)→ tmp 库 1056 行 ✓
|
||||
|
||||
## 部署验证
|
||||
|
||||
```bash
|
||||
bash review_service/restart.sh # 非空库:日志 "already has 1056 rows, skip"
|
||||
curl 'http://127.0.0.1:8100/api/review/journals/directory?lang=en' # total=779
|
||||
# 空库冷启动(本机已验证等价路径):日志 "seeded 1056 rows (en=779, zh=277, N.Ns)"
|
||||
# 特殊部署:REVIEW_SEED_ON_STARTUP=false → 日志 "seed skipped"
|
||||
```
|
||||
|
||||
本机已实测:真机重启走 skip 路径(`INFO:app.main:already has 1056 rows, skip`),CLI 薄壳回归 exit=0 / `en=779 zh=277 → ✓ 相符`。
|
||||
|
||||
## 是否需要立即部署
|
||||
|
||||
待管理员确认。线上已手动导入 1056 行,本改动属部署便利性补强(消除新环境忘跑导入脚本的风险),可随下次正常发版评估带上。
|
||||
@@ -1,56 +0,0 @@
|
||||
# fix(review): 冲突降级真影响 recommendation(major → minor 真降级)
|
||||
|
||||
- 分支:`feature/academic-review-assistant`
|
||||
- 日期:2026-10-12
|
||||
- 服务:`review_service`(:8100)/ 前端 `PresubmitPanel.vue`
|
||||
- 状态:**未 commit**,待验收
|
||||
|
||||
## 背景
|
||||
|
||||
s41588(已发表 Nature Genetics 论文)真实链路里,关键词扫描器报告"3 处冲突已降级为非硬性问题",但 `recommendation` 仍为大修、`major_weaknesses` 仍 5 条。诊断确认:原降级只做了"剥 hard_rule 标签 + 记 conflicts",**条目仍在 major 列表内**,`major_count` / 引擎输入未变,故判定不变。本次把"降级"从标签维度升级为**列表维度的真降级**(major → minor)。
|
||||
|
||||
## 变更
|
||||
|
||||
**后端**
|
||||
- `paper_scanner.reconcile_hard_rules`:命中冲突的 major 从列表**移出**,`hard_rule=None` + `scanner_downgraded=true`,落入 demoted 列表;`fatal=true` 条目例外(只记 conflict 不降级,须人工过目)。
|
||||
- `routers/presubmit.py`:demoted 追加进 `report.minor_weaknesses`;`major_weaknesses` 为过滤后列表;explanation 文案改"已降级为小修级问题"。
|
||||
- `self_review.WeaknessItem`:新增 `scanner_downgraded: bool = False`(coerce 不读、LLM 无法伪造)。
|
||||
- 引擎 `recommend_engine.decide_recommendation` **零改动**——计数因列表变短而自动真实。
|
||||
|
||||
**前端**
|
||||
- `PresubmitPanel.vue`:minor 卡片对 `scanner_downgraded` 显示黄徽标"冲突降级"(与 major 卡片"硬性"红徽标对仗);冲突明细块保留。
|
||||
|
||||
## 测试
|
||||
|
||||
- pytest **104 passed**(新增:`test_reconcile_fatal_not_demoted`、`test_http_partial_conflict_leaves_residual_soft`;改写 `test_reconcile_*` 与 `test_http_scanner_conflict_downgrades` 为真降级断言)。
|
||||
- vitest **382 passed / 5 failed**(5F 为既有他人 `benchpanel`+`Composer .mode-pill` labels.js 引发,与本改动无关);`npm run build` ✓。
|
||||
|
||||
## 验收(真 LLM,:8100)
|
||||
|
||||
| 样本 | rec | rule | M/S/minor | conflicts | downgraded→minor |
|
||||
|---|---|---|---|---|---|
|
||||
| diag_strong | minor_revision | R5 | 0/0/4 | 0 | 无 |
|
||||
| diag_weak | major_revision | R1 | 6/2/7 | 1(hic) | 有(真降级生效) |
|
||||
|
||||
弱稿仍大修:4 条硬性命中(busco/accession/voucher/abs)为真缺,扫描器无冲突、不降级——真缺陷不放行。
|
||||
|
||||
## 验收:s41588(真 LLM,:8100)
|
||||
|
||||
输入 `/tmp/opencode/s41588-025-02113-5.pdf`,target_journal=Genome Biology(tier 1 override)。
|
||||
|
||||
| 项 | 修复前 | 修复后 |
|
||||
|---|---|---|
|
||||
| recommendation | major_revision | **minor_revision**(R5) |
|
||||
| major_count / soft / minor | 5 / — / — | **0 / 0 / 6** |
|
||||
| hard_hits | 含 busco/accession… | **[]** |
|
||||
| scanner signals | — | accession_found=True, busco=partial, hic_found=True |
|
||||
| major_weaknesses 列表 | 5 条 | **0 条** |
|
||||
| scanner_conflicts | — | **2 条保留**(busco_missing 格式不完整、accession_missing 部分未给号)→ 均真降级进 minor(`scanner_downgraded=true`) |
|
||||
|
||||
判定链:扫描器证伪 2 条硬性 → 真降级 → 剩余为 minor → R5(tier1 accept 门槛 minor≤1,6 条 → minor_revision)。conflicts 透明保留、explanation 明示"已降级为小修级问题,请人工核对"。
|
||||
|
||||
注:本次 LLM 未再把"树图注未标数值 / GWAS 校正描述"判成 major(负向清单 Prompt 生效),故 major 从 5 归 0,优于预期的"降到 2"。
|
||||
|
||||
## 部署
|
||||
|
||||
待管理员确认。属判定正确性修复(真论文误判大修),建议随下次正常发版带上。
|
||||
@@ -1,81 +0,0 @@
|
||||
# fix(review): presubmit 四 Bug 修复——截断保真 / 分档 / 词表 / 扫描器兜底
|
||||
|
||||
## 背景
|
||||
|
||||
真实案例:已发表 Nature Genetics 论文(s41588-026-02682-z,黄瓜泛基因组)被判
|
||||
"大修",但论文实有 BUSCO 98.57%(P2)、94.18% sequences anchored(P2)、
|
||||
PRJCA038097 等 accession(P17 Data Availability)。诊断出四个独立缺陷:
|
||||
|
||||
1. **PDF 截断**:`max_text_chars=60000` 从头切,30 页论文 P17 的 Data Availability 被截掉;
|
||||
2. **期刊分档**:Nature Genetics 不在 5 本精选 → 默认 tier 2(应 tier 1);
|
||||
3. **硬性规则词表过窄**:缺 PRJCA/CRA(中国 NGDC/GSA)、Hi-C 同义词只认 anchoring rate、
|
||||
BUSCO 仅百分比也算缺失;
|
||||
4. **无程序层兜底**:hard_rule 全靠 LLM 输出,无关键词证伪机制。
|
||||
|
||||
## 修改内容
|
||||
|
||||
| 文件 | 变更 |
|
||||
|---|---|
|
||||
| `app/routers/presubmit.py` | **Bug 1**:段落感知装箱 `pack_feed_text`(超预算按
|
||||
Abstract→Data Availability 全文→Methods 前 60%→Results→Discussion 优先保留,
|
||||
dropped_sections 入 meta);无节标题回退头+尾窗(捞页尾 availability 块)。
|
||||
**双层保护**:估算 token(len/1.5)>48K → 自动降 100K 重装;LLM 400/413 → 降 80K
|
||||
重试一次。**Bug 4 接线**:coerce 后扫描器证伪 → rationale.scanner_conflicts |
|
||||
**Bug 2**:tier_source ∈ override/featured/jif/默认 |
|
||||
| `app/services/paper_scanner.py`(新增) | 关键词扫描器(纯 stdlib):accession
|
||||
词表 PRJNA/PRJEB/PRJCA/SRP/ERP/DRP/CRA/SRR/ERR/DRR/GCA_/GCF_ **只在 Data Availability
|
||||
段扫**(正文引用他人数据不误判);BUSCO 三级 none/partial/full;Hi-C 同义词
|
||||
(anchored to chromosome(s)/anchoring rate/pseudo-chromosome(s)/chromosome-level)。
|
||||
`reconcile_hard_rules`:LLM 报缺 X + 扫描器见 X → 剥硬性标签降 soft + 记冲突
|
||||
(只证伪不证实,不引入新误判) |
|
||||
| `app/services/recommend_engine.py` | **Bug 2**:tier 查找优先级 override
|
||||
(13 本顶刊硬编码,"The " 前缀归一化)→ featured(journals.json)→ jif 启发式
|
||||
(journal_directory 只读 jif 列;jif≥15→tier1;综述刊名含 Review/Trends/Annual
|
||||
Review 除外)→ 默认 tier 2 |
|
||||
| `app/services/self_review.py` | **Bug 3**:Prompt 词表扩充(PRJCA/CRA/SRR…);
|
||||
BUSCO 三级判定(仅百分比→minor 不打 hard_rule);Hi-C 同义词清单。
|
||||
**2026-10-12 追加 soft major 收紧**(s41588 复盘):维度三系统发育树分诊重写
|
||||
(完全未做→major / 已声明仅图未标数值→presentation / 附带树→minor);新增
|
||||
【负向清单】节(图未标统计量、标准方法描述不全、阈值缺数值、引用排版 →
|
||||
minor/presentation,严禁列 major;总纲"只有补实验/补数据/重跑分析/改核心论证
|
||||
才是 major";与各维度冲突时以本节为准) |
|
||||
| `app/config.py` | `max_text_chars` 60000 → **150000**(REVIEW_MAX_TEXT_CHARS 可覆盖) |
|
||||
| `ui/.../PresubmitPanel.vue` | 冲突黄标块(scanner_conflicts + 降级提示);截断
|
||||
dropped_sections 美化提示;截断文案改为"保摘要/数据可用性/方法/结果/讨论" |
|
||||
| `tests/test_presubmit.py` | 新增 15 用例(装箱/回退/token 守卫/400 重试/扫描器
|
||||
三类信号/DA 段限定/冲突降级/端到端/tier 优先级矩阵/Prompt 新词表) |
|
||||
|
||||
## 测试用例与结果
|
||||
|
||||
- **pytest:102 passed**(87 基线 + 15 新增)。
|
||||
- **vitest:379 passed / 5 failed**(5F 为既有他人 i18n labels 改动引发,与本改动
|
||||
无关;新增冲突黄标/截断提示 2 条全绿)。`npm run build` ✓。
|
||||
- **真实链路验收**(真 LLM,target_journal=Genome Biology → tier 1 override):
|
||||
|
||||
| 样本 | 修复前 | 修复后 | 扫描器 |
|
||||
|---|---|---|---|
|
||||
| 强稿 diag_strong.pdf | minor_revision | **minor_revision**(R5,0/0/5) | accession/busco=full/hic 全命中,0 冲突 ✓ |
|
||||
| 弱稿 diag_weak.pdf | major_revision | **major_revision**(R1,硬性 4 + soft 4 + minor 5) | accession/busco 未命中(真缺不降);Hi-C 冲突降级并记录 ✓ |
|
||||
| **真实案例 s41588**(NG 论文) | major_revision | 见下"验收"节 | — |
|
||||
|
||||
## 部署验证
|
||||
|
||||
```bash
|
||||
bash review_service/restart.sh
|
||||
curl -X POST http://127.0.0.1:8100/api/review/presubmit \
|
||||
-F "file=@paper.pdf" -F "target_journal=Nature Genetics"
|
||||
# 检查:meta.tier=1/tier_source=override;meta.dropped_sections;
|
||||
# report.recommendation_rationale.scanner_conflicts
|
||||
```
|
||||
|
||||
## 是否需要立即部署
|
||||
|
||||
待管理员确认。四修复均为判定正确性问题(真实发表论文误判大修),建议随下次正常发版带上。
|
||||
|
||||
## 遗留
|
||||
|
||||
- tier jif 启发式仅覆盖 journal_directory 已录入的 1056 刊;未收录刊名走默认 tier 2。
|
||||
- 扫描器 accession 仅扫 DA 段——若论文把 accession 放 Methods 数据声明处且无 DA 节,
|
||||
扫描器不证伪(保守方向,LLM 硬性判定仍生效,不会误放行)。
|
||||
- BUSCO full 判定要求 S: 与 D: 标记在 200 字符窗内相邻——极端排版可能判 partial
|
||||
(只会更严,不误放)。
|
||||
@@ -1,61 +0,0 @@
|
||||
# fix(review): presubmit recommendation 判定过严修复(配额制 → 规则引擎)
|
||||
|
||||
## 背景
|
||||
|
||||
用户反馈 AI 预审稿多次测试均返回 major_revision。2026-10-12 诊断确认根因是
|
||||
**配额制**而非"尺子紧":
|
||||
|
||||
- `self_review.py` 硬性规则要求 `major_weaknesses 3-8 条`(强制配额)→ accept/minor_revision 几乎不可达;
|
||||
- LLM 直接输出 recommendation,无程序层规则;JSON 示例锚定 `major_revision`、coerce 缺省回落 `major_revision`;
|
||||
- 对照实验:近 GB 完备强稿(BUSCO 96.8% / Hi-C 98.7% / PRJNA+GCA+figshare 齐 / UFBoot 1000)仍被凑出 6 条 major,与弱稿(7 条)几乎无区分度;
|
||||
- `target_journal` 传入但 Prompt 零分档标准。
|
||||
|
||||
## 修改内容
|
||||
|
||||
| 文件 | 变更 |
|
||||
|---|---|
|
||||
| `app/services/recommend_engine.py`(新增) | 纯函数规则引擎:R0 fatal→reject / R1 硬性 major→大修(不可协商,与 tier 无关)/ R2' tier3 仅 1-2 条 soft 可降小修 / R2 soft≥1→大修 / R4 0 major 且 minor≤tier 阈值→接收 / R5→小修;rationale 模板拼装(rule/hard_hits/计数/tier/explanation,含 tier 降级提示句);`resolve_tier` 按 journals.json `review_tier` 匹配,失败默认 tier 2 |
|
||||
| `app/services/self_review.py` | Prompt 重写:严重分级标准(major/minor/presentation),**三类均不设下限、可 0 条、严禁凑数**;硬性规则 5 项打标签(busco/hic/accession/voucher/abs_missing);删除 recommendation 输出字段与示例锚定;coerce 不再读 recommendation(防注入),hard_rule 白名单过滤 |
|
||||
| `app/routers/presubmit.py` | coerce 后接引擎;meta 增加 tier/tier_label/tier_source |
|
||||
| `data/journals/journals.json` | 精选 5 本加 `review_tier`(GB/MP=1;BMC PB/FPS/Rice=2)+ notes 一行 |
|
||||
| `ui/.../PresubmitPanel.vue` | 判定依据块(explanation + 硬性 chips);判定档位徽标;reject 红标"建议人工复核";顶部 AI 免责行;major 卡片"硬性/fatal"徽标 |
|
||||
| `tests/test_presubmit.py` | 用例 1-10(引擎分支全覆盖 + tier 对照 + 注入忽略 + rationale 字段) |
|
||||
|
||||
判定链与旧链路唯一对外契约变化:`report.recommendation` 语义不变(四档枚举),
|
||||
新增 `report.recommendation_rationale` 与 `meta.tier*`。
|
||||
|
||||
## 测试用例与结果
|
||||
|
||||
- **pytest:87 passed**(78 基线 + 9 新增/重写),含用例 1-10:
|
||||
强稿形态不判大修、弱稿形态判大修且 hard_hits 正确、0 major 0/多 minor 分支、
|
||||
soft major 分支、fatal→reject、tier3 降级与硬性豁免、tier1 accept 门槛、
|
||||
LLM recommendation 注入被忽略、hard_rule 白名单过滤、rationale 含 tier/explanation。
|
||||
- **vitest:377 passed / 5 failed**(5F 为既有他人 i18n labels 改动引发的 benchpanel
|
||||
失败,与本改动无关;新增 4 条 rationale/档位/fatal 展示用例全绿)。
|
||||
- **真实链路验收**(诊断同款样本、同 target_journal=Genome Biology,真 LLM 调用):
|
||||
|
||||
| 样本 | 修复前 | 修复后 | 依据 |
|
||||
|---|---|---|---|
|
||||
| 弱稿 diag_weak.pdf | major_revision(7 major) | **major_revision**(R1;硬性 5 命中:BUSCO/Hi-C/Accession/voucher/ABS + soft 2 + minor 6) | 真缺陷不漏判 ✓ |
|
||||
| 强稿 diag_strong.pdf | major_revision(凑出 6 major) | **minor_revision**(R5;0 major / 5 minor) | 区分度恢复 ✓ |
|
||||
|
||||
## 部署验证
|
||||
|
||||
```bash
|
||||
bash review_service/restart.sh
|
||||
# 上传强/弱样本对比(或任意 PDF):
|
||||
curl -X POST http://127.0.0.1:8100/api/review/presubmit \
|
||||
-F "file=@paper.pdf" -F "target_journal=Genome Biology"
|
||||
# 检查 report.recommendation_rationale.rule / meta.tier;前端看"判定依据"块
|
||||
```
|
||||
|
||||
## 是否需要立即部署
|
||||
|
||||
待管理员确认。属判定质量修复(误报大修影响用户体验与信任),建议随下次正常发版带上。
|
||||
|
||||
## 遗留(Phase 2 候选)
|
||||
|
||||
- 程序层兜底 `_sanity_check_missing_hard_rules`:正文无 PRJNA/GCA_/SRP 关键词但 LLM
|
||||
未报 accession_missing → 追加 warning(不进 recommendation 计算);先观察 LLM 是否
|
||||
"躺平"再决定。
|
||||
- tier 数据仅覆盖精选 5 本;扩展到 journal_directory 1056 条需人工核验后另行入库。
|
||||
@@ -1,161 +0,0 @@
|
||||
# PR 报告:Rebuttal Phase 1(审稿回复助手)
|
||||
|
||||
- 日期:2026-10-12
|
||||
- 分支:`feature/academic-review-assistant`(未 commit,按用户约定统一处理)
|
||||
- 范围:`review_service/` 新增 5 文件 + main 注册;`ui/` 新增面板与 store/api 扩展;无 sdk/、agent/、vstore/ 改动;无新增依赖。
|
||||
|
||||
## 功能概述
|
||||
|
||||
粘贴审稿意见文本(主入口)/ 上传批注版 PDF(次要入口)→ LLM+正则拆条(Reviewer→Comment)
|
||||
→ 逐条填回复要点(5 类下拉,无 auto)→ 生成草稿(反幻觉校验前置)
|
||||
→ 逐条勾确认 → 拼装 response letter → 下载 `.md`(human_ack 门槛)。docx 属 Phase 2。
|
||||
|
||||
## 10 个 API 端点(review_service:8100;BFF 8095 通用透传已验证可用)
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
|---|---|---|
|
||||
| POST | /api/review/rebuttal/jobs | 文本粘贴主入口 |
|
||||
| POST | /api/review/rebuttal/jobs/pdf | PDF 次要入口(vstore 解析;非 PDF/超限→VALIDATION) |
|
||||
| GET | /api/review/rebuttal/jobs/{id} | 全量还原 |
|
||||
| PATCH | .../comments/{cid}/response | 保存 parts(改后旧 draft/validation 作废) |
|
||||
| POST | .../comments/{cid}/draft | 生成草稿(校验C前置;MISSING_RESPONSE 时不调 LLM) |
|
||||
| PATCH | .../comments/{cid}/structure | 手工 merge(≥2)/ split(≥2 非空子项) |
|
||||
| PATCH | .../comments/{cid}/confirm | 逐条勾;后端重算 human_ack(全勾才 true,空 job 不置位) |
|
||||
| POST | .../letter | 拼装 letter_md |
|
||||
| GET | .../letter.md | 下载(human_ack=false → 400 HUMAN_ACK_REQUIRED) |
|
||||
| DELETE | .../jobs/{id} | 手动删(替代 TTL) |
|
||||
|
||||
信封错误复用约定:HTTP 200 + `{ok:false,error:{code,message,degraded}}`。
|
||||
|
||||
## 数据模型
|
||||
|
||||
- Job:`{id,status,source:{kind,filename,raw_text},template:'generic',reviewers:[...],letter_md,validation,human_ack,created_at,updated_at}`
|
||||
- Comment:`{id,group_id,severity,title,body,sub_items[],cross_refs[],response:{parts[]},draft,validation,human_confirmed}`
|
||||
- part.type 枚举:`new_data | reanalysis | location | clarification | rebuttal`(决策 2:无 auto)
|
||||
- SQLite 表 `rebuttal_jobs` 落在既有 `review_service/review.db`(自含短连接,不动 app/db.py);
|
||||
启动幂等建表(main lifespan)。
|
||||
|
||||
## 关键实现决策(方案未写清处,按"保守、可测、不猜测")
|
||||
|
||||
1. **LLM 注入形态**:parse/draft 用 `async (messages, temperature) → dict` 回调,路由内适配
|
||||
`llm.complete_json(system, user)`;单测直接 monkeypatch `llm.complete_json`。
|
||||
2. **拆条双轨**:正则先切骨架(Reviewer 段头 + 编号行,兼容行内 `. 2.` 编号);
|
||||
LLM 逐段出 `{title,severity,sub_items[]}`。LLM 失败/结构非法 → 该段回退正则
|
||||
(title=首 60 字,severity=major)。`sub_items ≥2 且 max confidence ≥0.7` 才拆
|
||||
(决策 3),拆出条目共享 group_id;未达阈值多子问题并回一条且 **body 保留段原文**
|
||||
(不采信 LLM 改写)。
|
||||
3. **交叉引用**:`see comment N / 参见第 N 条 / 同上` → 解析目标 id;解析不到
|
||||
`target_id=null` 保留 raw(不猜)。
|
||||
4. **反幻觉校验(负向词表收窄版)**:
|
||||
- location:`shows that/as shown` 且无 `added/新增` → `LOCATION_EVIDENCE_WEAK` FAIL;
|
||||
- rebuttal:命中 `we agree/accept/concede/you are right/we were wrong` →
|
||||
`REBUTTAL_NEGATIVE` FAIL;无立场词(maintain/disagree/respectfully…)→
|
||||
`REBUTTAL_NO_STANCE` FAIL;
|
||||
- 其余三类 Phase 1 仅查非空。校验在 draft 端点前置,FAIL 不调 LLM。
|
||||
5. **human_ack**:confirm 端点改完后由 store 重算(所有 comment 全 true 才置位;
|
||||
空 job 不置位防平凡真)。前端无法直接置位。
|
||||
6. **letter.md 下载**:纯文本端点(非信封),门槛失败返回 400 + 明文
|
||||
`HUMAN_ACK_REQUIRED:…`;前端 `rebuttalDownloadLetterMd` 适配成 ApiError。
|
||||
7. **MISSING_RESPONSE**:draft 端点 parts 为空或全空内容 → 信封错误,LLM 调用次数=0
|
||||
(单测断言)。
|
||||
8. **PDF 入口**:复用 vstore `parse_pdf_bytes`;校验顺序 = 非 .pdf → 空文件 → 超
|
||||
`max_pdf_bytes`(50MB)→ 解析。与任务清单"PDF 超限/非 PDF"信封一致。
|
||||
9. **前端状态**:`store.rebuttal` 纯内存(切面板不丢、刷新丢——与任务"状态持久化"
|
||||
不同诉求,Phase 1 未要求 localStorage);draft 失败信封里的 job/validation 一并镜像。
|
||||
10. **不改禁区**:sdk/agent/vstore/未动;PresubmitPanel 等既有面板未动;无新增依赖。
|
||||
|
||||
## 测试
|
||||
|
||||
### 后端 pytest(9 例新增,全量 113 passed)
|
||||
|
||||
```
|
||||
tests/test_rebuttal.py::test_single_reviewer_single_comment PASSED
|
||||
tests/test_rebuttal.py::test_multi_reviewer_attribution PASSED
|
||||
tests/test_rebuttal.py::test_subitem_split_shares_group_id PASSED
|
||||
tests/test_rebuttal.py::test_location_validation_fail_and_pass PASSED
|
||||
tests/test_rebuttal.py::test_rebuttal_validation PASSED
|
||||
tests/test_rebuttal.py::test_missing_response_no_llm_call PASSED
|
||||
tests/test_rebuttal.py::test_cross_ref_unresolved_keeps_raw PASSED
|
||||
tests/test_rebuttal.py::test_envelope_errors PASSED
|
||||
tests/test_rebuttal.py::test_human_ack_gate_on_letter_download PASSED
|
||||
|
||||
全量: 113 passed in 10.05s(104 基线 + 9 新增)
|
||||
```
|
||||
|
||||
### 前端 vitest(5 例新增)
|
||||
|
||||
```
|
||||
rebuttal.panel.test.js 5 passed(5 类下拉无 auto / location hint 条件显示 /
|
||||
未填禁用生成 / 未全确认禁用下载+横幅 / merge-split 仅 active 表单)
|
||||
全量: 387 passed | 5 failed(失败全部为已知 labels.js 引发的 benchpanel×4 +
|
||||
Composer .mode-pill×1 预存失败,与本次改动无关;基线 382P/5F → 387P/5F)
|
||||
```
|
||||
|
||||
### build
|
||||
|
||||
`npm run build` ✓ built in 3.04s
|
||||
|
||||
## 手工冒烟(curl,8100 真 LLM + 真 SQLite)
|
||||
|
||||
```
|
||||
POST /jobs(真实双 Reviewer 意见)→ ok=True job=rbt_ed245cd63b3a reviewers=2 comments=4
|
||||
titles: 缺少FDR校正与q值报告 / Figure 3 blurry… / Missing coordinates… / Overstated novelty…
|
||||
POST draft(未填) → ok=False code=MISSING_RESPONSE(LLM 0 调用)
|
||||
PATCH response(location 弱) + POST draft
|
||||
→ ok=False VALIDATION_FAILED findings=[LOCATION_EVIDENCE_WEAK]
|
||||
PATCH response(location 合法) + POST draft → ok=True(真 LLM 草稿)
|
||||
PATCH response(rebuttal 让步词) + draft → VALIDATION_FAILED findings=[REBUTTAL_NEGATIVE]
|
||||
3 条合理回复(new_data/clarification/rebuttal)→ draft 全 ok=True
|
||||
GET letter.md(未全勾) → HTTP 400 HUMAN_ACK_REQUIRED
|
||||
PATCH confirm ×3 → human_ack=False(c0001 漏勾,恰好验证逐条门槛)
|
||||
PATCH confirm c0001 → human_ack=True
|
||||
POST letter → ok=True letter_md len=2731
|
||||
GET letter.md → HTTP 200(/tmp/opencode/letter.md 落盘,含
|
||||
Response to Reviewers + 4 条原文/回复)
|
||||
DELETE job → ok=True meta={deleted:…};GET → NOT_FOUND
|
||||
BFF 8095 探测 /rebuttal/jobs/nonexistent → HTTP 200(透传信封,REVIEW_ENABLED=1)
|
||||
```
|
||||
|
||||
## 交付物清单
|
||||
|
||||
| 文件 | 动作 |
|
||||
|---|---|
|
||||
| review_service/app/services/rebuttal_store.py | 新增(SQLite CRUD + human_ack 重算) |
|
||||
| review_service/app/services/rebuttal_parse.py | 新增(正则骨架 + LLM 拆分 + 交叉引用) |
|
||||
| review_service/app/services/rebuttal_draft.py | 新增(5 类 Prompt + 反幻觉校验 + letter) |
|
||||
| review_service/app/routers/rebuttal.py | 新增(10 端点) |
|
||||
| review_service/app/main.py | +import +include_router +lifespan 建表(约 8 行) |
|
||||
| review_service/tests/test_rebuttal.py | 新增(9 例) |
|
||||
| ui/src/api/review.js | +10 API 封装 + rebuttal 错误码未进 CODE_HINTS(后端原话直达) |
|
||||
| ui/src/stores/edu.js | +SECTIONS 'rebuttal' + rebuttal 状态块 + 10 actions |
|
||||
| ui/src/components/edu/panels/RebuttalPanel.vue | 新增(拆条树/逐条表单/校验红条/draft 预览/手工拆合/letter) |
|
||||
| ui/src/components/edu/EduSideNav.vue | +1 行菜单(审稿回复助手 💬) |
|
||||
| ui/src/components/edu/EduWorkspace.vue | +import +panelMap 挂载 |
|
||||
| tests/frontend/unit/edu/rebuttal.panel.test.js | 新增(5 例) |
|
||||
| docs/pr-reports/2026-10-12-rebuttal-phase1.md | 本报告 |
|
||||
|
||||
## git status --short(新增部分)
|
||||
|
||||
```
|
||||
?? docs/pr-reports/2026-10-12-rebuttal-phase1.md
|
||||
?? review_service/app/routers/rebuttal.py
|
||||
?? review_service/app/services/rebuttal_draft.py
|
||||
?? review_service/app/services/rebuttal_parse.py
|
||||
?? review_service/app/services/rebuttal_store.py
|
||||
?? review_service/tests/test_rebuttal.py
|
||||
?? tests/frontend/unit/edu/rebuttal.panel.test.js
|
||||
?? ui/src/components/edu/panels/RebuttalPanel.vue
|
||||
M review_service/app/main.py
|
||||
M ui/src/api/review.js
|
||||
M ui/src/components/edu/EduSideNav.vue
|
||||
M ui/src/components/edu/EduWorkspace.vue
|
||||
M ui/src/stores/edu.js
|
||||
(此前 AI 预审稿修复的未 commit 改动保持原样,未混入本报告范围)
|
||||
```
|
||||
|
||||
## 已知边界(Phase 1 明确不做)
|
||||
|
||||
- docx 导出(Phase 2,python-docx 已在 lock 备用);
|
||||
- job TTL 自动清理(决策 5:手动 DELETE 替代);
|
||||
- 跨 Reviewer 合并(merge 限同 Reviewer 内);
|
||||
- 刷新页面后 job 恢复(纯内存态;GET /jobs/{id} 已备,Phase 2 可接列表入口)。
|
||||
@@ -1,13 +0,0 @@
|
||||
Reviewer #1:
|
||||
|
||||
1. The GWAS analysis lacks multiple-testing correction. Please re-run with FDR control and report q-values for all significant loci. In addition, the sample size justification is missing — please add a power analysis or at least discuss the detectable effect size.
|
||||
|
||||
2. Figure 3 is blurry and the axis labels are unreadable. As noted in Comment 3, the same issue applies to Figure S2. Please replace all figures with high-resolution versions.
|
||||
|
||||
3. Minor: please add the standard error estimates to Supplementary Table 2.
|
||||
|
||||
Reviewer #2:
|
||||
|
||||
1. The claim of novelty overstates the advance relative to Zhang et al. 2023. The introduction should more clearly delineate what is new.
|
||||
|
||||
2. How were the batch effects between the two sequencing centers corrected? This is essential to support the conclusions.
|
||||
@@ -1,27 +0,0 @@
|
||||
# review_service/.env.example —— 运行时配置模板
|
||||
# 用法: cp .env.example .env 后填写(.env 已被仓库 .gitignore 忽略,勿提交)
|
||||
# restart.sh 会 `set -a; source .env` 导出给 uvicorn 进程。
|
||||
|
||||
# ---- LLM(DeepSeek;REVIEW_LLM_* 优先 → DOCGEN_LLM_* 回退 → 内置默认,见 services/llm.py)----
|
||||
# 最简做法:从部署树抄一行即可复用同一把 key:
|
||||
# grep '^DOCGEN_LLM_API_KEY=' /home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new/sdk/.env >> .env
|
||||
REVIEW_LLM_BASE_URL=https://api.deepseek.com/v1
|
||||
REVIEW_LLM_API_KEY=
|
||||
REVIEW_LLM_MODEL=deepseek-chat
|
||||
#REVIEW_LLM_TIMEOUT=300
|
||||
|
||||
# ---- 上游(只读消费)----
|
||||
#REVIEW_VSTORE_BASE_URL=http://127.0.0.1:8094
|
||||
#REVIEW_METHODOLOGY_BASE_URL=http://127.0.0.1:8092 # 预留(MinerU 重链路,当前未用)
|
||||
|
||||
# ---- PDF 解析(经 vstore)----
|
||||
#REVIEW_PARSE_TIMEOUT=180 # ingest→取文→清理 总读超时(秒)
|
||||
#REVIEW_PARSE_DELETE_AFTER=1 # 1=解析后删除 vstore 临时文档(0=保留便于排查)
|
||||
|
||||
# ---- 自审稿 ----
|
||||
#REVIEW_PRESUBMIT_READ_TIMEOUT=360 # LLM 长调用读超时(秒)
|
||||
#REVIEW_MAX_TEXT_CHARS=60000 # 送 LLM 的正文截断上限
|
||||
#REVIEW_MAX_PDF_BYTES=52428800 # 上传 PDF 上限 50MB
|
||||
|
||||
# ---- 服务(restart.sh 读取)----
|
||||
#REVIEW_BIND_HOST=127.0.0.1
|
||||
@@ -1,7 +0,0 @@
|
||||
"""review_service —— 科研投稿与智能审稿助手(8100)。
|
||||
|
||||
独立部署树(模式同 paper_writer_service 8097 / bioflow_service 8098):
|
||||
- BFF(sdk 8090) 经 routers/review_proxy.py 条件反代 /api/review/*(REVIEW_ENABLED)。
|
||||
- 上游只读消费:vstore 8094(PDF 解析)、DeepSeek(LLM,REVIEW_LLM_*/DOCGEN_LLM_*)。
|
||||
- 约束见根目录 AGENTS.md:审稿相关新代码只允许落在本目录。
|
||||
"""
|
||||
@@ -1 +0,0 @@
|
||||
"""review 服务(8100)应用包。"""
|
||||
@@ -1,112 +0,0 @@
|
||||
"""config.py —— review 服务(8100)环境变量驱动配置(pydantic-settings)。
|
||||
|
||||
全部键可用环境变量覆盖(前缀 REVIEW_)。LLM 沿用 docgen 的裸键模式:
|
||||
优先 REVIEW_LLM_*,回退 DOCGEN_LLM_*(与 paper_writer_service 共享同一套
|
||||
DeepSeek 凭据,由 services/llm.py 直接读 os.environ)。零硬编码宿主路径。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(
|
||||
env_file=".env",
|
||||
env_file_encoding="utf-8",
|
||||
extra="ignore",
|
||||
env_prefix="REVIEW_",
|
||||
)
|
||||
|
||||
app_name: str = "plantai-review-service"
|
||||
version: str = "0.1.0"
|
||||
|
||||
# ---- 上游(只读消费;不修改 vstore/methodology 任何代码)----
|
||||
# vstore 8094:POST /api/vstore/ingest + GET/DELETE /api/vstore/documents/{id}
|
||||
# (pymupdf 段落解析,同步秒级;methodology 8092 的 MinerU 链路是
|
||||
# 分钟级异步 job,不适配同步自审稿,仅预留地址供后续扩展)
|
||||
vstore_base_url: str = "http://127.0.0.1:8094"
|
||||
methodology_base_url: str = "http://127.0.0.1:8092"
|
||||
|
||||
# ---- PDF 解析 ----
|
||||
parse_timeout: float = 180.0 # vstore ingest→fetch→delete 读超时(秒)
|
||||
parse_delete_after: bool = True # 解析完成后删除 vstore 临时文档(不污染私有库)
|
||||
|
||||
# ---- 自审稿(presubmit)----
|
||||
presubmit_read_timeout: float = 360.0 # LLM 长调用读超时(秒)
|
||||
max_text_chars: int = 150000 # 送 LLM 的正文截断上限(REVIEW_MAX_TEXT_CHARS 覆盖;
|
||||
# token 超 48K 自动降 100K,LLM 400/413 再降 80K 重试)
|
||||
max_pdf_bytes: int = 50 * 1024 * 1024 # 上传 PDF 上限 50MB(对齐 extraction FR-EXTRACT-004)
|
||||
|
||||
# ---- 期刊情报与投稿推荐(recommend)----
|
||||
db_path: str = "review.db" # 独立 SQLite(相对 review_service/;REVIEW_DB_PATH 覆盖)
|
||||
journals_seed_path: str = "data/journals/journals.json" # 种子(相对 review_service/)
|
||||
journal_directory_seed_path: str = "data/journals/source_v4.xlsx" # 目录源 xlsx(相对 review_service/)
|
||||
seed_on_startup: bool = True # 启动自动 seed journal_directory(空表才导;false 关闭)
|
||||
vstore_match_timeout: float = 30.0 # scope 入库/向量检索读超时(秒)
|
||||
recommend_read_timeout: float = 60.0 # LLM 重排读超时(秒)
|
||||
recommend_query_chars: int = 3000 # 送 vstore 的检索 query 截断(防嵌入模型超长)
|
||||
recommend_scope_top_k: int = 10 # search-chunks top_k(vstore 上限 20)
|
||||
|
||||
# ---- 论文重排版(reformat,P1:.tex 源码级变换,不编译)----
|
||||
max_tex_bytes: int = 2 * 1024 * 1024 # 上传 .tex 上限 2MB(PDF 的 50MB 不适用)
|
||||
reformat_dir: str = "data/reformat_jobs" # 产物落盘(相对 review_service/)
|
||||
|
||||
# ---- LaTeX 编译 PDF(compile,P3:tectonic 单文件二进制)----
|
||||
tectonic_path: str = "tools/tectonic" # 相对 review_service/;也可绝对路径
|
||||
tectonic_cache_dir: str = "data/tectonic-cache" # 宏包缓存(部署时预热,运行期只读命中)
|
||||
compile_timeout_s: float = 60.0 # 单次编译墙钟上限,超时整进程组 kill
|
||||
max_compile_pdf_bytes: int = 50 * 1024 * 1024 # 产物 PDF 上限 50MB
|
||||
compile_queue_limit: int = 5 # 等待槽位上限(>5 → QUEUE_FULL 429)
|
||||
compile_retention_days: int = 7 # 编译产物保留天数(后台清扫)
|
||||
|
||||
@property
|
||||
def _service_root(self) -> "Path":
|
||||
# review_service/app/config.py → review_service/(相对本文件定位,不依赖 cwd)
|
||||
return Path(__file__).resolve().parent.parent
|
||||
|
||||
@property
|
||||
def db_path_abs(self) -> "Path":
|
||||
# 绝对 db_path 会被 Path 除法原样返回;相对路径落在 review_service/ 下
|
||||
return self._service_root / self.db_path
|
||||
|
||||
@property
|
||||
def journals_seed_abs(self) -> "Path":
|
||||
return self._service_root / self.journals_seed_path
|
||||
|
||||
@property
|
||||
def journal_directory_seed_abs(self) -> "Path":
|
||||
return self._service_root / self.journal_directory_seed_path
|
||||
|
||||
@property
|
||||
def reformat_dir_abs(self) -> "Path":
|
||||
p = Path(self.reformat_dir)
|
||||
return p if p.is_absolute() else self._service_root / p
|
||||
|
||||
@property
|
||||
def tectonic_abs(self) -> "Path":
|
||||
p = Path(self.tectonic_path)
|
||||
return p if p.is_absolute() else self._service_root / p
|
||||
|
||||
@property
|
||||
def tectonic_cache_abs(self) -> "Path":
|
||||
p = Path(self.tectonic_cache_dir)
|
||||
return p if p.is_absolute() else self._service_root / p
|
||||
|
||||
@property
|
||||
def vstore_base(self) -> str:
|
||||
return self.vstore_base_url.rstrip("/")
|
||||
|
||||
@property
|
||||
def methodology_base(self) -> str:
|
||||
return self.methodology_base_url.rstrip("/")
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_settings() -> Settings:
|
||||
return Settings()
|
||||
|
||||
|
||||
settings = get_settings()
|
||||
@@ -1,191 +0,0 @@
|
||||
"""app/db.py —— review 服务独立 SQLite(aiosqlite + WAL,无 ORM):journals 表。
|
||||
|
||||
- 单连接 + asyncio 写锁(模式对齐 paper_writer_service/bioflow,但表结构与种子
|
||||
逻辑为本模块自研);路径默认 review_service/review.db(REVIEW_DB_PATH 覆盖)。
|
||||
- 种子:data/journals/journals.json → journals 表(幂等 upsert,vstore_doc_id 列
|
||||
永不被种子覆盖——该列记录 scope_desc 在 vstore 8094 的 document_id)。
|
||||
- 只在 review_service/ 内使用,不触碰 BFF/生产服务数据库。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
import aiosqlite
|
||||
|
||||
from .config import get_settings
|
||||
|
||||
_SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS journals (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
scope_desc TEXT NOT NULL,
|
||||
impact_factor REAL,
|
||||
review_time_weeks REAL,
|
||||
acceptance_rate REAL,
|
||||
submission_deadline TEXT,
|
||||
official_url TEXT,
|
||||
last_updated TEXT NOT NULL,
|
||||
vstore_doc_id TEXT
|
||||
);
|
||||
"""
|
||||
|
||||
# 种子 upsert 列(刻意不含 vstore_doc_id:种子刷新不得清掉向量库映射)
|
||||
_SEED_COLS = (
|
||||
"id",
|
||||
"name",
|
||||
"scope_desc",
|
||||
"impact_factor",
|
||||
"review_time_weeks",
|
||||
"acceptance_rate",
|
||||
"submission_deadline",
|
||||
"official_url",
|
||||
"last_updated",
|
||||
)
|
||||
_UPSERT_SQL = (
|
||||
"INSERT INTO journals (" + ", ".join(_SEED_COLS) + ") VALUES ("
|
||||
+ ", ".join("?" for _ in _SEED_COLS) + ") "
|
||||
+ "ON CONFLICT(id) DO UPDATE SET "
|
||||
+ ", ".join(f"{c}=excluded.{c}" for c in _SEED_COLS if c != "id")
|
||||
)
|
||||
|
||||
|
||||
class SeedError(Exception):
|
||||
"""种子文件缺失/损坏(journals.json 不可读)。"""
|
||||
|
||||
|
||||
class Database:
|
||||
"""aiosqlite 单连接封装(WAL;写串行化)。"""
|
||||
|
||||
def __init__(self, path: str | Path | None = None) -> None:
|
||||
self._path: Path | None = Path(path) if path is not None else None
|
||||
self._conn: aiosqlite.Connection | None = None
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
@property
|
||||
def path(self) -> Path:
|
||||
if self._path is None:
|
||||
self._path = get_settings().db_path_abs
|
||||
return self._path
|
||||
|
||||
def set_path(self, path: str | Path) -> None:
|
||||
"""测试注入:在进入 TestClient lifespan 之前预置库路径(须尚未 connect)。"""
|
||||
if self._conn is not None:
|
||||
raise RuntimeError("Database 已连接,无法再改路径(先 close_db())")
|
||||
self._path = Path(path)
|
||||
|
||||
@property
|
||||
def connected(self) -> bool:
|
||||
return self._conn is not None
|
||||
|
||||
async def connect(self) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._conn = await aiosqlite.connect(self.path)
|
||||
self._conn.row_factory = aiosqlite.Row
|
||||
await self._conn.execute("PRAGMA journal_mode=WAL")
|
||||
await self._conn.execute("PRAGMA busy_timeout=5000")
|
||||
await self._conn.executescript(_SCHEMA)
|
||||
await self._conn.commit()
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._conn is not None:
|
||||
await self._conn.close()
|
||||
self._conn = None
|
||||
|
||||
@property
|
||||
def conn(self) -> aiosqlite.Connection:
|
||||
if self._conn is None:
|
||||
raise RuntimeError("Database 未初始化:先 await init_db()")
|
||||
return self._conn
|
||||
|
||||
async def fetchall(self, sql: str, params: Iterable[Any] = ()) -> list[dict[str, Any]]:
|
||||
async with self.conn.execute(sql, tuple(params)) as cur:
|
||||
rows = await cur.fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
async def fetchone(self, sql: str, params: Iterable[Any] = ()) -> dict[str, Any] | None:
|
||||
async with self.conn.execute(sql, tuple(params)) as cur:
|
||||
row = await cur.fetchone()
|
||||
return dict(row) if row is not None else None
|
||||
|
||||
async def execute(self, sql: str, params: Iterable[Any] = ()) -> int:
|
||||
async with self._lock:
|
||||
cur = await self.conn.execute(sql, tuple(params))
|
||||
await self._conn.commit() # type: ignore[union-attr]
|
||||
return cur.lastrowid or 0
|
||||
|
||||
async def executemany(self, sql: str, seq: list[tuple[Any, ...]]) -> None:
|
||||
async with self._lock:
|
||||
await self.conn.executemany(sql, seq)
|
||||
await self._conn.commit() # type: ignore[union-attr]
|
||||
|
||||
|
||||
db = Database()
|
||||
|
||||
|
||||
async def init_db(path: str | Path | None = None) -> None:
|
||||
"""幂等初始化:已连接则跳过(测试可在 lifespan 前 set_path 注入路径)。"""
|
||||
if path is not None:
|
||||
db.set_path(path)
|
||||
if db.connected:
|
||||
return
|
||||
await db.connect()
|
||||
|
||||
|
||||
async def close_db() -> None:
|
||||
await db.close()
|
||||
|
||||
|
||||
def set_path(path: str | Path) -> None:
|
||||
"""同步注入库路径(测试 fixture 用;须在 init_db 之前调用)。"""
|
||||
db.set_path(path)
|
||||
|
||||
|
||||
async def ensure_journals_seeded() -> int:
|
||||
"""装载种子期刊(幂等)。返回 journals 表行数。"""
|
||||
row = await db.fetchone("SELECT COUNT(*) AS c FROM journals")
|
||||
if row and int(row["c"]) > 0:
|
||||
return int(row["c"])
|
||||
|
||||
seed_file = get_settings().journals_seed_abs
|
||||
try:
|
||||
payload = json.loads(Path(seed_file).read_text(encoding="utf-8"))
|
||||
except FileNotFoundError as exc:
|
||||
raise SeedError(f"期刊种子文件不存在:{seed_file}") from exc
|
||||
except (OSError, ValueError) as exc:
|
||||
raise SeedError(f"期刊种子文件不可读:{exc}") from exc
|
||||
|
||||
journals = payload.get("journals") if isinstance(payload, dict) else None
|
||||
if not isinstance(journals, list) or not journals:
|
||||
raise SeedError(f"期刊种子文件缺少 journals 数组:{seed_file}")
|
||||
|
||||
rows: list[tuple[Any, ...]] = []
|
||||
for item in journals:
|
||||
if not isinstance(item, dict) or not item.get("id") or not item.get("name"):
|
||||
continue
|
||||
rows.append(tuple(item.get(c) for c in _SEED_COLS))
|
||||
if not rows:
|
||||
raise SeedError(f"期刊种子文件无有效条目:{seed_file}")
|
||||
|
||||
await db.executemany(_UPSERT_SQL, rows)
|
||||
return len(rows)
|
||||
|
||||
|
||||
async def list_journals() -> list[dict[str, Any]]:
|
||||
"""全部期刊(impact_factor 降序,NULL 靠后)。"""
|
||||
return await db.fetchall(
|
||||
"SELECT * FROM journals ORDER BY impact_factor DESC, name ASC"
|
||||
)
|
||||
|
||||
|
||||
async def get_journal(journal_id: str) -> dict[str, Any] | None:
|
||||
return await db.fetchone("SELECT * FROM journals WHERE id = ?", (journal_id,))
|
||||
|
||||
|
||||
async def set_vstore_doc_id(journal_id: str, doc_id: str) -> None:
|
||||
"""记录 scope_desc 在 vstore 的 document_id(向量白名单检索用)。"""
|
||||
await db.execute(
|
||||
"UPDATE journals SET vstore_doc_id = ? WHERE id = ?", (doc_id, journal_id)
|
||||
)
|
||||
@@ -1,124 +0,0 @@
|
||||
"""main.py —— review 服务应用入口(uvicorn 入口)。
|
||||
|
||||
- 独立部署树 review_service/(科研投稿与智能审稿助手,端口 8100)。
|
||||
- 端点:/health(GET,BFF 探活反代)+ /api/review/*(自审稿等业务路由)。
|
||||
- 不依赖 BFF/生产服务(8090 等绝不探测/连接);上游仅消费 vstore 8094。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
import httpx
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from .config import get_settings
|
||||
from .db import close_db, db, init_db
|
||||
from .routers import (
|
||||
compile,
|
||||
conferences,
|
||||
format_check,
|
||||
journal_directory,
|
||||
journals,
|
||||
presubmit,
|
||||
reformat,
|
||||
rebuttal,
|
||||
)
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
settings = get_settings()
|
||||
app = FastAPI(title=settings.app_name, version=settings.version)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=False,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
@asynccontextmanager
|
||||
async def _lifespan(_app: FastAPI):
|
||||
# journals 推荐依赖独立 SQLite:启动即建表(幂等;测试可先 set_path 注入
|
||||
# tmp 路径,本函数沿用注入路径)
|
||||
await init_db()
|
||||
# rebuttal Phase 1:独立表,启动幂等建表(自含连接,不走 db.py)
|
||||
try:
|
||||
from .services.rebuttal_store import ensure_table
|
||||
ensure_table()
|
||||
except Exception: # noqa: BLE001
|
||||
logger.warning("rebuttal 建表失败(忽略)", exc_info=True)
|
||||
# 编译产物过期清扫:启动跑一次(失败不阻塞启动)
|
||||
try:
|
||||
from .services.latex_compile import sweep_expired
|
||||
sweep_expired()
|
||||
except Exception: # noqa: BLE001
|
||||
logger.warning("compile sweep 启动清扫失败(忽略)", exc_info=True)
|
||||
# journal_directory 自动 seed:空表才导(xlsx 解析约 1-3s,不堵 event loop;
|
||||
# 路径沿用 db.path 注入链——生产=settings.db_path_abs,测试=tmp)
|
||||
if settings.seed_on_startup:
|
||||
try:
|
||||
from .services.journal_seed import ensure_journal_directory_seeded
|
||||
_t0 = time.time()
|
||||
_result = await asyncio.to_thread(
|
||||
ensure_journal_directory_seeded, db.path)
|
||||
if _result.get("seeded"):
|
||||
_c = _result.get("counts") or {}
|
||||
logger.info(
|
||||
"seeded %s rows (en=%s, zh=%s, %.1fs)",
|
||||
_result["rows"], _c.get("en"), _c.get("zh"),
|
||||
time.time() - _t0)
|
||||
elif _result.get("error"):
|
||||
logger.warning(
|
||||
"journal_directory 启动 seed 失败(忽略,不阻塞启动): %s",
|
||||
_result["error"])
|
||||
else:
|
||||
logger.info("%s", _result.get("reason") or "journal_directory seed skipped")
|
||||
except Exception: # noqa: BLE001
|
||||
logger.warning("journal_directory 启动 seed 失败(忽略,不阻塞启动)", exc_info=True)
|
||||
else:
|
||||
logger.info("journal_directory seed skipped (REVIEW_SEED_ON_STARTUP=false)")
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await close_db()
|
||||
|
||||
app.router.lifespan_context = _lifespan
|
||||
|
||||
@app.get("/health")
|
||||
async def health() -> dict[str, object]:
|
||||
vstore_ok = False
|
||||
vstore_error: str | None = None
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=2.0) as client:
|
||||
resp = await client.get(f"{settings.vstore_base}/api/vstore/health")
|
||||
resp.raise_for_status()
|
||||
vstore_ok = bool(resp.json().get("ok"))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
vstore_error = str(exc)
|
||||
return {
|
||||
"ok": True,
|
||||
"status": "ok" if vstore_ok else "degraded",
|
||||
"upstreams": {"vstore": {"connected": vstore_ok, "error": vstore_error}},
|
||||
"version": settings.version,
|
||||
}
|
||||
|
||||
app.include_router(presubmit.router)
|
||||
app.include_router(rebuttal.router)
|
||||
app.include_router(journals.router)
|
||||
app.include_router(format_check.router)
|
||||
app.include_router(conferences.router)
|
||||
app.include_router(journal_directory.router)
|
||||
app.include_router(reformat.router)
|
||||
app.include_router(compile.router)
|
||||
return app
|
||||
|
||||
|
||||
app = create_app()
|
||||
@@ -1 +0,0 @@
|
||||
"""review 服务路由包。"""
|
||||
@@ -1,184 +0,0 @@
|
||||
"""routers/compile.py —— LaTeX 编译 PDF(P3a:tectonic)。
|
||||
|
||||
端点:
|
||||
POST /api/review/compile multipart:file(.tex 可选) | job_id(可选,二选一)
|
||||
→ {ok, compile_id, pdf_url, log_url, meta}
|
||||
GET /api/review/compile/{cid} 编译元信息(meta.json)
|
||||
GET /api/review/compile/{cid}/pdf output.pdf(FileResponse)
|
||||
GET /api/review/compile/{cid}/log output.log(text/plain)
|
||||
|
||||
信封:业务错误 = HTTP 200 + {ok:false,error:{code,message,detail?,degraded:false}}
|
||||
(QUEUE_FULL 例外:HTTP 429 + 同款信封体,前端仍按 ok/error 单一口径处理)。
|
||||
detail = log 尾 200 行(COMPILE_FAILED / TIMEOUT)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, File, Form, UploadFile
|
||||
from fastapi.responses import FileResponse, JSONResponse, PlainTextResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..config import get_settings
|
||||
from ..services import latex_compile as lc
|
||||
from ..services.latex_compile import (
|
||||
CompileError,
|
||||
QueueFullError,
|
||||
ToolUnavailableError,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-compile"])
|
||||
|
||||
_CID_RE = re.compile(r"^[0-9a-f]{12}$")
|
||||
|
||||
|
||||
class CompileResponse(BaseModel):
|
||||
ok: bool
|
||||
compile_id: str | None = None
|
||||
pdf_url: str | None = None
|
||||
log_url: str | None = None
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str, detail: str = "") -> CompileResponse:
|
||||
err: dict[str, Any] = {"code": code, "message": message, "degraded": False}
|
||||
if detail:
|
||||
err["detail"] = detail
|
||||
return CompileResponse(ok=False, error=err)
|
||||
|
||||
|
||||
def _jobs_root() -> Path:
|
||||
d = get_settings().reformat_dir_abs
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
@router.post("/compile", response_model=CompileResponse)
|
||||
async def compile_tex(
|
||||
file: UploadFile | None = File(None, description="LaTeX 源码(.tex,与 job_id 二选一)"),
|
||||
job_id: str = Form("", description="P1 重排版任务 id(优先于 file)"),
|
||||
) -> CompileResponse:
|
||||
settings = get_settings()
|
||||
|
||||
# ---- 输入解析:job_id 优先,否则直接上传的 .tex ----
|
||||
jid = (job_id or "").strip()
|
||||
if jid:
|
||||
if not _CID_RE.fullmatch(jid):
|
||||
return _error("VALIDATION", "job_id 格式非法(应为 12 位 hex)")
|
||||
src = _jobs_root() / f"{jid}.tex"
|
||||
if not src.is_file():
|
||||
return _error("VALIDATION", "job_id 对应的重排版结果不存在,请重新排版")
|
||||
try:
|
||||
source = src.read_text("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return _error("VALIDATION", "任务源码不是合法 UTF-8")
|
||||
anchor_id = jid
|
||||
else:
|
||||
if file is None:
|
||||
return _error("VALIDATION", "必须提供 file(.tex)或 job_id")
|
||||
filename = file.filename or "paper.tex"
|
||||
if Path(filename).name != filename or ".." in filename:
|
||||
return _error("VALIDATION", "非法文件名")
|
||||
if not filename.lower().endswith(".tex"):
|
||||
return _error("VALIDATION", "仅支持 .tex 文件")
|
||||
content = await file.read()
|
||||
if not content:
|
||||
return _error("VALIDATION", "上传文件为空")
|
||||
if len(content) > settings.max_tex_bytes:
|
||||
return _error("VALIDATION", "源码超过 2MB 上限")
|
||||
try:
|
||||
source = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return _error("VALIDATION", "源码必须为 UTF-8 编码")
|
||||
anchor_id = uuid.uuid4().hex[:12]
|
||||
|
||||
# ---- 编译(隔离目录;失败信封带 log 尾)----
|
||||
lc.prepare_input(anchor_id, source)
|
||||
try:
|
||||
result = await lc.compile_tex(anchor_id)
|
||||
except ToolUnavailableError as exc:
|
||||
return _error("TOOL_UNAVAILABLE", str(exc))
|
||||
except QueueFullError as exc:
|
||||
return JSONResponse(
|
||||
status_code=429,
|
||||
content={
|
||||
"ok": False,
|
||||
"compile_id": None, "pdf_url": None, "log_url": None, "meta": None,
|
||||
"error": {"code": "QUEUE_FULL", "message": str(exc), "degraded": False},
|
||||
},
|
||||
)
|
||||
except CompileError as exc:
|
||||
return _error(exc.code, str(exc), detail=exc.log_tail)
|
||||
|
||||
meta_path = lc.compile_dir(anchor_id) / "meta.json"
|
||||
try:
|
||||
meta = json.loads(meta_path.read_text("utf-8"))
|
||||
except (OSError, ValueError):
|
||||
meta = {"compile_id": anchor_id, "seconds": result["seconds"]}
|
||||
return CompileResponse(
|
||||
ok=True,
|
||||
compile_id=anchor_id,
|
||||
pdf_url=f"/api/review/compile/{anchor_id}/pdf",
|
||||
log_url=f"/api/review/compile/{anchor_id}/log",
|
||||
meta=meta,
|
||||
)
|
||||
|
||||
|
||||
def _cid_dir(cid: str) -> Path | None:
|
||||
if not _CID_RE.fullmatch(cid or ""):
|
||||
return None
|
||||
d = lc.compile_dir(cid)
|
||||
return d if d.is_dir() else None
|
||||
|
||||
|
||||
@router.get("/compile/{compile_id}", response_model=CompileResponse)
|
||||
async def get_compile(compile_id: str) -> CompileResponse:
|
||||
cdir = _cid_dir(compile_id)
|
||||
meta_path = cdir / "meta.json" if cdir else None
|
||||
if meta_path is None or not meta_path.is_file():
|
||||
return _error("NOT_FOUND", "编译任务不存在或已过期,请重新编译")
|
||||
try:
|
||||
meta = json.loads(meta_path.read_text("utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return _error("NOT_FOUND", "编译产物损坏,请重新编译")
|
||||
return CompileResponse(
|
||||
ok=True,
|
||||
compile_id=compile_id,
|
||||
pdf_url=f"/api/review/compile/{compile_id}/pdf",
|
||||
log_url=f"/api/review/compile/{compile_id}/log",
|
||||
meta=meta,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/compile/{compile_id}/pdf", response_model=None)
|
||||
async def download_compile_pdf(compile_id: str) -> FileResponse | JSONResponse:
|
||||
cdir = _cid_dir(compile_id)
|
||||
pdf = cdir / "output.pdf" if cdir else None
|
||||
if pdf is None or not pdf.is_file():
|
||||
return JSONResponse(
|
||||
status_code=200,
|
||||
content={"ok": False, "error": {
|
||||
"code": "NOT_FOUND", "message": "PDF 不存在或已过期", "degraded": False}},
|
||||
)
|
||||
return FileResponse(pdf, media_type="application/pdf", filename=f"{compile_id}.pdf")
|
||||
|
||||
|
||||
@router.get("/compile/{compile_id}/log", response_model=None)
|
||||
async def download_compile_log(compile_id: str) -> PlainTextResponse | JSONResponse:
|
||||
cdir = _cid_dir(compile_id)
|
||||
log = cdir / "output.log" if cdir else None
|
||||
if log is None or not log.is_file():
|
||||
return JSONResponse(
|
||||
status_code=200,
|
||||
content={"ok": False, "error": {
|
||||
"code": "NOT_FOUND", "message": "日志不存在或已过期", "degraded": False}},
|
||||
)
|
||||
return PlainTextResponse(log.read_text("utf-8", errors="replace"))
|
||||
@@ -1,209 +0,0 @@
|
||||
"""routers/conferences.py —— 会议投稿时间线(只读静态种子)。
|
||||
|
||||
- GET /api/review/conferences?domain=&ccf=&upcoming_days=180
|
||||
data/journals/conferences.json → 过滤/排序 → 列表 + meta。
|
||||
- 数据诚实性铁律:每条会议必须带 deadline_source ∈ {placeholder, official}、
|
||||
last_verified 与 deadline_confidence ∈ {high, medium, low},缺失/非法条目加载时
|
||||
跳过并记日志(宁缺毋造);本端点无任何生成日期的逻辑,日期只来自种子常量。
|
||||
v2 共 27/27 全 placeholder,meta.data_source_note 显式声明占位与置信度,
|
||||
由前端渲染「⚠ 日期待核验」及置信度提示。
|
||||
- 过滤语义:domain 对 domain_tags 精确匹配(大小写不敏感);ccf ∈ {A,B,C};
|
||||
upcoming_days=0 不过滤,>0 仅保留 deadline_full_paper ∈ [UTC now, now+N 天],
|
||||
缺失/不可解析截稿日期在过滤开启时排除。排序:deadline_full_paper 升序,
|
||||
缺失/不可解析恒排最后。
|
||||
- 只读无写接口;文件不可读 → ok=false + DATA_UNAVAILABLE 信封;参数非法 →
|
||||
VALIDATION 信封(均 HTTP 200,对齐本服务"错误信封永远 200"约定)。
|
||||
- 静态文件首次请求加载进内存(lru_cache),进程内 O(1);不触碰 SQLite/BFF。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-conferences"])
|
||||
|
||||
# review_service/app/routers/ → parents[2] = review_service(对齐 format_check 写法)
|
||||
_CONFERENCES_PATH = (
|
||||
Path(__file__).resolve().parents[2] / "data" / "journals" / "conferences.json"
|
||||
)
|
||||
|
||||
_CCF_OK = frozenset({"A", "B", "C"})
|
||||
_REQUIRED = ("id", "name", "deadline_source", "last_verified", "deadline_confidence")
|
||||
_CONFIDENCE_OK = frozenset({"high", "medium", "low"})
|
||||
_SOURCE_NOTE = (
|
||||
"会议截稿/通知日期首版均为占位数据(deadline_source=placeholder),"
|
||||
"接入官方渠道前须人工核验;deadline_confidence 标注日期置信度"
|
||||
"(high=官方已公布/历年极稳定,medium=按历年规律推算,low=待核验);"
|
||||
"所有时间按 AoE (UTC-12) 时区。"
|
||||
)
|
||||
|
||||
|
||||
# ---- 请求/响应模型 ----
|
||||
|
||||
class ConferenceItem(BaseModel):
|
||||
"""会议公开字段(种子全字段白名单;未知键丢弃)。"""
|
||||
id: str
|
||||
name: str
|
||||
full_name: str = ""
|
||||
domain_tags: list[str] = []
|
||||
application_tags: list[str] = []
|
||||
ccf_rank: str | None = None
|
||||
deadline_abstract: str | None = None
|
||||
deadline_full_paper: str | None = None
|
||||
notification_date: str | None = None
|
||||
conference_date: str | None = None
|
||||
location: str | None = None
|
||||
official_url: str | None = None
|
||||
timezone_note: str | None = None
|
||||
deadline_source: str
|
||||
last_verified: str
|
||||
deadline_confidence: str = ""
|
||||
|
||||
|
||||
class ConferencesResponse(BaseModel):
|
||||
ok: bool
|
||||
items: list[ConferenceItem] = []
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str) -> ConferencesResponse:
|
||||
return ConferencesResponse(
|
||||
ok=False, items=[], meta=None,
|
||||
error={"code": code, "message": message, "degraded": False},
|
||||
)
|
||||
|
||||
|
||||
# ---- 种子加载(首次调用读盘,进程内缓存)----
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _load_conferences() -> list[dict[str, Any]] | None:
|
||||
"""读 conferences.json → 合法会议列表;文件不可读 → None(DATA_UNAVAILABLE)。"""
|
||||
try:
|
||||
payload = json.loads(_CONFERENCES_PATH.read_text("utf-8"))
|
||||
except (OSError, ValueError) as exc:
|
||||
logger.warning("conferences: 种子文件不可读 %s: %s", _CONFERENCES_PATH, exc)
|
||||
return None
|
||||
raw = payload.get("conferences") if isinstance(payload, dict) else None
|
||||
if not isinstance(raw, list):
|
||||
logger.warning("conferences: 种子缺少 conferences 数组:%s", _CONFERENCES_PATH)
|
||||
return None
|
||||
items: list[dict[str, Any]] = []
|
||||
for conf in raw:
|
||||
if not isinstance(conf, dict):
|
||||
continue
|
||||
missing = [k for k in _REQUIRED if not conf.get(k)]
|
||||
if missing:
|
||||
logger.warning("conferences: 条目缺字段 %s(id=%s),跳过", missing, conf.get("id"))
|
||||
continue
|
||||
if conf.get("deadline_source") not in ("placeholder", "official"):
|
||||
logger.warning(
|
||||
"conferences: deadline_source 非法(id=%s),跳过", conf.get("id")
|
||||
)
|
||||
continue
|
||||
if str(conf.get("deadline_confidence") or "").strip().lower() not in _CONFIDENCE_OK:
|
||||
logger.warning(
|
||||
"conferences: deadline_confidence 非法(id=%s),跳过", conf.get("id")
|
||||
)
|
||||
continue
|
||||
items.append(conf)
|
||||
return items
|
||||
|
||||
|
||||
def _parse_deadline(value: Any) -> datetime | None:
|
||||
"""截稿时间 → aware datetime(naive 视为 UTC);非法 → None。"""
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
return None
|
||||
try:
|
||||
dt = datetime.fromisoformat(value.strip())
|
||||
except ValueError:
|
||||
return None
|
||||
return dt.replace(tzinfo=timezone.utc) if dt.tzinfo is None else dt
|
||||
|
||||
|
||||
# ---- GET /api/review/conferences ----
|
||||
|
||||
@router.get("/conferences", response_model=ConferencesResponse)
|
||||
async def conferences_list(
|
||||
domain: str = "",
|
||||
ccf: str = "",
|
||||
upcoming_days: int = 180,
|
||||
) -> ConferencesResponse:
|
||||
"""只读:会议投稿时间线(无写/删/改接口)。
|
||||
|
||||
- ?domain= 对 domain_tags 精确匹配(大小写不敏感),空=不过滤;
|
||||
- ?ccf= ccf_rank ∈ A/B/C,空=不过滤,非法值 → VALIDATION 信封;
|
||||
- ?upcoming_days= int≥0,默认 180;0=不过滤,>0 仅保留截稿在
|
||||
[UTC now, now+N 天] 内的会议,负数 → VALIDATION 信封;
|
||||
- 排序按 deadline_full_paper 升序,缺失/不可解析恒排最后;
|
||||
- 空结果 → items=[] + meta.count=0(HTTP 200,不报错)。
|
||||
"""
|
||||
ccf_norm = (ccf or "").strip().upper()
|
||||
if ccf_norm and ccf_norm not in _CCF_OK:
|
||||
return _error("VALIDATION", f"不支持的 ccf={ccf!r},可选:A / B / C")
|
||||
if upcoming_days < 0:
|
||||
return _error(
|
||||
"VALIDATION", f"upcoming_days 必须 ≥ 0,收到 {upcoming_days}"
|
||||
)
|
||||
|
||||
all_items = _load_conferences()
|
||||
if all_items is None:
|
||||
return _error("DATA_UNAVAILABLE", "会议种子数据不可读,请联系管理员")
|
||||
|
||||
items = list(all_items)
|
||||
|
||||
dom = (domain or "").strip().casefold()
|
||||
if dom:
|
||||
items = [
|
||||
c for c in items
|
||||
if any(
|
||||
str(t).strip().casefold() == dom
|
||||
for t in (c.get("domain_tags") or [])
|
||||
)
|
||||
]
|
||||
if ccf_norm:
|
||||
items = [
|
||||
c for c in items
|
||||
if str(c.get("ccf_rank") or "").strip().upper() == ccf_norm
|
||||
]
|
||||
if upcoming_days > 0:
|
||||
now = datetime.now(timezone.utc)
|
||||
limit = now + timedelta(days=upcoming_days)
|
||||
items = [
|
||||
c for c in items
|
||||
if (dl := _parse_deadline(c.get("deadline_full_paper"))) is not None
|
||||
and now <= dl <= limit
|
||||
]
|
||||
|
||||
def _sort_key(c: dict[str, Any]) -> tuple[bool, datetime]:
|
||||
dl = _parse_deadline(c.get("deadline_full_paper"))
|
||||
if dl is None:
|
||||
return (True, datetime.max.replace(tzinfo=timezone.utc))
|
||||
return (False, dl)
|
||||
|
||||
items.sort(key=_sort_key)
|
||||
|
||||
out = [ConferenceItem(**c) for c in items]
|
||||
verified = max(
|
||||
(str(c.get("last_verified") or "") for c in all_items), default=""
|
||||
)
|
||||
meta: dict[str, Any] = {
|
||||
"count": len(out),
|
||||
"last_verified": verified or None,
|
||||
"data_source_note": _SOURCE_NOTE,
|
||||
"query": {
|
||||
"domain": (domain or "").strip(),
|
||||
"ccf": ccf_norm,
|
||||
"upcoming_days": upcoming_days,
|
||||
},
|
||||
}
|
||||
return ConferencesResponse(ok=True, items=out, meta=meta, error=None)
|
||||
@@ -1,81 +0,0 @@
|
||||
"""routers/format_check.py —— 版式与规范检查(POST /api/review/format-check)。
|
||||
|
||||
纯规则引擎(services/format_check.py):PDF/Word 本地解档取版式元数据 +
|
||||
journals.json 规则集(format_rules_default + 每条 format_rules 带 rules_source)。
|
||||
无 LLM / 无 vstore,同步执行通常 <1s;恒 200 + {ok, checks, summary, meta|error} 信封。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, File, Form, UploadFile
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..config import get_settings
|
||||
from ..services import format_check
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-format"])
|
||||
|
||||
|
||||
class CheckItem(BaseModel):
|
||||
key: str
|
||||
name: str
|
||||
status: str # pass | warn | fail
|
||||
detail: str
|
||||
suggestion: str = ""
|
||||
|
||||
|
||||
class FormatCheckResponse(BaseModel):
|
||||
ok: bool
|
||||
checks: list[CheckItem] | None = None
|
||||
summary: dict[str, int] | None = None
|
||||
meta: dict[str, object] | None = None
|
||||
error: dict[str, object] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str, *, degraded: bool = False) -> FormatCheckResponse:
|
||||
return FormatCheckResponse(
|
||||
ok=False, error={"code": code, "message": message, "degraded": degraded},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/format-check", response_model=FormatCheckResponse)
|
||||
async def format_check_endpoint(
|
||||
file: UploadFile = File(..., description="待检查的 PDF / Word(.docx) / LaTeX(.tex) 文档"),
|
||||
target_journal: str = Form("", description="目标期刊(可选,传 journals 的 id)"),
|
||||
) -> FormatCheckResponse:
|
||||
settings = get_settings()
|
||||
|
||||
filename = file.filename or ""
|
||||
low = filename.lower()
|
||||
if low.endswith(".pdf"):
|
||||
kind = "pdf"
|
||||
elif low.endswith(".docx"):
|
||||
kind = "docx"
|
||||
elif low.endswith(".tex"):
|
||||
kind = "tex"
|
||||
else:
|
||||
return _error("VALIDATION", "仅支持 PDF / Word(.docx)/ LaTeX(.tex)文件")
|
||||
|
||||
content = await file.read()
|
||||
if not content:
|
||||
return _error("VALIDATION", "上传文件为空")
|
||||
if len(content) > settings.max_pdf_bytes:
|
||||
return _error("VALIDATION", "上传文件超过 50MB 上限")
|
||||
|
||||
try:
|
||||
result = format_check.check_document(content, kind, (target_journal or "").strip())
|
||||
except format_check.FormatCheckError as exc:
|
||||
logger.warning("format-check: 解档失败:%s", exc)
|
||||
return _error("PARSE_FAILED", str(exc))
|
||||
|
||||
meta = dict(result.get("meta") or {})
|
||||
meta["filename"] = filename
|
||||
return FormatCheckResponse(
|
||||
ok=True,
|
||||
checks=[CheckItem(**c) for c in result["checks"]],
|
||||
summary=result["summary"],
|
||||
meta=meta,
|
||||
)
|
||||
@@ -1,280 +0,0 @@
|
||||
"""routers/journal_directory.py —— 期刊目录浏览(1056 本,只读分页)。
|
||||
|
||||
- GET /api/review/journals/directory:journal_directory 表(scripts/import_journals.py
|
||||
导入的 v4 目录:2025 中科院分区 + 2025 JIF(2026 年 JCR 发布)+ 年报 2025 版
|
||||
(中文统计年 2024)+ 官网网址/核对状态)→ 过滤/排序/分页 → 列表 + meta。
|
||||
- 与 GET /journals(journals.json 精选 5 本,供 FormatPanel 下拉/recommend 语义)物理
|
||||
分离(方案 A);本端点绝不读写 journals 表,recommend/journal_kb/journal_rank 零触碰。
|
||||
- 读路径只 SELECT;表由导入脚本拥有(db.py 零改动);表不存在 → data_imported=false
|
||||
优雅降级(HTTP 200 空列表)。
|
||||
|
||||
读时归一化(拍板细节 1-3,均在 API 层做,不动脚本/DB 原值):
|
||||
细节1 lang=zh → cas_partition 恒 null + cas_partition_note="不按中科院分区口径列示";
|
||||
细节2 jif_note 源文本映射(v4 值域已是用户可读文本,旧值映射保留兜底);
|
||||
细节3 缺值即矛盾清零:en 且 jif=null → jif_source_url=null。
|
||||
(zh 的 IF 载体是 composite/comprehensive,其 jif_source_url 本恒 null。)
|
||||
决策2:响应不下发"来源"文本字段(partition_source/jif_source/data_source_label
|
||||
已移除);jif_source_url/data_source_url 保留供展开行链接。
|
||||
决策4(来源展示优化):partition_source_url 不再下发——DB 原值分毫未动
|
||||
(仍 SELECT * 读出后被模型丢弃),响应改为常量字段 partition_source_note
|
||||
文案,前端以灰色说明文字替代原"分区来源"超链接。
|
||||
新增 official_check_status / jcr_release_year。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import sqlite3
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..db import db
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-directory"])
|
||||
|
||||
_LANGS = frozenset({"en", "zh"})
|
||||
# 排序列白名单(列名不可参数化 → 白名单防注入)
|
||||
_FIELDS = frozenset({
|
||||
"name",
|
||||
"jif",
|
||||
"composite_impact_factor",
|
||||
"comprehensive_impact_factor",
|
||||
"research_direction",
|
||||
"cas_partition",
|
||||
})
|
||||
_ORDERS = frozenset({"asc", "desc"})
|
||||
_PARTS = {"1": "1区", "2": "2区", "3": "3区", "4": "4区"}
|
||||
_MAX_SIZE = 200
|
||||
|
||||
# 细节2:源核对状态 → 用户可读 note
|
||||
_JIF_NOTE_MAP = {
|
||||
"源表标N/A": "未被2024 JCR收录",
|
||||
"所用年度来源未检得": "本次汇编未查到,建议查阅最新 JCR",
|
||||
}
|
||||
_ZH_PART_NOTE = "不按中科院分区口径列示"
|
||||
# 决策4:分区来源改为常量说明文案(不下发 partition_source_url;DB 原值不动)
|
||||
_PARTITION_SOURCE_NOTE = (
|
||||
"2025 中科院官方表(2026 年起停更;官方入口 fenqubiao.com 需机构登录)"
|
||||
)
|
||||
|
||||
_NOTE_EN = (
|
||||
"分区:2025年中科院公开表(多表比对,2026年起停止更新);"
|
||||
"JIF:2025年度指标(2026年JCR发布,两份公开汇编一致或已核对出版社官方指标);"
|
||||
"整理日期2026-10-10。空白=来源未给值≠0;研究方向/期刊标记/文章类型为导航标签,非官方评级。"
|
||||
)
|
||||
_NOTE_ZH = (
|
||||
"指标:国内自然科学期刊年报2025版(统计年2024,official,含复合/综合IF两口径);"
|
||||
"中文期刊不按中科院分区口径列示;国内IF与JIF统计源不同,不可混排。"
|
||||
)
|
||||
_UNIMPORTED_NOTE = "目录数据未导入(请先运行 scripts/import_journals.py)"
|
||||
|
||||
|
||||
# ---- 响应模型 ----
|
||||
|
||||
class DirectoryItem(BaseModel):
|
||||
"""目录公开字段(DB 全列白名单;imported_at 等内部列丢弃)。"""
|
||||
id: str
|
||||
lang: str
|
||||
name: str
|
||||
research_direction: str | None = None
|
||||
cas_partition: str | None = None
|
||||
cas_partition_note: str | None = None
|
||||
cas_partition_year: int | None = None
|
||||
jif: float | None = None
|
||||
jif_year: int | None = None
|
||||
jif_note: str | None = None
|
||||
jif_check_status: str | None = None
|
||||
composite_impact_factor: float | None = None
|
||||
comprehensive_impact_factor: float | None = None
|
||||
metric_year: int | None = None
|
||||
metric_report_year: int | None = None
|
||||
metric_caliber: str | None = None
|
||||
annual_report_page: int | None = None
|
||||
top_flag: str | None = None
|
||||
publisher_series: str | None = None
|
||||
journal_mark: str | None = None
|
||||
article_type: str | None = None
|
||||
issn: str | None = None
|
||||
eissn: str | None = None
|
||||
jcr_disciplines: list[str] = []
|
||||
supplement_note: str | None = None
|
||||
evidence_note: str | None = None
|
||||
official_url: str | None = None
|
||||
official_check_status: str | None = None
|
||||
jcr_release_year: int | None = None
|
||||
jif_source_url: str | None = None
|
||||
partition_source_note: str = _PARTITION_SOURCE_NOTE # 决策4:常量,替代分区来源链接
|
||||
data_source_url: str | None = None
|
||||
|
||||
|
||||
class DirectoryResponse(BaseModel):
|
||||
ok: bool
|
||||
items: list[DirectoryItem] = []
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str) -> DirectoryResponse:
|
||||
return DirectoryResponse(
|
||||
ok=False, items=[], meta=None,
|
||||
error={"code": code, "message": message, "degraded": False},
|
||||
)
|
||||
|
||||
|
||||
# ---- 读时归一化(拍板细节 1-3)----
|
||||
|
||||
def _present(raw: dict[str, Any], lang: str) -> dict[str, Any]:
|
||||
d = dict(raw)
|
||||
# jcr_disciplines:JSON 数组字符串 → list
|
||||
jcr = d.get("jcr_disciplines")
|
||||
if jcr:
|
||||
try:
|
||||
parsed = json.loads(jcr)
|
||||
d["jcr_disciplines"] = parsed if isinstance(parsed, list) else []
|
||||
except (TypeError, ValueError):
|
||||
d["jcr_disciplines"] = []
|
||||
else:
|
||||
d["jcr_disciplines"] = []
|
||||
|
||||
# 细节2:jif_note 源文本 → 用户可读
|
||||
note = d.get("jif_note")
|
||||
if note in _JIF_NOTE_MAP:
|
||||
d["jif_note"] = _JIF_NOTE_MAP[note]
|
||||
|
||||
# 细节1:zh 分区归一(DB 原值"不按此口径列示"只留在库里)
|
||||
if lang == "zh":
|
||||
d["cas_partition"] = None
|
||||
d["cas_partition_note"] = _ZH_PART_NOTE
|
||||
|
||||
# 细节3:缺值 → 来源链接清零(消除 null 值却挂来源链接的矛盾;
|
||||
# 分区来源链接已按决策4 移出响应,改为常量 partition_source_note)
|
||||
if lang == "en":
|
||||
if d.get("jif") is None:
|
||||
d["jif_source_url"] = None
|
||||
return d
|
||||
|
||||
|
||||
def _escape_like(s: str) -> str:
|
||||
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
|
||||
|
||||
# ---- GET /api/review/journals/directory ----
|
||||
|
||||
@router.get("/journals/directory", response_model=DirectoryResponse)
|
||||
async def journals_directory(
|
||||
lang: str = "en",
|
||||
domain: str = "",
|
||||
cas_partition: str = "",
|
||||
q: str = "",
|
||||
page: int = 1,
|
||||
size: int = 50,
|
||||
field: str = "",
|
||||
order: str = "",
|
||||
) -> DirectoryResponse:
|
||||
"""只读:期刊目录分页浏览(无写接口)。
|
||||
|
||||
- ?lang=en|zh(默认 en;**强制单语,保证国际/中文永不混排**);
|
||||
- ?domain= 研究方向精确匹配;?cas_partition=1|2|3|4(仅 lang=en,zh 忽略);
|
||||
- ?q= 期刊名模糊(ASCII 大小写不敏感);?page≥1 ?size 1-200(默认 50);
|
||||
- ?field= 排序白名单(name/jif/composite_impact_factor/
|
||||
comprehensive_impact_factor/research_direction/cas_partition),?order=asc|desc;
|
||||
未传 field → 默认排序 en=jif 降序 / zh=复合IF 降序(NULL 恒排最后);
|
||||
- 参数非法 → VALIDATION 信封;表不存在 → data_imported=false 空列表(HTTP 200)。
|
||||
"""
|
||||
lang_n = (lang or "").strip().lower()
|
||||
if lang_n not in _LANGS:
|
||||
return _error("VALIDATION", f"不支持的 lang={lang!r},可选:en / zh")
|
||||
if page < 1:
|
||||
return _error("VALIDATION", f"page 必须 ≥ 1,收到 {page}")
|
||||
if size < 1 or size > _MAX_SIZE:
|
||||
return _error("VALIDATION", f"size 必须在 1-{_MAX_SIZE},收到 {size}")
|
||||
field_n = (field or "").strip()
|
||||
if field_n and field_n not in _FIELDS:
|
||||
return _error(
|
||||
"VALIDATION",
|
||||
f"不支持的 field={field!r},可选:{', '.join(sorted(_FIELDS))}",
|
||||
)
|
||||
order_n = (order or "").strip().lower()
|
||||
if order_n and order_n not in _ORDERS:
|
||||
return _error("VALIDATION", f"不支持的 order={order!r},可选:asc / desc")
|
||||
part_q = (cas_partition or "").strip()
|
||||
if part_q:
|
||||
part_q = _PARTS.get(part_q, _PARTS.get(part_q.rstrip("区").strip(), ""))
|
||||
if not part_q:
|
||||
return _error(
|
||||
"VALIDATION",
|
||||
f"不支持的 cas_partition={cas_partition!r},可选:1 / 2 / 3 / 4",
|
||||
)
|
||||
|
||||
# 过滤条件(白名单拼接;参数全部占位符)
|
||||
where = ["lang = ?"]
|
||||
params: list[Any] = [lang_n]
|
||||
domain_n = (domain or "").strip()
|
||||
if domain_n:
|
||||
where.append("research_direction = ?")
|
||||
params.append(domain_n)
|
||||
if part_q and lang_n == "en": # zh 不分区:忽略(meta 回显)
|
||||
where.append("cas_partition = ?")
|
||||
params.append(part_q)
|
||||
q_n = (q or "").strip()
|
||||
if q_n:
|
||||
where.append("name LIKE ? ESCAPE '\\'")
|
||||
params.append(f"%{_escape_like(q_n)}%")
|
||||
where_sql = " AND ".join(where)
|
||||
|
||||
# 排序(列名出自白名单;NULL 恒排最后)
|
||||
if field_n:
|
||||
dir_sql = "DESC" if order_n == "desc" else "ASC"
|
||||
order_sql = f"{field_n} IS NULL, {field_n} {dir_sql}"
|
||||
else:
|
||||
col = "jif" if lang_n == "en" else "composite_impact_factor"
|
||||
order_sql = f"{col} IS NULL, {col} DESC"
|
||||
|
||||
try:
|
||||
total_row = await db.fetchone(
|
||||
f"SELECT COUNT(*) AS c FROM journal_directory WHERE {where_sql}", params # noqa: S608
|
||||
)
|
||||
total = int(total_row["c"]) if total_row else 0
|
||||
rows = await db.fetchall(
|
||||
f"SELECT * FROM journal_directory WHERE {where_sql} " # noqa: S608
|
||||
f"ORDER BY {order_sql} LIMIT ? OFFSET ?",
|
||||
[*params, size, (page - 1) * size],
|
||||
)
|
||||
dom_rows = await db.fetchall(
|
||||
"SELECT research_direction AS d, COUNT(*) AS c FROM journal_directory "
|
||||
"WHERE lang = ? AND research_direction IS NOT NULL "
|
||||
"GROUP BY 1 ORDER BY c DESC, d ASC",
|
||||
[lang_n],
|
||||
)
|
||||
except sqlite3.OperationalError as exc:
|
||||
# 表不存在(脚本未跑)→ 优雅降级,HTTP 200
|
||||
logger.warning("journal_directory: 表不可用(目录未导入?): %s", exc)
|
||||
return DirectoryResponse(
|
||||
ok=True, items=[],
|
||||
meta={
|
||||
"total": 0, "page": page, "size": size, "lang": lang_n,
|
||||
"domain": domain_n, "cas_partition": part_q, "q": q_n,
|
||||
"domains": [], "data_source_note": _UNIMPORTED_NOTE,
|
||||
"data_imported": False,
|
||||
},
|
||||
error=None,
|
||||
)
|
||||
|
||||
items = [DirectoryItem(**_present(dict(r), lang_n)) for r in rows]
|
||||
meta: dict[str, Any] = {
|
||||
"total": total,
|
||||
"page": page,
|
||||
"size": size,
|
||||
"lang": lang_n,
|
||||
"domain": domain_n,
|
||||
"cas_partition": part_q if lang_n == "en" else "",
|
||||
"q": q_n,
|
||||
"domains": [r["d"] for r in dom_rows if r["d"]],
|
||||
"data_source_note": _NOTE_EN if lang_n == "en" else _NOTE_ZH,
|
||||
"data_imported": True,
|
||||
}
|
||||
return DirectoryResponse(ok=True, items=items, meta=meta, error=None)
|
||||
@@ -1,354 +0,0 @@
|
||||
"""routers/journals.py —— 期刊情报与智能投稿推荐。
|
||||
|
||||
- GET /api/review/journals 只读:journals 表全量列表(?field=/&order= 排序),
|
||||
meta 含 count / last_updated / data_source_note。无任何写接口;空表返回
|
||||
空数组 + meta.count=0(HTTP 200,绝不 500)。
|
||||
- POST /api/review/recommend 智能投稿推荐:vstore 向量匹配(摘要 vs
|
||||
scope_desc,白名单检索)→ LLM 按 id 白名单重排(幻觉期刊丢弃、缺漏规则
|
||||
补位)→ 三档输出:挑战型 / 适配型 / 稳妥型,附匹配理由与风险提示。
|
||||
|
||||
降级哲学(同 presubmit):向量失败 → 全量候选 + vector_ok=false;
|
||||
LLM 失败 → 规则分档 + llm_ok=false;推荐永不 5xx。仅入参非法返回
|
||||
VALIDATION 错误信封(HTTP 200,与本服务其余端点一致)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..config import get_settings
|
||||
from ..db import SeedError, ensure_journals_seeded, list_journals
|
||||
from ..services import journal_kb, journal_rank, llm
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-journals"])
|
||||
|
||||
|
||||
# ---- 请求/响应模型 ----
|
||||
|
||||
class Preferences(BaseModel):
|
||||
max_review_weeks: float | None = Field(default=None, ge=1, le=104)
|
||||
min_impact_factor: float | None = Field(default=None, ge=0, le=200)
|
||||
tier_focus: Literal["all", "challenge", "fit", "safe"] = "all"
|
||||
|
||||
|
||||
class RecommendRequest(BaseModel):
|
||||
abstract: str
|
||||
preferences: Preferences = Preferences()
|
||||
|
||||
|
||||
class JournalItem(BaseModel):
|
||||
id: str
|
||||
name: str
|
||||
impact_factor: float | None = None
|
||||
review_time_weeks: float | None = None
|
||||
acceptance_rate: float | None = None
|
||||
official_url: str | None = None
|
||||
similarity: float | None = None
|
||||
tier: str
|
||||
match_reason: str
|
||||
risk_note: str
|
||||
|
||||
|
||||
class TierBox(BaseModel):
|
||||
label: str
|
||||
items: list[JournalItem]
|
||||
|
||||
|
||||
class RecommendResponse(BaseModel):
|
||||
ok: bool
|
||||
tiers: dict[str, TierBox] | None = None
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
# ---- GET /api/review/journals(只读列表)----
|
||||
|
||||
class JournalBrief(BaseModel):
|
||||
"""期刊公开字段(journals 表全列;不含 vstore_doc_id 内部映射)。
|
||||
|
||||
标签/分区 6 字段不进 SQLite:GET 时按 id 从 journals.json 内存缓存合并;
|
||||
JSON 缺该 id → 空数组/null 默认值。
|
||||
"""
|
||||
id: str
|
||||
name: str
|
||||
scope_desc: str = ""
|
||||
impact_factor: float | None = None
|
||||
review_time_weeks: float | None = None
|
||||
acceptance_rate: float | None = None
|
||||
submission_deadline: str | None = None
|
||||
official_url: str | None = None
|
||||
last_updated: str = ""
|
||||
scope_tags: list[str] = []
|
||||
application_tags: list[str] = []
|
||||
cas_partition: str | None = None
|
||||
jcr_quartile: str | None = None
|
||||
partition_source: str | None = None
|
||||
cross_domain: list[str] = []
|
||||
|
||||
|
||||
class JournalsListResponse(BaseModel):
|
||||
ok: bool
|
||||
items: list[JournalBrief] = []
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
# ?field= 排序白名单(列名无法参数化,白名单防注入;非法值 → VALIDATION 信封)
|
||||
_SORT_FIELDS = frozenset({
|
||||
"id",
|
||||
"name",
|
||||
"impact_factor",
|
||||
"review_time_weeks",
|
||||
"acceptance_rate",
|
||||
"last_updated",
|
||||
})
|
||||
_DATA_SOURCE_NOTE = (
|
||||
"IF/审稿周期为公开近似值,acceptance_rate 为 null 表示暂无公开数据,需人工核验。"
|
||||
"CAS/JCR 分区为首版占位数据,需核验。"
|
||||
)
|
||||
|
||||
# journals.json 新增字段(标签/分区;刻意不进 SQLite,见 C-2 方案)
|
||||
_TAG_FIELDS = (
|
||||
"scope_tags",
|
||||
"application_tags",
|
||||
"cas_partition",
|
||||
"jcr_quartile",
|
||||
"partition_source",
|
||||
"cross_domain",
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _load_tag_extras() -> dict[str, dict[str, Any]]:
|
||||
"""journals.json 标签/分区字段 → {id: {6 新字段}}。
|
||||
|
||||
首次请求读盘后进程内缓存,后续 GET O(1) 按 id 合并(不做每请求磁盘 IO)。
|
||||
JSON 缺该 id → 合并时用空数组/null 默认;文件不可读 → 空 dict 降级(不 500)。
|
||||
"""
|
||||
seed = get_settings().journals_seed_abs
|
||||
try:
|
||||
payload = json.loads(Path(seed).read_text("utf-8"))
|
||||
except (OSError, ValueError) as exc:
|
||||
logger.warning("journals: 标签扩展文件不可读,合并降级为默认值:%s", exc)
|
||||
return {}
|
||||
raw = payload.get("journals") if isinstance(payload, dict) else None
|
||||
if not isinstance(raw, list):
|
||||
return {}
|
||||
out: dict[str, dict[str, Any]] = {}
|
||||
for j in raw:
|
||||
if isinstance(j, dict) and j.get("id"):
|
||||
out[str(j["id"])] = {k: j.get(k) for k in _TAG_FIELDS}
|
||||
return out
|
||||
|
||||
|
||||
def _tag_fields(extra: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""缓存条目 → JournalBrief 关键字参数(JSON 缺该 id → 空数组/null 默认)。"""
|
||||
ex = extra or {}
|
||||
return {
|
||||
"scope_tags": list(ex.get("scope_tags") or []),
|
||||
"application_tags": list(ex.get("application_tags") or []),
|
||||
"cas_partition": ex.get("cas_partition"),
|
||||
"jcr_quartile": ex.get("jcr_quartile"),
|
||||
"partition_source": ex.get("partition_source"),
|
||||
"cross_domain": list(ex.get("cross_domain") or []),
|
||||
}
|
||||
|
||||
|
||||
def _sort_rows(rows: list[dict[str, Any]], field: str, order: str) -> list[dict[str, Any]]:
|
||||
"""按白名单字段排序;NULL 恒排最后;次序不稳定时按 db 默认序(稳定排序保留)。"""
|
||||
present = [r for r in rows if r.get(field) is not None]
|
||||
missing = [r for r in rows if r.get(field) is None]
|
||||
present.sort(key=lambda r: r.get(field), reverse=(order == "desc"))
|
||||
return present + missing
|
||||
|
||||
|
||||
def _list_error(message: str) -> JournalsListResponse:
|
||||
return JournalsListResponse(
|
||||
ok=False, items=[], meta=None,
|
||||
error={"code": "VALIDATION", "message": message, "degraded": False},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/journals", response_model=JournalsListResponse)
|
||||
async def journals_list(
|
||||
field: str = "name",
|
||||
order: str = "asc",
|
||||
) -> JournalsListResponse:
|
||||
"""只读:journals 表全量列表(无写/删/改接口)。
|
||||
|
||||
- ?field= 排序字段白名单:id/name/impact_factor/review_time_weeks/
|
||||
acceptance_rate/last_updated,默认 name;
|
||||
- ?order=asc|desc,默认 asc;
|
||||
- 空表 → items=[] + meta.count=0(HTTP 200,不报错)。种子装载由
|
||||
POST /recommend 首次调用幂等完成,本端点严格只读。
|
||||
"""
|
||||
col = (field or "").strip()
|
||||
if col not in _SORT_FIELDS:
|
||||
return _list_error(
|
||||
f"不支持的排序字段 field={field!r},可选:{', '.join(sorted(_SORT_FIELDS))}"
|
||||
)
|
||||
ord_norm = (order or "asc").strip().lower()
|
||||
if ord_norm not in ("asc", "desc"):
|
||||
return _list_error(f"不支持的排序方向 order={order!r},可选:asc, desc")
|
||||
|
||||
rows = await list_journals()
|
||||
rows = _sort_rows(rows, col, ord_norm)
|
||||
|
||||
tag_extras = _load_tag_extras()
|
||||
items = [
|
||||
JournalBrief(
|
||||
id=str(r.get("id") or ""),
|
||||
name=str(r.get("name") or ""),
|
||||
scope_desc=str(r.get("scope_desc") or ""),
|
||||
impact_factor=r.get("impact_factor"),
|
||||
review_time_weeks=r.get("review_time_weeks"),
|
||||
acceptance_rate=r.get("acceptance_rate"),
|
||||
submission_deadline=r.get("submission_deadline"),
|
||||
official_url=r.get("official_url"),
|
||||
last_updated=str(r.get("last_updated") or ""),
|
||||
**_tag_fields(tag_extras.get(str(r.get("id") or ""))),
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
latest = max((str(r.get("last_updated") or "") for r in rows), default="")
|
||||
meta: dict[str, Any] = {
|
||||
"count": len(items),
|
||||
"last_updated": latest or None,
|
||||
"data_source_note": _DATA_SOURCE_NOTE,
|
||||
}
|
||||
return JournalsListResponse(ok=True, items=items, meta=meta, error=None)
|
||||
|
||||
|
||||
def _error(code: str, message: str, *, degraded: bool = False) -> RecommendResponse:
|
||||
return RecommendResponse(
|
||||
ok=False, tiers=None, meta=None,
|
||||
error={"code": code, "message": message, "degraded": degraded},
|
||||
)
|
||||
|
||||
|
||||
def _item_sort_key(it: JournalItem) -> tuple[int, float, float]:
|
||||
return (
|
||||
1 if it.similarity is None else 0,
|
||||
-(it.similarity or 0.0),
|
||||
-(it.impact_factor or 0.0),
|
||||
)
|
||||
|
||||
|
||||
def _apply_soft_preferences(item: JournalItem, prefs: Preferences) -> None:
|
||||
"""软偏好:不满足只加风险提示,不删条目。"""
|
||||
weeks = item.review_time_weeks
|
||||
if prefs.max_review_weeks is not None and weeks is not None and weeks > prefs.max_review_weeks:
|
||||
prefix = f"⚠ 审稿约 {weeks:g} 周,超出你 {prefs.max_review_weeks:g} 周的周期偏好;"
|
||||
item.risk_note = prefix + ("无" if item.risk_note == "无" else item.risk_note)
|
||||
inf = item.impact_factor
|
||||
if prefs.min_impact_factor is not None:
|
||||
if inf is None:
|
||||
item.risk_note = "⚠ 影响因子未收录,无法确认满足你的要求;" + item.risk_note
|
||||
elif inf < prefs.min_impact_factor:
|
||||
prefix = f"⚠ 影响因子 {inf:g} 低于你 {prefs.min_impact_factor:g} 的要求;"
|
||||
item.risk_note = prefix + ("无" if item.risk_note == "无" else item.risk_note)
|
||||
|
||||
|
||||
@router.post("/recommend", response_model=RecommendResponse)
|
||||
async def recommend(req: RecommendRequest) -> RecommendResponse:
|
||||
settings = get_settings()
|
||||
|
||||
# ---- 入参校验(手动,错误信封与本服务其余端点一致)----
|
||||
abstract = (req.abstract or "").strip()
|
||||
if len(abstract) < 50:
|
||||
return _error("VALIDATION", "abstract 至少需要 50 个字符(请提供摘要或正文片段)")
|
||||
if len(abstract) > 20000:
|
||||
return _error("VALIDATION", "abstract 超过 20000 字符上限")
|
||||
|
||||
prefs = req.preferences
|
||||
|
||||
# ---- 种子 + 期刊元数据 ----
|
||||
try:
|
||||
await ensure_journals_seeded()
|
||||
journals = await list_journals()
|
||||
except SeedError as exc:
|
||||
logger.error("recommend: 种子装载失败:%s", exc)
|
||||
return _error("SEED_UNAVAILABLE", str(exc), degraded=True)
|
||||
if not journals:
|
||||
return _error("SEED_UNAVAILABLE", "期刊库为空", degraded=True)
|
||||
|
||||
# ---- ① vstore 向量匹配(摘要 vs scope_desc)----
|
||||
try:
|
||||
candidates, vector_ok = await journal_kb.match(abstract, journals)
|
||||
except journal_kb.MatchError as exc:
|
||||
logger.warning("recommend: 向量匹配降级:%s", exc)
|
||||
candidates, vector_ok = [], False
|
||||
if not candidates:
|
||||
# 无有效语义分 → 全量候选(similarity=None),LLM 仍可按 scope 文本重排
|
||||
candidates = [{"journal": j, "similarity": None} for j in journals]
|
||||
vector_ok = False
|
||||
|
||||
# ---- ② LLM 重排(id 白名单硬校验)----
|
||||
prefs_dict = prefs.model_dump(exclude_none=True)
|
||||
llm_ok = True
|
||||
dropped = 0
|
||||
repaired = 0
|
||||
try:
|
||||
rankings, dropped, repaired = await journal_rank.rerank(abstract, candidates, prefs_dict)
|
||||
except (llm.LLMError, journal_rank.RankError, ValueError) as exc:
|
||||
# ValueError:complete_json 最终 json.loads 修复仍失败(同 presubmit 宽容口径)
|
||||
logger.warning("recommend: LLM 重排降级为规则分档:%s", exc)
|
||||
rankings = journal_rank.fallback_rankings(candidates)
|
||||
llm_ok = False
|
||||
|
||||
# ---- ③ 三档组装(名称/指标全部来自 DB,不采信 LLM 文本)----
|
||||
by_id = {str(c["journal"]["id"]): c for c in candidates}
|
||||
tiers: dict[str, list[JournalItem]] = {t: [] for t in journal_rank.TIERS}
|
||||
for r in rankings:
|
||||
cand = by_id.get(r.journal_id)
|
||||
if cand is None:
|
||||
continue
|
||||
j = cand["journal"]
|
||||
sim = cand.get("similarity")
|
||||
item = JournalItem(
|
||||
id=str(j["id"]),
|
||||
name=str(j.get("name") or j["id"]),
|
||||
impact_factor=j.get("impact_factor"),
|
||||
review_time_weeks=j.get("review_time_weeks"),
|
||||
acceptance_rate=j.get("acceptance_rate"),
|
||||
official_url=j.get("official_url"),
|
||||
similarity=round(float(sim), 4) if sim is not None else None,
|
||||
tier=r.tier,
|
||||
match_reason=r.match_reason,
|
||||
risk_note=r.risk_note,
|
||||
)
|
||||
_apply_soft_preferences(item, prefs)
|
||||
tiers.setdefault(r.tier, []).append(item)
|
||||
|
||||
# 档内排序:相似度(None 靠后)→ 影响因子
|
||||
for box in tiers.values():
|
||||
box.sort(key=_item_sort_key)
|
||||
|
||||
# tier_focus:软展示聚焦(默认 all;聚焦时其余档 items 置空,label 保留)
|
||||
out_tiers: dict[str, TierBox] = {}
|
||||
for t in journal_rank.TIERS:
|
||||
items = tiers.get(t, [])
|
||||
if prefs.tier_focus != "all" and t != prefs.tier_focus:
|
||||
items = []
|
||||
out_tiers[t] = TierBox(label=journal_rank.TIER_LABELS[t], items=items)
|
||||
|
||||
meta: dict[str, Any] = {
|
||||
"model": llm._llm_config()["model"],
|
||||
"vector_ok": vector_ok,
|
||||
"llm_ok": llm_ok,
|
||||
"seed_count": len(journals),
|
||||
"candidate_count": len(candidates),
|
||||
"dropped_hallucinated": dropped,
|
||||
"repaired": repaired,
|
||||
"preferences": prefs_dict,
|
||||
"abstract_chars": len(abstract),
|
||||
}
|
||||
return RecommendResponse(ok=True, tiers=out_tiers, meta=meta, error=None)
|
||||
@@ -1,269 +0,0 @@
|
||||
"""routers/presubmit.py —— 投稿前自审稿(POST /api/review/presubmit)。
|
||||
|
||||
流程:上传 PDF → paper_parse(vstore 8094 解析)→ 段落感知装箱 + token 双层保护
|
||||
→ LLM 结构化审稿(DeepSeek json_object,System Prompt 为植物基因组学六维专属
|
||||
清单,见 services/self_review.py)→ coerce 归一化 → 关键词扫描器证伪硬性缺项
|
||||
(services/paper_scanner.py)→ 规则引擎计算 recommendation。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
|
||||
from fastapi import APIRouter, File, Form, UploadFile
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..config import get_settings
|
||||
from ..services import llm, paper_parse, paper_scanner, self_review
|
||||
from ..services.recommend_engine import decide_recommendation, resolve_tier
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-presubmit"])
|
||||
|
||||
# ---- token 双层保护(2026-10-12)----
|
||||
# 第一层:估算 token 超阈值 → 自动降到 TOKEN_GUARD_REPACK_CHARS 字符重装箱
|
||||
# 第二层:LLM 返回 HTTP 400/413 → 降到 LLM_REJECT_RETRY_CHARS 字符重试一次
|
||||
_EST_TOKENS_DIVISOR = 1.5 # 粗略上界:英文 ~4 字符/token,1.5 偏保守
|
||||
_TOKEN_GUARD_THRESHOLD = 48000
|
||||
_TOKEN_GUARD_REPACK_CHARS = 100000
|
||||
_LLM_REJECT_RETRY_CHARS = 80000
|
||||
# 无节标题可辨时的尾窗(捞页尾 Data Availability 块)
|
||||
_TAIL_FALLBACK_CHARS = 5000
|
||||
|
||||
# 节标题(行首、允许编号前缀、独占一行);键 → 装箱分组名
|
||||
_SECTION_HEADER_RE = re.compile(
|
||||
r"^[ \t]*(?:\d+(?:\.\d+)*\.?\s+)?"
|
||||
r"(Abstract|Summary|Introduction|Results|Discussion|Methods"
|
||||
r"|Materials\s+and\s+Methods"
|
||||
r"|Data\s+(?:and\s+(?:code\s+)?)?[Aa]vailability"
|
||||
r"|Availability\s+of\s+[Dd]ata"
|
||||
r"|Data\s+and\s+[Mm]aterials\s+[Aa]vailability"
|
||||
r"|References|Acknowledg(?:e)?ments?)\s*$",
|
||||
re.IGNORECASE | re.MULTILINE,
|
||||
)
|
||||
_SECTION_GROUP = {
|
||||
"abstract": "abstract", "summary": "abstract",
|
||||
"introduction": "introduction",
|
||||
"results": "results",
|
||||
"discussion": "discussion",
|
||||
"methods": "methods", "materials and methods": "methods",
|
||||
"data availability": "data_availability",
|
||||
"data and code availability": "data_availability",
|
||||
"data availability statement": "data_availability",
|
||||
"availability of data": "data_availability",
|
||||
"data and materials availability": "data_availability",
|
||||
"references": "references", "acknowledgements": "acknowledgements",
|
||||
"acknowledgments": "acknowledgements",
|
||||
}
|
||||
# 装箱优先级(首条命中优先;methods 限 60% 预算——六维清单大量依赖 Methods)
|
||||
_PACK_PRIORITY = ("abstract", "data_availability", "methods", "results", "discussion")
|
||||
_METHODS_CAP_RATIO = 0.6
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""粗略 token 上界估计(len/1.5),无需 tokenizer 依赖。"""
|
||||
return int(len(text) / _EST_TOKENS_DIVISOR)
|
||||
|
||||
|
||||
def _split_segments(text: str) -> list[tuple[str, str]] | None:
|
||||
"""按节标题切段 → [(group, content)...]。无可辨标题 → None。"""
|
||||
matches = list(_SECTION_HEADER_RE.finditer(text))
|
||||
if not matches:
|
||||
return None
|
||||
segments: list[tuple[str, str]] = []
|
||||
if matches[0].start() > 0:
|
||||
segments.append(("preamble", text[:matches[0].start()]))
|
||||
for i, m in enumerate(matches):
|
||||
end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
|
||||
group = _SECTION_GROUP.get(m.group(1).casefold(), m.group(1).casefold())
|
||||
segments.append((group, text[m.start():end]))
|
||||
return segments
|
||||
|
||||
|
||||
def pack_feed_text(text: str, budget: int) -> tuple[str, list[str]]:
|
||||
"""段落感知装箱 → (feed_text, dropped_sections)。
|
||||
|
||||
预算内:原文返回。超预算:按优先级保留
|
||||
① Abstract ② Data Availability 全文 ③ Methods 前 60% ④ Results
|
||||
⑤ Discussion,其余按原序填满;未收段记入 dropped。
|
||||
无节标题可辨 → 头 + 尾窗拼接(捞页尾 availability 块),不优于现状亦不劣于。
|
||||
"""
|
||||
if len(text) <= budget:
|
||||
return text, []
|
||||
segments = _split_segments(text)
|
||||
if not segments:
|
||||
sep = "\n[...]\n"
|
||||
head = max(0, budget - _TAIL_FALLBACK_CHARS - len(sep))
|
||||
return (
|
||||
text[:head] + sep + text[-_TAIL_FALLBACK_CHARS:],
|
||||
["[无节标题回退:保留头+尾窗]"],
|
||||
)
|
||||
methods_cap = int(budget * _METHODS_CAP_RATIO)
|
||||
chosen: dict[int, str] = {}
|
||||
used = 0
|
||||
for group in _PACK_PRIORITY:
|
||||
for i, (g, content) in enumerate(segments):
|
||||
if g != group or i in chosen:
|
||||
continue
|
||||
room = budget - used
|
||||
if room <= 0:
|
||||
break
|
||||
piece = content[:methods_cap] if group == "methods" else content
|
||||
if len(piece) > room:
|
||||
piece = piece[:room]
|
||||
if not piece:
|
||||
break
|
||||
chosen[i] = piece
|
||||
used += len(piece)
|
||||
break
|
||||
for i, (_, content) in enumerate(segments):
|
||||
if i in chosen or used >= budget:
|
||||
continue
|
||||
room = budget - used
|
||||
chosen[i] = content if len(content) <= room else content[:room]
|
||||
used += len(chosen[i])
|
||||
feed = "".join(chosen[i] for i in sorted(chosen))
|
||||
dropped: list[str] = []
|
||||
for i, (g, _) in enumerate(segments):
|
||||
if i not in chosen and g not in dropped:
|
||||
dropped.append(g)
|
||||
return feed, dropped
|
||||
|
||||
|
||||
def _fit_feed_text(text: str, budget: int) -> tuple[str, list[str]]:
|
||||
"""装箱 + 第一层 token 保护(估算超阈值自动降到 100K 字符重装)。"""
|
||||
feed, dropped = pack_feed_text(text, budget)
|
||||
if estimate_tokens(feed) > _TOKEN_GUARD_THRESHOLD and len(feed) > _TOKEN_GUARD_REPACK_CHARS:
|
||||
logger.info(
|
||||
"presubmit: token 估算 %d > %d,降到 %d 字符重装箱",
|
||||
estimate_tokens(feed), _TOKEN_GUARD_THRESHOLD, _TOKEN_GUARD_REPACK_CHARS,
|
||||
)
|
||||
feed, dropped = pack_feed_text(text, _TOKEN_GUARD_REPACK_CHARS)
|
||||
return feed, dropped
|
||||
|
||||
|
||||
class PresubmitResponse(BaseModel):
|
||||
ok: bool
|
||||
report: self_review.SelfReviewReport | None = None
|
||||
error: dict[str, object] | None = None
|
||||
meta: dict[str, object] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str, *, degraded: bool = False) -> PresubmitResponse:
|
||||
return PresubmitResponse(
|
||||
ok=False, report=None,
|
||||
error={"code": code, "message": message, "degraded": degraded},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/presubmit", response_model=PresubmitResponse)
|
||||
async def presubmit_review(
|
||||
file: UploadFile = File(..., description="待审稿 PDF"),
|
||||
target_journal: str = Form("", description="目标期刊(可选,影响格式类意见)"),
|
||||
) -> PresubmitResponse:
|
||||
settings = get_settings()
|
||||
|
||||
filename = file.filename or "paper.pdf"
|
||||
if not filename.lower().endswith(".pdf"):
|
||||
return _error("VALIDATION", "仅支持 PDF 文件")
|
||||
|
||||
content = await file.read()
|
||||
if not content:
|
||||
return _error("VALIDATION", "上传文件为空")
|
||||
if len(content) > settings.max_pdf_bytes:
|
||||
return _error("VALIDATION", "PDF 超过 50MB 上限")
|
||||
|
||||
# ---- 解析(vstore 8094)----
|
||||
try:
|
||||
text = await paper_parse.parse_pdf_bytes(content, filename=filename)
|
||||
except paper_parse.ParseError as exc:
|
||||
logger.warning("presubmit: 解析失败:%s", exc)
|
||||
return _error("PARSE_FAILED", str(exc), degraded=True)
|
||||
|
||||
truncated = len(text) > settings.max_text_chars
|
||||
feed_text, dropped_sections = _fit_feed_text(text, settings.max_text_chars)
|
||||
|
||||
# ---- LLM 结构化自审(植物基因组学六维清单)----
|
||||
# 第二层保护:HTTP 400/413(疑似超长/形态拒绝)→ 降 80K 字符重试一次
|
||||
user_prompt = self_review.build_user_prompt(feed_text, (target_journal or "").strip())
|
||||
try:
|
||||
try:
|
||||
payload = await llm.complete_json(
|
||||
self_review.SYSTEM_PROMPT,
|
||||
user_prompt,
|
||||
temperature=0.2,
|
||||
max_tokens=8000,
|
||||
timeout=settings.presubmit_read_timeout,
|
||||
)
|
||||
except llm.LLMError as exc:
|
||||
msg = str(exc)
|
||||
if ("HTTP 400" in msg or "HTTP 413" in msg) and len(feed_text) > _LLM_REJECT_RETRY_CHARS:
|
||||
logger.warning("presubmit: LLM 拒绝输入(%s),降到 %d 字符重试",
|
||||
msg[:120], _LLM_REJECT_RETRY_CHARS)
|
||||
feed_text, dropped_sections = pack_feed_text(text, _LLM_REJECT_RETRY_CHARS)
|
||||
truncated = True
|
||||
payload = await llm.complete_json(
|
||||
self_review.SYSTEM_PROMPT,
|
||||
self_review.build_user_prompt(feed_text, (target_journal or "").strip()),
|
||||
temperature=0.2,
|
||||
max_tokens=8000,
|
||||
timeout=settings.presubmit_read_timeout,
|
||||
)
|
||||
else:
|
||||
raise
|
||||
except llm.LLMError as exc:
|
||||
logger.error("presubmit: LLM 审稿失败:%s", exc)
|
||||
return _error("UPSTREAM_DOWN", f"审稿模型调用失败:{exc}", degraded=True)
|
||||
|
||||
report = self_review.coerce_report(payload)
|
||||
|
||||
# ---- 关键词扫描器:证伪 LLM 的硬性缺项(有 X 却报缺 X → 真降级至 minor)----
|
||||
# A2(2026-10-12):被冲突的 major 不再留在 major 列表里只剥标签,而是移出
|
||||
# major、append 到 minor(scanner_downgraded=True)——major_count/引擎输入真实
|
||||
# 减少;fatal 条目例外(只记冲突不降级,须人工过目)。
|
||||
signals = paper_scanner.scan_paper_signals(feed_text)
|
||||
kept_majors, demoted_minors, scanner_conflicts = paper_scanner.reconcile_hard_rules(
|
||||
report.major_weaknesses, signals
|
||||
)
|
||||
report.major_weaknesses = kept_majors
|
||||
if demoted_minors:
|
||||
report.minor_weaknesses = list(report.minor_weaknesses) + list(demoted_minors)
|
||||
|
||||
# recommendation 由规则引擎计算(LLM 输出的同名字段已被 coerce 忽略)
|
||||
tier, tier_source = resolve_tier((target_journal or "").strip())
|
||||
decision = decide_recommendation(
|
||||
majors=report.major_weaknesses,
|
||||
minors=report.minor_weaknesses,
|
||||
presentations=report.presentation_format,
|
||||
tier=tier,
|
||||
)
|
||||
rationale = decision["rationale"]
|
||||
rationale["scanner_conflicts"] = scanner_conflicts
|
||||
if scanner_conflicts:
|
||||
rationale["explanation"] += (
|
||||
f" 关键词扫描器与审稿意见存在 {len(scanner_conflicts)} 处冲突,"
|
||||
f"已降级为小修级问题,请人工核对。"
|
||||
)
|
||||
report.recommendation = decision["recommendation"]
|
||||
report.recommendation_rationale = rationale
|
||||
return PresubmitResponse(
|
||||
ok=True,
|
||||
report=report,
|
||||
meta={
|
||||
"filename": filename,
|
||||
"target_journal": (target_journal or "").strip(),
|
||||
"text_chars": len(feed_text),
|
||||
"truncated": truncated,
|
||||
"dropped_sections": dropped_sections,
|
||||
"scanner": {
|
||||
"accession_found": bool((signals.get("accession") or {}).get("found")),
|
||||
"busco_level": (signals.get("busco") or {}).get("level"),
|
||||
"hic_found": bool((signals.get("hic") or {}).get("found")),
|
||||
},
|
||||
"model": llm._llm_config()["model"],
|
||||
"tier": tier,
|
||||
"tier_label": decision["rationale"]["tier_label"],
|
||||
"tier_source": tier_source,
|
||||
},
|
||||
)
|
||||
@@ -1,288 +0,0 @@
|
||||
"""routers/rebuttal.py —— Rebuttal Phase 1(审稿回复助手,10 端点)。
|
||||
|
||||
信封与其余 review 端点一致:业务错误 = HTTP 200 + {ok:false,error:{code,message,degraded}}。
|
||||
- 文本粘贴主入口 / PDF 次要入口(PDF 经 vstore 8094 解析);
|
||||
- 逐条 response → draft(校验C前置)→ confirm(后端重算 human_ack)→ letter 拼装;
|
||||
- letter.md 下载门槛 = human_ack;DELETE 手动清(无 TTL,决策 5)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, Literal
|
||||
|
||||
from fastapi import APIRouter, File, UploadFile
|
||||
from fastapi.responses import PlainTextResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..config import get_settings
|
||||
from ..services import llm, paper_parse
|
||||
from ..services.rebuttal_draft import PART_TYPES, assemble_letter, generate_draft, validate_parts
|
||||
from ..services.rebuttal_parse import parse_review_text
|
||||
from ..services import rebuttal_store
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review/rebuttal", tags=["review-rebuttal"])
|
||||
|
||||
_MAX_RAW_CHARS = 200_000 # 粘贴文本上限(审稿意见通常 <20K,保守给大)
|
||||
|
||||
|
||||
# ---------- 信封 ----------
|
||||
class RebuttalResponse(BaseModel):
|
||||
ok: bool
|
||||
job: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
meta: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str) -> RebuttalResponse:
|
||||
return RebuttalResponse(ok=False, error={"code": code, "message": message, "degraded": False})
|
||||
|
||||
|
||||
def _ok(job: dict[str, Any], **meta: Any) -> RebuttalResponse:
|
||||
return RebuttalResponse(ok=True, job=job, meta=meta or None)
|
||||
|
||||
|
||||
# ---------- 请求体 ----------
|
||||
class JobCreateBody(BaseModel):
|
||||
text: str = Field(min_length=1)
|
||||
|
||||
|
||||
class PartBody(BaseModel):
|
||||
type: Literal["new_data", "reanalysis", "location", "clarification", "rebuttal"]
|
||||
content: str = ""
|
||||
location_hint: str | None = None
|
||||
|
||||
|
||||
class ResponseBody(BaseModel):
|
||||
parts: list[PartBody] = Field(default_factory=list)
|
||||
|
||||
|
||||
class StructureBody(BaseModel):
|
||||
action: Literal["merge", "split"]
|
||||
# merge: comment_ids(含当前 cid,≥2 条);split: items [{title, body}]
|
||||
comment_ids: list[str] | None = None
|
||||
items: list[dict[str, str]] | None = None
|
||||
|
||||
|
||||
class ConfirmBody(BaseModel):
|
||||
confirmed: bool
|
||||
|
||||
|
||||
# ---------- 内部工具 ----------
|
||||
async def _messages_llm(messages: list[dict[str, str]], *, temperature: float = 0.3) -> dict[str, Any]:
|
||||
"""适配 rebuttal_parse/draft 的 messages 形态 → llm.complete_json。"""
|
||||
system = next((m["content"] for m in messages if m.get("role") == "system"), "")
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
return await llm.complete_json(system, user, temperature=temperature)
|
||||
|
||||
|
||||
def _find_comment(job: dict[str, Any], cid: str) -> tuple[dict, dict] | None:
|
||||
for r in job["reviewers"]:
|
||||
for c in r["comments"]:
|
||||
if c["id"] == cid:
|
||||
return r, c
|
||||
return None
|
||||
|
||||
|
||||
async def _build_job(raw_text: str, *, kind: str, filename: str | None) -> RebuttalResponse:
|
||||
raw_text = (raw_text or "").strip()
|
||||
if not raw_text:
|
||||
return _error("VALIDATION", "审稿意见文本为空")
|
||||
if len(raw_text) > _MAX_RAW_CHARS:
|
||||
return _error("VALIDATION", f"文本超过 {_MAX_RAW_CHARS} 字符上限")
|
||||
reviewers = await parse_review_text(raw_text, _messages_llm)
|
||||
if not reviewers:
|
||||
return _error("PARSE_FAILED", "未能从文本中解析出任何审稿意见条目")
|
||||
job = {
|
||||
"id": rebuttal_store.new_id("rbt_"),
|
||||
"status": "ready",
|
||||
"source": {"kind": kind, "filename": filename, "raw_text": raw_text},
|
||||
"template": "generic",
|
||||
"reviewers": reviewers,
|
||||
"letter_md": None,
|
||||
"validation": None,
|
||||
"human_ack": False,
|
||||
}
|
||||
rebuttal_store.insert_job(job)
|
||||
n_comments = sum(len(r["comments"]) for r in reviewers)
|
||||
return _ok(job, reviewers=len(reviewers), comments=n_comments)
|
||||
|
||||
|
||||
# ---------- 端点 ----------
|
||||
@router.post("/jobs", response_model=RebuttalResponse)
|
||||
async def create_job(body: JobCreateBody) -> RebuttalResponse:
|
||||
"""文本粘贴主入口。"""
|
||||
return await _build_job(body.text, kind="text", filename=None)
|
||||
|
||||
|
||||
@router.post("/jobs/pdf", response_model=RebuttalResponse)
|
||||
async def create_job_pdf(file: UploadFile = File(...)) -> RebuttalResponse:
|
||||
"""PDF 次要入口(批注版审稿 PDF;经 vstore 8094 解析)。"""
|
||||
settings = get_settings()
|
||||
filename = file.filename or "review.pdf"
|
||||
if not filename.lower().endswith(".pdf"):
|
||||
return _error("VALIDATION", "仅支持 PDF 文件(推荐直接粘贴文本)")
|
||||
data = await file.read()
|
||||
if not data:
|
||||
return _error("VALIDATION", "上传文件为空")
|
||||
if len(data) > settings.max_pdf_bytes:
|
||||
return _error("VALIDATION", f"PDF 超过 {settings.max_pdf_bytes // (1024 * 1024)}MB 上限")
|
||||
try:
|
||||
text = await paper_parse.parse_pdf_bytes(data, filename=filename)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return _error("UPSTREAM_DOWN", f"PDF 解析失败:{exc}")
|
||||
return await _build_job(text, kind="pdf", filename=filename)
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}", response_model=RebuttalResponse)
|
||||
async def get_job(job_id: str) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
return _ok(job)
|
||||
|
||||
|
||||
@router.patch("/jobs/{job_id}/comments/{cid}/response", response_model=RebuttalResponse)
|
||||
async def patch_response(job_id: str, cid: str, body: ResponseBody) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
found = _find_comment(job, cid)
|
||||
if found is None:
|
||||
return _error("NOT_FOUND", f"comment {cid} 不存在")
|
||||
_, comment = found
|
||||
for p in body.parts:
|
||||
if p.type not in PART_TYPES:
|
||||
return _error("VALIDATION", f"未知 part.type: {p.type}")
|
||||
comment["response"] = {"parts": [p.model_dump() for p in body.parts]}
|
||||
# 改回复 → 旧 draft/validation 作废(保守:不保留过期草稿)
|
||||
comment["draft"] = None
|
||||
comment["validation"] = None
|
||||
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
|
||||
return _ok(job)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/comments/{cid}/draft", response_model=RebuttalResponse)
|
||||
async def post_draft(job_id: str, cid: str) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
found = _find_comment(job, cid)
|
||||
if found is None:
|
||||
return _error("NOT_FOUND", f"comment {cid} 不存在")
|
||||
_, comment = found
|
||||
parts = (comment.get("response") or {}).get("parts") or []
|
||||
if not parts or not any(str(p.get("content") or "").strip() for p in parts):
|
||||
# 决策 4:生成前必须填 user_response;LLM 调用次数=0
|
||||
return _error("MISSING_RESPONSE", "请先填写回复要点(user_response)再生成草稿")
|
||||
validation = validate_parts(parts)
|
||||
comment["validation"] = validation
|
||||
if not validation["passed"]:
|
||||
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
|
||||
return RebuttalResponse(
|
||||
ok=False, job=job,
|
||||
error={"code": "VALIDATION_FAILED", "message": "反幻觉校验未通过",
|
||||
"degraded": False},
|
||||
meta={"validation": validation},
|
||||
)
|
||||
comment["draft"] = await generate_draft(_messages_llm, comment)
|
||||
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
|
||||
return _ok(job, validation=validation)
|
||||
|
||||
|
||||
@router.patch("/jobs/{job_id}/comments/{cid}/structure", response_model=RebuttalResponse)
|
||||
async def patch_structure(job_id: str, cid: str, body: StructureBody) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
found = _find_comment(job, cid)
|
||||
if found is None:
|
||||
return _error("NOT_FOUND", f"comment {cid} 不存在")
|
||||
reviewer, comment = found
|
||||
comments = reviewer["comments"]
|
||||
idx = comments.index(comment)
|
||||
if body.action == "merge":
|
||||
ids = body.comment_ids or []
|
||||
if cid not in ids:
|
||||
ids = [cid] + ids
|
||||
if len(ids) < 2:
|
||||
return _error("VALIDATION", "merge 至少需要 2 条 comment")
|
||||
targets = [c for c in comments if c["id"] in ids]
|
||||
if len(targets) < 2:
|
||||
return _error("VALIDATION", "merge 目标中有效 comment 不足 2 条")
|
||||
merged_body = "\n\n".join(c["body"] for c in targets)
|
||||
comment["body"] = merged_body
|
||||
comment["title"] = f"{comment['title']} (+{len(targets) - 1} merged)"
|
||||
reviewer["comments"] = [
|
||||
c for c in comments
|
||||
if c is comment or c["id"] not in ids
|
||||
]
|
||||
else: # split
|
||||
items = body.items or []
|
||||
items = [i for i in items if str(i.get("body") or "").strip()]
|
||||
if len(items) < 2:
|
||||
return _error("VALIDATION", "split 至少需要 2 个非空子项")
|
||||
group_id = comment.get("group_id") or f"g_{cid}"
|
||||
new_comments = []
|
||||
for n, it in enumerate(items, start=1):
|
||||
c = dict(comment)
|
||||
c["id"] = rebuttal_store.new_id(f"{cid}s{n}")
|
||||
c["group_id"] = group_id
|
||||
c["title"] = str(it.get("title") or f"{comment['title']} ({n})")
|
||||
c["body"] = str(it.get("body") or "").strip()
|
||||
c["sub_items"] = []
|
||||
c["response"] = {"parts": []}
|
||||
c["draft"] = None
|
||||
c["validation"] = None
|
||||
c["human_confirmed"] = False
|
||||
new_comments.append(c)
|
||||
comments[idx:idx + 1] = new_comments
|
||||
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
|
||||
return _ok(job)
|
||||
|
||||
|
||||
@router.patch("/jobs/{job_id}/comments/{cid}/confirm", response_model=RebuttalResponse)
|
||||
async def patch_confirm(job_id: str, cid: str, body: ConfirmBody) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
found = _find_comment(job, cid)
|
||||
if found is None:
|
||||
return _error("NOT_FOUND", f"comment {cid} 不存在")
|
||||
_, comment = found
|
||||
comment["human_confirmed"] = bool(body.confirmed)
|
||||
# human_ack 由 store 重算(决策 4:全勾才置位,不信前端)
|
||||
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
|
||||
return _ok(job, human_ack=job["human_ack"])
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/letter", response_model=RebuttalResponse)
|
||||
async def post_letter(job_id: str) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
job = rebuttal_store.update_job(job_id, letter_md=assemble_letter(job)) or job
|
||||
return _ok(job)
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}/letter.md")
|
||||
async def download_letter(job_id: str) -> PlainTextResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return PlainTextResponse(
|
||||
"job not found", status_code=404, media_type="text/plain; charset=utf-8")
|
||||
if not job.get("human_ack"):
|
||||
# 信封口径与下载端点适配:GET 下载返回纯文本,业务拒绝用 400 + 明文
|
||||
return PlainTextResponse(
|
||||
"HUMAN_ACK_REQUIRED: 请逐条勾选确认后再下载 letter.md",
|
||||
status_code=400, media_type="text/plain; charset=utf-8")
|
||||
md = job.get("letter_md") or assemble_letter(job)
|
||||
return PlainTextResponse(md, media_type="text/markdown; charset=utf-8")
|
||||
|
||||
|
||||
@router.delete("/jobs/{job_id}", response_model=RebuttalResponse)
|
||||
async def delete_job(job_id: str) -> RebuttalResponse:
|
||||
if not rebuttal_store.delete_job(job_id):
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
return RebuttalResponse(ok=True, job=None, meta={"deleted": job_id})
|
||||
@@ -1,165 +0,0 @@
|
||||
"""routers/reformat.py —— 论文自动排版 P1(.tex 源码级重排版,不编译)。
|
||||
|
||||
端点:
|
||||
POST /api/review/reformat multipart 上传 .tex + target_journal →
|
||||
修改后源码 + 逐行 diff 报告(job 落盘)
|
||||
GET /api/review/reformat/{id} 取完整报告(重启后仍在)
|
||||
GET /api/review/reformat/{id}/file 下载修改后的 .tex
|
||||
|
||||
信封与其余 review 端点一致:业务错误 = HTTP 200 + {ok:false,error:{code,message,degraded:false}},
|
||||
前端经 review.js unwrap 归一为 ApiError。本次改动无上游依赖(零 LLM/零 vstore),
|
||||
degraded 恒 false。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, File, Form, UploadFile
|
||||
from fastapi.responses import FileResponse, JSONResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..config import get_settings
|
||||
from ..services.reformat import ReformatError, reformat_tex, resolve_rules
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-reformat"])
|
||||
|
||||
_JOB_ID_RE = re.compile(r"^[0-9a-f]{12}$")
|
||||
_ENGINE = "reformat-p1/1.0"
|
||||
|
||||
|
||||
class ReformatResponse(BaseModel):
|
||||
ok: bool
|
||||
job_id: str | None = None
|
||||
modified_tex: str | None = None
|
||||
changes: list[dict[str, Any]] | None = None
|
||||
warnings: list[dict[str, Any]] | None = None
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str) -> ReformatResponse:
|
||||
return ReformatResponse(
|
||||
ok=False,
|
||||
error={"code": code, "message": message, "degraded": False},
|
||||
)
|
||||
|
||||
|
||||
def _jobs_dir() -> Path:
|
||||
d = get_settings().reformat_dir_abs
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
@router.post("/reformat", response_model=ReformatResponse)
|
||||
async def reformat(
|
||||
file: UploadFile = File(..., description="LaTeX 源码(.tex 单文件)"),
|
||||
target_journal: str = Form("", description="目标期刊 id/名称,或 default=通用规则"),
|
||||
) -> ReformatResponse:
|
||||
settings = get_settings()
|
||||
|
||||
# ---- 入参校验(拍板 5:只收 .tex,docx/pdf 走 format-check 审核不重排)----
|
||||
filename = file.filename or "paper.tex"
|
||||
if not filename.lower().endswith(".tex"):
|
||||
return _error("VALIDATION", "仅支持 .tex 文件(docx/pdf 请走格式审核)")
|
||||
target = (target_journal or "").strip()
|
||||
if not target:
|
||||
return _error("VALIDATION", "必须指定目标期刊(或选择「通用规则」)")
|
||||
|
||||
content = await file.read()
|
||||
if not content:
|
||||
return _error("VALIDATION", "上传文件为空")
|
||||
if len(content) > settings.max_tex_bytes:
|
||||
return _error("VALIDATION", "源码超过 2MB 上限")
|
||||
try:
|
||||
source = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return _error("VALIDATION", "源码必须为 UTF-8 编码")
|
||||
|
||||
# ---- 规则解析(journals.json / format_cache / journal_directory 只读回链)----
|
||||
resolved = await resolve_rules(target)
|
||||
|
||||
# ---- 变换(纯规则引擎;结构畸形 → PARSER_FAILED)----
|
||||
try:
|
||||
result = reformat_tex(source, resolved["rules"])
|
||||
except ReformatError as exc:
|
||||
logger.warning("reformat: 结构守卫拒收 %s: %s", filename, exc)
|
||||
return _error(exc.code, str(exc))
|
||||
|
||||
# ---- 落盘(.tex + .json 报告;重启后 GET 仍可用)----
|
||||
job_id = uuid.uuid4().hex[:12]
|
||||
base = _jobs_dir()
|
||||
(base / f"{job_id}.tex").write_text(result["modified_tex"], encoding="utf-8")
|
||||
|
||||
meta: dict[str, Any] = {
|
||||
"journal_id": resolved["journal_id"],
|
||||
"journal_name": resolved["journal_name"],
|
||||
"rules": resolved["rules"],
|
||||
"rules_source": resolved["rules_source"],
|
||||
"rules_note": resolved["rules_note"],
|
||||
"filename": filename,
|
||||
"original_sha256": hashlib.sha256(content).hexdigest(),
|
||||
"engine": _ENGINE,
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||
}
|
||||
payload = {
|
||||
"ok": True,
|
||||
"job_id": job_id,
|
||||
"modified_tex": result["modified_tex"],
|
||||
"changes": result["changes"],
|
||||
"warnings": result["warnings"],
|
||||
"meta": meta,
|
||||
}
|
||||
(base / f"{job_id}.json").write_text(
|
||||
json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
|
||||
)
|
||||
logger.info(
|
||||
"reformat: %s → job %s(%d 处修改, %d 条提示, 规则 %s)",
|
||||
filename, job_id,
|
||||
sum(1 for c in result["changes"] if c["action"] == "modified"),
|
||||
sum(1 for c in result["changes"] if c["action"] == "warning"),
|
||||
resolved["rules_source"],
|
||||
)
|
||||
return ReformatResponse(**payload)
|
||||
|
||||
|
||||
def _load_job(job_id: str) -> dict[str, Any] | None:
|
||||
if not _JOB_ID_RE.fullmatch(job_id or ""):
|
||||
return None
|
||||
path = _jobs_dir() / f"{job_id}.json"
|
||||
try:
|
||||
payload = json.loads(path.read_text("utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
return payload if isinstance(payload, dict) and payload.get("ok") else None
|
||||
|
||||
|
||||
_NOT_FOUND = {"ok": False, "error": {"code": "NOT_FOUND",
|
||||
"message": "任务不存在或已过期,请重新上传源码",
|
||||
"degraded": False}}
|
||||
|
||||
|
||||
@router.get("/reformat/{job_id}", response_model=ReformatResponse)
|
||||
async def get_reformat_job(job_id: str) -> ReformatResponse:
|
||||
payload = _load_job(job_id)
|
||||
if payload is None:
|
||||
return _error("NOT_FOUND", "任务不存在或已过期,请重新上传源码")
|
||||
return ReformatResponse(**payload)
|
||||
|
||||
|
||||
@router.get("/reformat/{job_id}/file", response_model=None)
|
||||
async def download_reformat_file(job_id: str) -> FileResponse | JSONResponse:
|
||||
"""下载修改后的 .tex;缺件/非法 id → 200 信封(前端单一错误口径)。"""
|
||||
valid = bool(_JOB_ID_RE.fullmatch(job_id or ""))
|
||||
tex = _jobs_dir() / f"{job_id}.tex" if valid else None
|
||||
if tex is None or not tex.is_file():
|
||||
return JSONResponse(status_code=200, content=_NOT_FOUND)
|
||||
return FileResponse(tex, media_type="application/x-tex", filename=f"{job_id}.tex")
|
||||
@@ -1 +0,0 @@
|
||||
"""review 服务服务层包。"""
|
||||
@@ -1,788 +0,0 @@
|
||||
"""services/format_check.py —— 版式与规范规则引擎(⑤ 论文排版与规范)。
|
||||
|
||||
纯规则、无 LLM、无 vstore、同步执行(通常 <1s):
|
||||
- PDF → pymupdf 本地解档(AGENTS.md 约束 6 例外:vstore 只回文本,字体/边距/
|
||||
行距/页数等版式元数据拿不到;正文提取仍走 vstore,本服务不碰正文语义);
|
||||
- Word → python-docx(页边距/行距精确,页数取 docProps/app.xml 的 <Pages>);
|
||||
- 规则集 → data/journals/journals.json:
|
||||
顶层 format_rules_default(通用默认值,rules_source="default")
|
||||
每条目可选 format_rules(按期刊覆盖,必须带 rules_source: placeholder|official|default)
|
||||
⚠ 诚实原则:不知道期刊真实投稿指南 → 该条只写 {"rules_source": "placeholder"},
|
||||
所有具体数字回退 default 通用值,绝不编造期刊专属数字。
|
||||
|
||||
输出:{checks[{name,key,status,detail,suggestion}], summary{pass,warn,fail}, meta{...}}
|
||||
status ∈ pass|warn|fail。参考文献样式宽容化:仅"明确不符"(如 [J] 标引但规则要求 APA)
|
||||
才 fail,"疑似但不确定/检测不到"一律 warn,避免误伤。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import statistics
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import fitz # pymupdf(venv 已装 1.28.x)
|
||||
from docx import Document
|
||||
from docx.shared import Length
|
||||
|
||||
# review_service/app/services/ → parents[2] = review_service
|
||||
_JOURNALS_PATH = Path(__file__).resolve().parents[2] / "data" / "journals" / "journals.json"
|
||||
|
||||
_MAX_SCAN_PAGES = 80 # 版式扫描页数上限(正文/字体/边距;稿件通常远小于此)
|
||||
_MAX_REFS_SCAN = 40 # 从文末向前找参考文献标题的页数窗口
|
||||
_MAX_PARAS = 8000 # docx 段落扫描上限
|
||||
_PT_TO_MM = 25.4 / 72.0
|
||||
_EMU_TO_MM = 1.0 / 36000.0
|
||||
|
||||
_HEADING_RE = re.compile(
|
||||
r"^\s*(references|bibliography|literature cited|works cited|参考文献|文献)\s*[::]?\s*$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_FIG_RE = re.compile(r"(?im)^\s*(figure|fig\.?|图)\s*\d+")
|
||||
_TAB_RE = re.compile(r"(?im)^\s*(table|表)\s*\d+")
|
||||
_GB_TAG_RE = re.compile(r"\[[A-Z]{1,3}\d*\]") # [J] [M] [D] [C] …(GB/T 7714 标引)
|
||||
_PAREN_YEAR_RE = re.compile(r"\((?:19|20)\d{2}\)") # (2024) 作者-年份制
|
||||
_BRACKET_NUM_RE = re.compile(r"^\[\d+\]") # [1] 编号制
|
||||
_DOI_RE = re.compile(r"doi:?\.?org|doi:\s*1\.", re.IGNORECASE)
|
||||
|
||||
_STYLE_LABEL = {
|
||||
"gb7714": "GB/T 7714",
|
||||
"author_year": "作者-年份制(类 APA)",
|
||||
"numbered": "编号制(顺序编码)",
|
||||
"unknown": "无法确定",
|
||||
}
|
||||
|
||||
|
||||
class FormatCheckError(Exception):
|
||||
"""解档/规则加载失败(路由器转 PARSE_FAILED)。"""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 规则集加载
|
||||
|
||||
def load_ruleset(target_journal: str) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
"""返回 (rules, ruleset_meta)。target_journal 传 id 或 name 均可;未知 → default。"""
|
||||
try:
|
||||
data = json.loads(_JOURNALS_PATH.read_text("utf-8"))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise FormatCheckError(f"规则集加载失败:{exc}") from exc
|
||||
|
||||
default = data.get("format_rules_default") or {}
|
||||
jid = (target_journal or "").strip()
|
||||
entry = None
|
||||
if jid:
|
||||
for j in data.get("journals") or []:
|
||||
if j.get("id") == jid or j.get("name") == jid:
|
||||
entry = j
|
||||
break
|
||||
|
||||
if entry is None:
|
||||
rules = dict(default)
|
||||
meta = {
|
||||
"journal_id": None,
|
||||
"journal_name": None,
|
||||
"ruleset_source": "default",
|
||||
"rules_source": str(default.get("rules_source", "default")),
|
||||
}
|
||||
else:
|
||||
journal_rules = entry.get("format_rules") or {}
|
||||
rules = {**default, **journal_rules}
|
||||
meta = {
|
||||
"journal_id": entry.get("id"),
|
||||
"journal_name": entry.get("name"),
|
||||
"ruleset_source": "journal",
|
||||
# 占位规则(无官方核验)→ 前端黄字提示
|
||||
"rules_source": str(journal_rules.get("rules_source", "placeholder")),
|
||||
}
|
||||
rules.setdefault("rules_source", "default")
|
||||
return rules, meta
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- PDF 解档(pymupdf)
|
||||
|
||||
def _norm_font(name: str) -> str:
|
||||
n = (name or "").split("+")[-1].strip()
|
||||
return n or "unknown"
|
||||
|
||||
|
||||
def extract_profile_pdf(data: bytes) -> dict[str, Any]:
|
||||
try:
|
||||
doc = fitz.open(stream=data, filetype="pdf")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise FormatCheckError(f"PDF 解档失败:{exc}") from exc
|
||||
if getattr(doc, "needs_pass", False):
|
||||
doc.close()
|
||||
raise FormatCheckError("PDF 已加密,无法解析")
|
||||
|
||||
pages_total = doc.page_count
|
||||
scan = min(pages_total, _MAX_SCAN_PAGES)
|
||||
margin_rows: list[tuple[float, float, float, float]] = [] # t r b l (mm)
|
||||
font_w: dict[str, int] = {}
|
||||
size_w: dict[float, int] = {}
|
||||
ratios: list[float] = []
|
||||
img_ids: set[int] = set()
|
||||
fig_caps = tab_caps = 0
|
||||
|
||||
for pno in range(scan):
|
||||
page = doc[pno]
|
||||
rect = page.rect
|
||||
blocks = page.get_text("blocks")
|
||||
content = [b for b in blocks if b[4].strip()]
|
||||
if content:
|
||||
left = min(b[0] for b in content)
|
||||
top = min(b[1] for b in content)
|
||||
right = rect.width - max(b[2] for b in content)
|
||||
bottom = rect.height - max(b[3] for b in content)
|
||||
margin_rows.append(
|
||||
(
|
||||
max(0.0, top) * _PT_TO_MM,
|
||||
max(0.0, right) * _PT_TO_MM,
|
||||
max(0.0, bottom) * _PT_TO_MM,
|
||||
max(0.0, left) * _PT_TO_MM,
|
||||
)
|
||||
)
|
||||
|
||||
for img in page.get_images(full=True):
|
||||
img_ids.add(img[0])
|
||||
|
||||
text = page.get_text()
|
||||
fig_caps += len(_FIG_RE.findall(text))
|
||||
tab_caps += len(_TAB_RE.findall(text))
|
||||
|
||||
d = page.get_text("dict")
|
||||
for block in d.get("blocks", []):
|
||||
if block.get("type") != 0:
|
||||
continue
|
||||
line_meta: list[tuple[float, float]] = [] # (基线 y, 字号)
|
||||
for line in block.get("lines", []):
|
||||
spans = [s for s in line.get("spans", []) if s.get("text", "").strip()]
|
||||
if not spans:
|
||||
continue
|
||||
row_size = max(s.get("size", 0.0) for s in spans)
|
||||
if row_size <= 0:
|
||||
continue
|
||||
for s in spans:
|
||||
size = round(float(s.get("size", 0.0)), 1)
|
||||
chars = len(s.get("text", ""))
|
||||
if size > 0:
|
||||
norm = _norm_font(s.get("font", ""))
|
||||
font_w[norm] = font_w.get(norm, 0) + chars
|
||||
size_w[size] = size_w.get(size, 0) + chars
|
||||
# 行距基准用 span.origin 基线 y,不用 bbox 顶缘:bbox 顶随字形高低
|
||||
# 漂移,表格/图注等同基线并列文本会产出假性小 delta,混入中位数会把
|
||||
# 行距拉成 0.1x 的不可能值。
|
||||
origin = spans[0].get("origin")
|
||||
line_meta.append((origin[1] if origin else line["bbox"][1], row_size))
|
||||
for i in range(1, len(line_meta)):
|
||||
delta = line_meta[i][0] - line_meta[i - 1][0]
|
||||
size = line_meta[i][1]
|
||||
# 行距倍数 = 相邻行基线 y 差 / 当前行字号;
|
||||
# 过滤物理不可能值:<0.5(行重叠/同基线并列表格)、>4.0(异常结构)
|
||||
ratio = delta / size
|
||||
if 0.5 <= ratio <= 4.0:
|
||||
ratios.append(ratio)
|
||||
|
||||
refs_text, refs_found = _find_refs_pdf(doc, pages_total)
|
||||
doc.close()
|
||||
|
||||
if margin_rows:
|
||||
med = [statistics.median(col) for col in zip(*margin_rows)]
|
||||
margins = {"top": med[0], "right": med[1], "bottom": med[2], "left": med[3]}
|
||||
else:
|
||||
margins = None
|
||||
|
||||
return {
|
||||
"source_kind": "pdf",
|
||||
"pages": pages_total,
|
||||
"margins_mm": margins,
|
||||
"font_weights": font_w,
|
||||
"size_weights": size_w,
|
||||
"line_spacing_median": statistics.median(ratios) if ratios else None,
|
||||
"figures": max(len(img_ids), fig_caps),
|
||||
"tables": tab_caps,
|
||||
"refs_text": refs_text,
|
||||
"refs_found": refs_found,
|
||||
}
|
||||
|
||||
|
||||
def _find_refs_pdf(doc: "fitz.Document", pages_total: int) -> tuple[str | None, bool]:
|
||||
"""从文末向前找参考文献标题行;找到 → 标题行至文末文本;找不到 → (None, False)。"""
|
||||
buf: list[str] = []
|
||||
start = max(0, pages_total - _MAX_REFS_SCAN)
|
||||
for pno in range(pages_total - 1, start - 1, -1):
|
||||
page_text = doc[pno].get_text()
|
||||
lines = page_text.splitlines()
|
||||
for idx in range(len(lines) - 1, -1, -1):
|
||||
if _HEADING_RE.match(lines[idx].strip()):
|
||||
tail = "\n".join(lines[idx + 1:])
|
||||
joined = "\n".join([tail] + buf).strip()
|
||||
return (joined if len(joined) >= 50 else None), bool(joined)
|
||||
buf.insert(0, page_text)
|
||||
return None, False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- Word 解档(python-docx)
|
||||
|
||||
def _docx_app_pages(data: bytes) -> int | None:
|
||||
try:
|
||||
with zipfile.ZipFile(io.BytesIO(data)) as zf:
|
||||
xml = zf.read("docProps/app.xml").decode("utf-8", "ignore")
|
||||
m = re.search(r"<Pages>(\d+)</Pages>", xml)
|
||||
return int(m.group(1)) if m else None
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
|
||||
|
||||
def extract_profile_docx(data: bytes) -> dict[str, Any]:
|
||||
try:
|
||||
doc = Document(io.BytesIO(data))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise FormatCheckError(f"Word 文件解析失败:{exc}") from exc
|
||||
|
||||
side_lists: dict[str, list[float]] = {"top": [], "right": [], "bottom": [], "left": []}
|
||||
for sec in doc.sections:
|
||||
for k, v in (
|
||||
("top", sec.top_margin),
|
||||
("right", sec.right_margin),
|
||||
("bottom", sec.bottom_margin),
|
||||
("left", sec.left_margin),
|
||||
):
|
||||
if v is not None:
|
||||
side_lists[k].append(float(v) * _EMU_TO_MM)
|
||||
margins = None
|
||||
if all(side_lists.values()):
|
||||
margins = {k: statistics.median(v) for k, v in side_lists.items()}
|
||||
|
||||
try:
|
||||
normal = doc.styles["Normal"].font
|
||||
default_name = normal.name
|
||||
default_size = float(normal.size.pt) if normal.size is not None else None
|
||||
except Exception: # noqa: BLE001
|
||||
default_name, default_size = None, None
|
||||
|
||||
font_w: dict[str, int] = {}
|
||||
size_w: dict[float, int] = {}
|
||||
ratios: list[float] = []
|
||||
fig_caps = tab_caps = 0
|
||||
paras = doc.paragraphs
|
||||
scan_count = min(len(paras), _MAX_PARAS)
|
||||
|
||||
for i in range(scan_count):
|
||||
para = paras[i]
|
||||
text = para.text
|
||||
if not text.strip():
|
||||
continue
|
||||
ls = para.paragraph_format.line_spacing
|
||||
if isinstance(ls, Length):
|
||||
# 精确行距(exactly)以 EMU 存储,Length 是 int 子类——直接当倍数会得出
|
||||
# 十万级的假值;用正文默认字号换算成倍数
|
||||
if default_size:
|
||||
r = float(ls.pt) / default_size
|
||||
if 0.5 <= r <= 4.0:
|
||||
ratios.append(r)
|
||||
elif isinstance(ls, (int, float)) and not isinstance(ls, bool) and ls > 0:
|
||||
r = float(ls) # multiple 行距本身就是倍数
|
||||
if 0.5 <= r <= 4.0:
|
||||
ratios.append(r)
|
||||
if _FIG_RE.match(text):
|
||||
fig_caps += 1
|
||||
elif _TAB_RE.match(text):
|
||||
tab_caps += 1
|
||||
for run in para.runs:
|
||||
if not run.text:
|
||||
continue
|
||||
name = run.font.name or default_name or "unknown"
|
||||
pt = run.font.size.pt if run.font.size is not None else default_size
|
||||
chars = len(run.text)
|
||||
font_w[str(name)] = font_w.get(str(name), 0) + chars
|
||||
if pt:
|
||||
size_w[round(float(pt), 1)] = size_w.get(round(float(pt), 1), 0) + chars
|
||||
|
||||
# 参考文献节:标题段落之后全部文本
|
||||
refs_text, refs_found = None, False
|
||||
for i, para in enumerate(paras):
|
||||
if _HEADING_RE.match((para.text or "").strip()):
|
||||
tail = "\n".join(p.text for p in paras[i + 1:] if (p.text or "").strip())
|
||||
if len(tail) >= 50:
|
||||
refs_text, refs_found = tail, True
|
||||
break
|
||||
|
||||
return {
|
||||
"source_kind": "docx",
|
||||
"pages": _docx_app_pages(data), # Word 未记录 → None(检查项降级为 warn)
|
||||
"margins_mm": margins,
|
||||
"font_weights": font_w,
|
||||
"size_weights": size_w,
|
||||
"line_spacing_median": statistics.median(ratios) if ratios else None,
|
||||
"figures": max(len(doc.inline_shapes), fig_caps),
|
||||
"tables": max(len(doc.tables), tab_caps),
|
||||
"refs_text": refs_text,
|
||||
"refs_found": refs_found,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- LaTeX 解档(正则+状态机,零新依赖)
|
||||
#
|
||||
# 决策前提:只做"解析 + 检查 + 建议代码片段",不自动改写 LaTeX(用户可能有自定义宏,
|
||||
# 自动改写会引错;LaTeX 应由用户自己编辑)。不引入 pylatexenc/texsoup。
|
||||
|
||||
# 会相互覆盖的字体类宏包:≥2 个同时引入 → warn(times + mathptmx 类冲突)
|
||||
_TEX_FONT_PKGS = {"times", "mathptmx", "newtx", "newpx", "txfonts", "pxfonts", "mathpazo"}
|
||||
|
||||
|
||||
def _strip_tex_comments(src: str) -> str:
|
||||
r"""去行内 % 注释(r-string,避免 \% 被解释为转义序列)。"""
|
||||
return re.sub(r"(?<!\\)%.*", "", src)
|
||||
|
||||
|
||||
def _parse_geometry(src: str) -> dict[str, float] | None:
|
||||
"""\\usepackage[...]{geometry} 或 \\geometry{...} → {边: mm}(可缺边,检查端容缺)。"""
|
||||
m = re.search(r"\\usepackage\s*\[([^\]]*)\]\s*\{\s*geometry\s*\}", src)
|
||||
if not m:
|
||||
m = re.search(r"\\geometry\s*\{([^}]*)\}", src)
|
||||
if not m:
|
||||
return None
|
||||
opts = m.group(1)
|
||||
|
||||
def to_mm(val: str) -> float | None:
|
||||
mm = re.match(r"\s*([\d.]+)\s*(mm|cm|in|pt)?\s*$", val)
|
||||
if not mm:
|
||||
return None
|
||||
x = float(mm.group(1))
|
||||
unit = mm.group(2) or "pt" # geometry 裸数字按 TeX 惯例为 pt
|
||||
return {"mm": x, "cm": x * 10, "in": x * 25.4, "pt": x * 25.4 / 72.0}[unit]
|
||||
|
||||
kv = {}
|
||||
for pair in opts.split(","):
|
||||
if "=" in pair:
|
||||
k, v = pair.split("=", 1)
|
||||
kv[k.strip().lower()] = v.strip()
|
||||
out: dict[str, float] = {}
|
||||
if "margin" in kv:
|
||||
v = to_mm(kv["margin"])
|
||||
if v is not None:
|
||||
return {"top": v, "right": v, "bottom": v, "left": v}
|
||||
for k in ("top", "right", "bottom", "left"):
|
||||
if k in kv:
|
||||
v = to_mm(kv[k])
|
||||
if v is not None:
|
||||
out[k] = v
|
||||
if "inner" in kv:
|
||||
v = to_mm(kv["inner"])
|
||||
if v is not None:
|
||||
out.setdefault("left", v)
|
||||
if "outer" in kv:
|
||||
v = to_mm(kv["outer"])
|
||||
if v is not None:
|
||||
out.setdefault("right", v)
|
||||
return out or None
|
||||
|
||||
|
||||
def extract_tex_profile(data: bytes) -> dict[str, Any]:
|
||||
"""正则 + 轻量状态机解析 .tex 源文件 → 与 DocProfile 同构(+tex 扩展字段)。"""
|
||||
try:
|
||||
text = data.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
text = data.decode("latin-1", errors="replace")
|
||||
src = _strip_tex_comments(text)
|
||||
|
||||
# \documentclass[10pt,a4paper]{article}
|
||||
m_cls = re.search(r"\\documentclass\s*(?:\[[^\]]*\])?\s*\{\s*([\w*]+)\s*\}", src)
|
||||
cls = m_cls.group(1) if m_cls else None
|
||||
m_opt = re.search(r"\\documentclass\s*\[([^\]]*)\]", src)
|
||||
cls_opts = m_opt.group(1) if m_opt else ""
|
||||
m_pt = re.search(r"(\d+)\s*pt", cls_opts)
|
||||
# 标准类未写选项时 LaTeX 默认 10pt;无 documentclass → None(检查项降级 warn)
|
||||
fontsize = int(m_pt.group(1)) if m_pt else (10 if cls else None)
|
||||
paper = "a4paper" if "a4paper" in cls_opts else ("letterpaper" if "letterpaper" in cls_opts else None)
|
||||
|
||||
# \usepackage 列表(支持 {a,b} 逗号多包)
|
||||
packages: list[str] = []
|
||||
for m in re.finditer(r"\\usepackage\s*(?:\[[^\]]*\])?\s*\{\s*([^}]+?)\s*\}", src):
|
||||
packages += [p.strip().lower() for p in m.group(1).split(",") if p.strip()]
|
||||
|
||||
# 行距:\linespread{...} 或 \setstretch{...}(值直接按"行距倍数"口径评估——
|
||||
# 按步骤4 规格:低于 min_line_spacing 即 fail;LaTeX 实际倍数还含 baselinestretch
|
||||
# 的字号比,本引擎取显式设置值做保守判定)
|
||||
line_spread: float | None = None
|
||||
m = re.search(r"\\linespread\s*\{\s*([\d.]+)\s*\}", src)
|
||||
if not m:
|
||||
m = re.search(r"\\setstretch\s*\{\s*([\d.]+)\s*\}", src)
|
||||
if m:
|
||||
line_spread = float(m.group(1))
|
||||
|
||||
# 页边距(geometry)
|
||||
margins = _parse_geometry(src)
|
||||
|
||||
# 计数
|
||||
figures = len(re.findall(r"\\begin\s*\{\s*figure\*?\s*\}", src))
|
||||
tables = len(re.findall(r"\\begin\s*\{\s*table\*?\s*\}", src))
|
||||
cites = len(re.findall(r"\\cite\w*\s*\{", src))
|
||||
|
||||
# 参考文献:\bibliography{...}(BibTeX)/ \printbibliography(biblatex)/ thebibliography
|
||||
has_bibtex = bool(re.search(r"\\bibliography\s*\{", src))
|
||||
has_printbib = bool(re.search(r"\\printbibliography", src))
|
||||
has_thebib = bool(re.search(r"\\begin\s*\{\s*thebibliography\s*\}", src))
|
||||
has_bibstyle = bool(re.search(r"\\bibliographystyle\s*\{", src)) # BibTeX 必需;biblatex 不需要
|
||||
has_bib = has_bibtex or has_printbib or has_thebib
|
||||
refs_text: str | None = None
|
||||
m_bib = re.search(r"\\begin\s*\{\s*thebibliography\s*\}[\s\S]*?\\end\s*\{\s*thebibliography\s*\}", src)
|
||||
if m_bib:
|
||||
refs_text = m_bib.group(0)
|
||||
|
||||
# 宏表(\newcommand{\xxx} / \renewcommand)——用于解析准确性(识别自定义宏)
|
||||
macros = re.findall(r"\\(?:new|re)command\s*\{\s*\\(\w+)\s*\}", src)
|
||||
|
||||
# 匿名评审命令探测(规则集 anonymous:true 时检查)
|
||||
m_anon = re.search(r"\\(?:anonymi[sz]e|anon\b|hideauthor|ifdoubleblind)", src, re.IGNORECASE)
|
||||
|
||||
return {
|
||||
"source_kind": "tex",
|
||||
"pages": None, # LaTeX 源文件页数需编译后才有 → 检查项降级 warn(宽容化)
|
||||
"margins_mm": margins,
|
||||
"font_weights": {},
|
||||
"size_weights": {float(fontsize): 100} if fontsize else {},
|
||||
"line_spacing_median": line_spread,
|
||||
"figures": figures,
|
||||
"tables": tables,
|
||||
"refs_text": refs_text,
|
||||
"refs_found": has_bib,
|
||||
"tex": {
|
||||
"class": cls,
|
||||
"fontsize_pt": fontsize,
|
||||
"paper": paper,
|
||||
"packages": packages,
|
||||
"cites": cites,
|
||||
"macros": macros,
|
||||
"anon_cmd": m_anon.group(0) if m_anon else None,
|
||||
"bib_bibtex": has_bibtex,
|
||||
"bib_style": has_bibstyle,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 参考文献样式检测
|
||||
|
||||
def detect_refs_style(text: str | None) -> tuple[str, str]:
|
||||
"""宽容化检测:返回 (detected, sample_line)。含糊 → unknown(后续只 warn)。"""
|
||||
if not text:
|
||||
return "unknown", ""
|
||||
lines = [l.strip() for l in text.splitlines() if l.strip()][:40]
|
||||
if not lines:
|
||||
return "unknown", ""
|
||||
n = len(lines)
|
||||
gb = sum(1 for l in lines if _GB_TAG_RE.search(l))
|
||||
year = sum(1 for l in lines if _PAREN_YEAR_RE.search(l))
|
||||
num = sum(1 for l in lines if _BRACKET_NUM_RE.match(l))
|
||||
doi = sum(1 for l in lines if _DOI_RE.search(l))
|
||||
sample = lines[0][:80]
|
||||
|
||||
if gb / n >= 0.4:
|
||||
return "gb7714", sample
|
||||
if num / n >= 0.4 and year / n < 0.3:
|
||||
return "numbered", sample
|
||||
if year / n >= 0.3 or doi / n >= 0.3:
|
||||
return "author_year", sample
|
||||
return "unknown", sample
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 检查项构建
|
||||
|
||||
def _chk(key: str, name: str, status: str, detail: str, suggestion: str = "") -> dict[str, str]:
|
||||
return {"key": key, "name": name, "status": status, "detail": detail, "suggestion": suggestion}
|
||||
|
||||
|
||||
def _check_pages(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
limit = int(rules.get("max_pages", 10))
|
||||
pages = profile.get("pages")
|
||||
kind = profile.get("source_kind") or "doc"
|
||||
if pages is None:
|
||||
# 宽容化:Word/LaTeX 页数缺失绝不 fail
|
||||
if kind == "tex":
|
||||
return _chk(
|
||||
"pages", "页数", "warn",
|
||||
f"LaTeX 源无法直接统计页数(规则上限 {limit} 页),需编译后核对",
|
||||
"编译后确认(如 xelatex main.tex)总页数 ≤ 上限",
|
||||
)
|
||||
return _chk(
|
||||
"pages", "页数", "warn",
|
||||
"Word 未记录页数,请自行核对",
|
||||
f"在 Word 中按 Ctrl+End 查看总页数,或导出 PDF 后重查(规则上限 {limit} 页)",
|
||||
)
|
||||
if pages <= limit:
|
||||
return _chk("pages", "页数", "pass", f"共 {pages} 页,规则上限 {limit} 页")
|
||||
return _chk(
|
||||
"pages", "页数", "fail", f"共 {pages} 页,超出规则上限 {limit} 页",
|
||||
f"压缩至 ≤{limit} 页,或将方法/图表移入补充材料",
|
||||
)
|
||||
|
||||
|
||||
def _check_fonts(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
min_font = float(rules.get("min_font_pt", 9))
|
||||
# LaTeX:字号来自 \documentclass 选项(10/11/12pt,粒度粗)
|
||||
if profile.get("source_kind") == "tex":
|
||||
tex = profile.get("tex") or {}
|
||||
cls = tex.get("class") or "documentclass"
|
||||
fs = tex.get("fontsize_pt")
|
||||
pkgs = tex.get("packages") or []
|
||||
font_pkgs = sorted(p for p in pkgs if p in _TEX_FONT_PKGS or p in ("helvet", "courier", "charter", "fourier"))
|
||||
detail = (
|
||||
f"\\documentclass 选项字号 {fs}pt(类 {cls});"
|
||||
f"字体宏包:{'、'.join(font_pkgs) if font_pkgs else '未引入'}"
|
||||
+ (f";纸张 {tex['paper']}" if tex.get("paper") else "")
|
||||
)
|
||||
if fs is None:
|
||||
return _chk("fonts", "字体字号", "warn", detail + "\n建议:\n \\documentclass[11pt]{article}", "补充 \\documentclass 声明")
|
||||
if fs < min_font:
|
||||
# LaTeX 粒度粗 → 按步骤4 规格降为 warn(非 fail)
|
||||
return _chk(
|
||||
"fonts", "字体字号", "warn", detail + f"\n建议:\n \\documentclass[11pt]{{{cls}}}",
|
||||
f"正文字号低于下限 {min_font:.0f}pt,请改用 11pt/12pt 选项",
|
||||
)
|
||||
return _chk("fonts", "字体字号", "pass", detail)
|
||||
|
||||
size_w: dict[float, int] = profile.get("size_weights") or {}
|
||||
font_w: dict[str, int] = profile.get("font_weights") or {}
|
||||
if not size_w:
|
||||
return _chk("fonts", "字体字号", "warn", "未能提取字体信息", "确认文档含可选中文本(扫描件无法检查字体)")
|
||||
min_font = float(rules.get("min_font_pt", 9))
|
||||
max_fonts = int(rules.get("max_fonts", 6))
|
||||
total = sum(size_w.values())
|
||||
body_pt = max(size_w, key=size_w.get)
|
||||
body_ratio = int(size_w[body_pt] / total * 100)
|
||||
distinct = len(font_w)
|
||||
top_fonts = sorted(font_w.items(), key=lambda kv: -kv[1])[:3]
|
||||
top_text = "、".join(f"{k}({v}字)" for k, v in top_fonts)
|
||||
detail = f"主要字号 {body_pt}pt(约 {body_ratio}%),共 {distinct} 种字体;高频字体:{top_text}"
|
||||
if body_pt < min_font:
|
||||
return _chk(
|
||||
"fonts", "字体字号", "fail", detail,
|
||||
f"将正文字号提高至 ≥{min_font}pt(当前主字号 {body_pt}pt)",
|
||||
)
|
||||
if distinct > max_fonts:
|
||||
return _chk(
|
||||
"fonts", "字体字号", "warn", detail,
|
||||
f"字体种类偏多({distinct} > {max_fonts}),建议全文统一为 1–2 种正文字体",
|
||||
)
|
||||
return _chk("fonts", "字体字号", "pass", detail)
|
||||
|
||||
|
||||
def _check_margins(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
limit = float(rules.get("min_margin_mm", 15))
|
||||
margins = profile.get("margins_mm")
|
||||
# LaTeX:页边距来自 geometry;低于下限按步骤4 规格给 warn(非 fail),附建议片段
|
||||
if profile.get("source_kind") == "tex":
|
||||
if not margins:
|
||||
return _chk(
|
||||
"margins", "页边距", "warn",
|
||||
"未通过 geometry 设置页边距(按类默认评估)"
|
||||
+ f"\n建议:\n \\usepackage[margin=25mm]{{geometry}}",
|
||||
"在导言区显式声明页边距",
|
||||
)
|
||||
lo = min(margins.values())
|
||||
detail = f"geometry 页边距最小 {lo:.0f}mm(" + "、".join(f"{k}{v:.0f}" for k, v in margins.items()) + f"),规则下限 {limit:.0f}mm"
|
||||
if lo < limit:
|
||||
return _chk(
|
||||
"margins", "页边距", "warn",
|
||||
detail + f"\n建议:\n \\usepackage[margin={limit:.0f}mm]{{geometry}}",
|
||||
f"调大页边距至 ≥{limit:.0f}mm",
|
||||
)
|
||||
return _chk("margins", "页边距", "pass", detail)
|
||||
if not margins:
|
||||
return _chk("margins", "页边距", "warn", "未能提取页边距信息", "检查文档是否含可定位文本")
|
||||
vals = [margins.get(k) for k in ("top", "right", "bottom", "left")]
|
||||
vals = [v for v in vals if v is not None]
|
||||
if not vals:
|
||||
return _chk("margins", "页边距", "warn", "未能提取页边距信息", "检查文档是否含可定位文本")
|
||||
t, r = margins.get("top"), margins.get("right")
|
||||
b, l = margins.get("bottom"), margins.get("left")
|
||||
lo = min(vals)
|
||||
sides = "、".join(
|
||||
f"{name}{v:.0f}" for name, v in (("上", t), ("右", r), ("下", b), ("左", l)) if v is not None
|
||||
)
|
||||
detail = f"四边最小 {lo:.0f}mm({sides}),规则下限 {limit:.0f}mm"
|
||||
if lo >= limit:
|
||||
return _chk("margins", "页边距", "pass", detail)
|
||||
if lo >= limit * 0.8:
|
||||
return _chk("margins", "页边距", "warn", detail, f"将页边距调大至 ≥{limit:.0f}mm")
|
||||
return _chk("margins", "页边距", "fail", detail, f"页边距过小,请调至 ≥{limit:.0f}mm(版式设置)")
|
||||
|
||||
|
||||
def _check_spacing(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
lo = float(rules.get("min_line_spacing", 1.15))
|
||||
ratio = profile.get("line_spacing_median")
|
||||
# LaTeX:\linespread/\setstretch 显式设置值;低于下限按步骤4 规格直接 fail
|
||||
if profile.get("source_kind") == "tex":
|
||||
if ratio is None:
|
||||
return _chk(
|
||||
"spacing", "行距", "pass",
|
||||
"未显式设置 \\linespread/\\setstretch,LaTeX 类默认约 1.2 倍(如 10pt→12pt baselineskip)",
|
||||
)
|
||||
detail = f"\\linespread/\\setstretch = {ratio:.2f},规则下限 {lo:.2f}"
|
||||
if ratio < lo:
|
||||
return _chk(
|
||||
"spacing", "行距", "fail",
|
||||
detail + f"\n建议:\n \\linespread{{{lo:.2f}}} % 或 \\setstretch{{{lo:.2f}}}(导言区)",
|
||||
"在导言区调大行距设置",
|
||||
)
|
||||
return _chk("spacing", "行距", "pass", detail)
|
||||
if ratio is None:
|
||||
return _chk("spacing", "行距", "warn", "未能提取行距信息", "确认段落使用固定/多倍行距设置")
|
||||
lo = float(rules.get("min_line_spacing", 1.15))
|
||||
detail = f"正文中位行距约 {ratio:.2f} 倍,规则下限 {lo:.2f} 倍"
|
||||
if ratio >= lo:
|
||||
return _chk("spacing", "行距", "pass", detail)
|
||||
if ratio >= lo * 0.9:
|
||||
return _chk("spacing", "行距", "warn", detail, f"段落行距设为 ≥{lo:.2f} 倍(如 1.5 倍行距)")
|
||||
return _chk("spacing", "行距", "fail", detail, f"行距过密,请设为 ≥{lo:.2f} 倍行距")
|
||||
|
||||
|
||||
def _check_figures(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
figures = int(profile.get("figures") or 0)
|
||||
tables = int(profile.get("tables") or 0)
|
||||
min_f = int(rules.get("min_figures", 1))
|
||||
detail = f"检测到图 {figures} 张、表 {tables} 个(图注/表注正则 + 内嵌对象启发式)"
|
||||
if figures >= min_f:
|
||||
return _chk("figures", "图表数量", "pass", detail)
|
||||
# 启发式检测不可靠 → 低于阈值只 warn 不 fail
|
||||
return _chk(
|
||||
"figures", "图表数量", "warn", detail,
|
||||
f"常规研究论文建议 ≥{min_f} 图;若为综述/短文可忽略(本项为启发式检测)",
|
||||
)
|
||||
|
||||
|
||||
# 仅"明确不符"才 fail 的 (expected, detected) 组合(宽容化:其余含糊组合一律 warn)
|
||||
_REF_FAIL_PAIRS = {
|
||||
("apa", "gb7714"), # 找到 [J] 标引但规则要求 APA
|
||||
("gb7714", "author_year"), # 无 [J] 标引的作者-年份制,但规则要求 GB/T 7714
|
||||
("apa", "numbered"), # 编号制引文但规则要求 APA
|
||||
}
|
||||
|
||||
|
||||
def _check_refs(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
expected = str(rules.get("refs_style", "apa")).lower()
|
||||
exp_label = _STYLE_LABEL.get(expected, expected)
|
||||
if not profile.get("refs_found"):
|
||||
return _chk(
|
||||
"refs", "参考文献格式", "warn",
|
||||
"未找到参考文献节(References/参考文献 / \\bibliography / thebibliography)",
|
||||
"确认存在参考文献声明后重查",
|
||||
)
|
||||
if profile.get("source_kind") == "tex":
|
||||
tex = profile.get("tex") or {}
|
||||
# BibTeX:\bibliography 存在但缺 \bibliographystyle → warn(样式引擎未知)
|
||||
if tex.get("bib_bibtex") and not tex.get("bib_style"):
|
||||
return _chk(
|
||||
"refs", "参考文献格式", "warn",
|
||||
f"规则要求 {exp_label};检测到 \\bibliography 但缺少 \\bibliographystyle(BibTeX 无法确定引用样式)"
|
||||
+ "\n建议:\n \\bibliographystyle{plain} % 按期刊/模板指定实际样式",
|
||||
"在 \\bibliography 前补 \\bibliographystyle",
|
||||
)
|
||||
if not profile.get("refs_text"):
|
||||
# 条目在 .bbl/编译产物,源文件无文本 → 宽容化只 warn
|
||||
return _chk(
|
||||
"refs", "参考文献格式", "warn",
|
||||
f"规则要求 {exp_label};检测到参考文献声明(条目由编译生成,源文件无条目文本)",
|
||||
"编译后核对参考文献条目样式",
|
||||
)
|
||||
detected, sample = detect_refs_style(profile.get("refs_text"))
|
||||
det_label = _STYLE_LABEL[detected]
|
||||
detail = f"规则要求 {exp_label};检测到 {det_label};样例:{sample}"
|
||||
# 宽容化:仅"明确不符"(_REF_FAIL_PAIRS)才 fail;含糊/不确定一律 warn
|
||||
if (expected, detected) in _REF_FAIL_PAIRS:
|
||||
return _chk("refs", "参考文献格式", "fail", detail, f"按 {exp_label} 调整全部文献条目(可用文献管理工具一键转换)")
|
||||
if detected == "unknown":
|
||||
return _chk("refs", "参考文献格式", "warn", detail, f"请人工核对是否符合 {exp_label}")
|
||||
if expected == "gb7714" and detected == "numbered":
|
||||
return _chk("refs", "参考文献格式", "warn", detail, "GB/T 7714 顺序编码制也用编号,需人工确认条目含 [J] 等类型标引")
|
||||
if expected == "author-year-journal" and detected in ("gb7714", "numbered"):
|
||||
return _chk("refs", "参考文献格式", "warn", detail, f"请对照期刊投稿指南核对是否符合 {exp_label}")
|
||||
return _chk("refs", "参考文献格式", "pass", detail)
|
||||
|
||||
|
||||
def _check_font_packages(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
"""LaTeX 特有:多个相互覆盖的字体宏包(times + mathptmx 类)→ warn。"""
|
||||
pkgs = set((profile.get("tex") or {}).get("packages") or [])
|
||||
clash = sorted(pkgs & _TEX_FONT_PKGS)
|
||||
if len(clash) >= 2:
|
||||
detail = (
|
||||
f"检测到多个相互覆盖的字体宏包:{'、'.join(clash)}(后加载者覆盖前者,易产生冲突)"
|
||||
+ "\n建议:\n %只保留一套字体方案,例如:\n \\usepackage{newtx} % newtxtext + newtxmath 一体"
|
||||
)
|
||||
return _chk("fontpkgs", "字体包兼容", "warn", detail, "移除重复/冲突的字体宏包,只保留一套")
|
||||
return _chk(
|
||||
"fontpkgs", "字体包兼容", "pass",
|
||||
"字体宏包:" + ("、".join(clash) if clash else "未引入冲突类字体包"),
|
||||
)
|
||||
|
||||
|
||||
def _check_anonymous(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
"""LaTeX 特有且规则集 anonymous:true 时才出:源文件需含匿名处理命令。"""
|
||||
tex = profile.get("tex") or {}
|
||||
cmd = tex.get("anon_cmd")
|
||||
if cmd:
|
||||
return _chk("anonymous", "匿名评审", "pass", f"检测到匿名处理命令 {cmd}")
|
||||
return _chk(
|
||||
"anonymous", "匿名评审", "warn",
|
||||
"规则要求匿名评审,源文件未检测到匿名处理命令"
|
||||
+ "\n建议:\n %隐去作者/致谢/自引,或定义匿名宏:\n \\newcommand{\\anon}[1]{[ANON]}",
|
||||
"按期刊要求对作者信息做匿名处理",
|
||||
)
|
||||
|
||||
|
||||
def build_checks(profile: dict[str, Any], rules: dict[str, Any]) -> list[dict[str, str]]:
|
||||
checks = [
|
||||
_check_pages(profile, rules),
|
||||
_check_fonts(profile, rules),
|
||||
_check_margins(profile, rules),
|
||||
_check_spacing(profile, rules),
|
||||
]
|
||||
if profile.get("source_kind") == "tex":
|
||||
checks.append(_check_font_packages(profile, rules))
|
||||
checks.append(_check_figures(profile, rules))
|
||||
checks.append(_check_refs(profile, rules))
|
||||
if profile.get("source_kind") == "tex" and rules.get("anonymous"):
|
||||
checks.append(_check_anonymous(profile, rules))
|
||||
return checks
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 入口
|
||||
|
||||
def check_document(data: bytes, kind: str, target_journal: str) -> dict[str, Any]:
|
||||
"""kind ∈ pdf|docx|tex。返回 {checks, summary, meta}(filename 由路由器补入 meta)。"""
|
||||
if kind == "pdf":
|
||||
profile = extract_profile_pdf(data)
|
||||
elif kind == "docx":
|
||||
profile = extract_profile_docx(data)
|
||||
elif kind == "tex":
|
||||
profile = extract_tex_profile(data)
|
||||
else:
|
||||
raise FormatCheckError(f"不支持的文件类型:{kind}")
|
||||
|
||||
rules, ruleset_meta = load_ruleset(target_journal)
|
||||
checks = build_checks(profile, rules)
|
||||
summary = {"pass": 0, "warn": 0, "fail": 0}
|
||||
for c in checks:
|
||||
summary[c["status"]] = summary.get(c["status"], 0) + 1
|
||||
|
||||
detected, _ = detect_refs_style(profile.get("refs_text")) if profile.get("refs_found") else ("unknown", "")
|
||||
meta = {
|
||||
**ruleset_meta,
|
||||
"file_type": kind,
|
||||
"pages": profile.get("pages"),
|
||||
"refs_style_expected": str(rules.get("refs_style", "apa")).lower(),
|
||||
"refs_style_detected": detected,
|
||||
}
|
||||
if kind == "tex":
|
||||
tex = profile.get("tex") or {}
|
||||
meta["tex"] = {
|
||||
"class": tex.get("class"),
|
||||
"fontsize_pt": tex.get("fontsize_pt"),
|
||||
"paper": tex.get("paper"),
|
||||
"packages": tex.get("packages"),
|
||||
"cites": tex.get("cites"),
|
||||
"macros": (tex.get("macros") or [])[:10],
|
||||
}
|
||||
return {"checks": checks, "summary": summary, "meta": meta}
|
||||
@@ -1,232 +0,0 @@
|
||||
"""services/format_extractor.py —— 投稿指南 → 格式规则 JSON(LLM 抽取,第一批)。
|
||||
|
||||
- 调 services/llm.py 的 complete_json(DeepSeek json_object + 修复重试,零新依赖)
|
||||
- Prompt = 「格式规则抽取器」模板:目标字段与 journals.json format_rules_default
|
||||
同构(max_pages / min_font_pt / min_margin_mm / min_line_spacing / min_figures /
|
||||
max_fonts / refs_style / anonymous),强制每字段带 source_quotes + confidence
|
||||
- 诚实原则:正文没写 → null,绝不编造(与 format_check 诚实原则一致)
|
||||
- 解析/校验失败 → {"ok": False, "error": ...}
|
||||
|
||||
返回:
|
||||
成功 {"ok": True, "rules": {...}, "source_quotes": {...},
|
||||
"confidence": {...}, "notes": str, "raw": LLM原始JSON, "error": None}
|
||||
失败 {"ok": False, "error": "...", 其余键为 None}
|
||||
source_url 由调用方(scripts/fetch_format_rules.py)在落盘时补入结果文件。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from .llm import LLMError, complete_json
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 「格式规则抽取器」模板(system prompt)
|
||||
_SYSTEM = """你是「格式规则抽取器」:从期刊官方投稿指南的 Markdown 正文中抽取格式规则,供投稿排版检查引擎使用。
|
||||
|
||||
# 目标字段(rules 对象,严格用以下字段名)
|
||||
- max_pages:正文页数上限(整数;原文无明确上限 → null)
|
||||
- min_font_pt:最小字号(数字,单位 pt;原文若用 cm/mm/inch 给出则换算为 pt)
|
||||
- min_margin_mm:最小页边距(数字,单位 mm;原文若用 inch 给出则 ×25.4 换算)
|
||||
- min_line_spacing:最小行距倍数(数字;如 "double-spaced" → 2.0,"1.5 spacing" → 1.5)
|
||||
- min_figures:研究论文最少图数(整数;无明确要求 → null)
|
||||
- max_fonts:全文最多字体种类(整数;无明确要求 → null)
|
||||
- refs_style:参考文献样式,枚举 apa | gb7714 | author_year | numbered | vancouver | mla | chicago | other(无法判断 → null)
|
||||
- anonymous:是否要求匿名/双盲评审(true / false / null)
|
||||
|
||||
# 凭证(强制)
|
||||
- 每个非 null 字段必须给 source_quotes[字段名]:从正文【逐字】摘录的 1-3 句原文,
|
||||
不改写、不翻译、不丢掉含关键数字的句子。
|
||||
- source_quotes["_general"]:整体最有凭证价值的 1-3 条原文摘录。
|
||||
- confidence:{"overall": 0-1, "per_field": {"字段名": 0-1}}。
|
||||
直接原文命中 ≥0.9;单位换算 0.6-0.9;靠上下文推断 <0.6。
|
||||
|
||||
# 诚实原则
|
||||
- 正文没写到的字段一律 null,绝不用常识或训练记忆补值。
|
||||
- 单位换算允许做,但该字段 confidence 不得超过 0.9。
|
||||
- 换算过程、拿不准的点、页面疑似不完整,写进 notes。
|
||||
|
||||
# 输出
|
||||
只输出一个 JSON 对象,不要代码块、不要解释文字:
|
||||
{"rules": {"max_pages": null, "min_font_pt": null, "min_margin_mm": null,
|
||||
"min_line_spacing": null, "min_figures": null, "max_fonts": null,
|
||||
"refs_style": null, "anonymous": null},
|
||||
"source_quotes": {"字段名": ["原文1", "..."], "_general": ["原文1", "..."]},
|
||||
"confidence": {"overall": 0.0, "per_field": {"字段名": 0.0}},
|
||||
"notes": "一句话说明抽取依据与不确定点"}"""
|
||||
|
||||
_USER_TMPL = """期刊:{journal_name}
|
||||
|
||||
=== 官方投稿指南正文(Markdown,已截断) ===
|
||||
{markdown}"""
|
||||
|
||||
_RULE_INT_KEYS = ("max_pages", "min_figures", "max_fonts")
|
||||
_REFS_ALLOWED = {"apa", "gb7714", "author_year", "numbered", "vancouver",
|
||||
"mla", "chicago", "other"}
|
||||
_REFS_ALIAS = {
|
||||
"apa-7": "apa", "apa7": "apa", "apa 7th": "apa", "apa_7th": "apa",
|
||||
"harvard": "author_year", "author-year": "author_year", "authoryear": "author_year",
|
||||
"gb/t 7714": "gb7714", "gb/t-7714": "gb7714", "gb7714-2015": "gb7714",
|
||||
"gb/t7714": "gb7714", "chicago author-date": "chicago",
|
||||
}
|
||||
|
||||
|
||||
# 字符串值带显式单位时的目标单位换算表(LLM 若回传原文单位的兜底;无单位=数值即目标单位)
|
||||
_UNIT_TO_MM = {"mm": 1.0, "cm": 10.0, "in": 25.4, "inch": 25.4, "inches": 25.4, "pt": 25.4 / 72}
|
||||
_UNIT_TO_PT = {"pt": 1.0, "mm": 72 / 25.4, "cm": 72 / 2.54, "in": 72.0, "inch": 72.0, "inches": 72.0}
|
||||
|
||||
|
||||
def _to_int(v: Any) -> int | None:
|
||||
if isinstance(v, bool) or v is None:
|
||||
return None
|
||||
if isinstance(v, int):
|
||||
return v
|
||||
if isinstance(v, float):
|
||||
return int(round(v))
|
||||
if isinstance(v, str):
|
||||
m = re.search(r"-?\d+", v)
|
||||
if m:
|
||||
return int(m.group(0))
|
||||
return None
|
||||
|
||||
|
||||
def _to_float(v: Any, units: dict[str, float] | None = None) -> float | None:
|
||||
if isinstance(v, bool) or v is None:
|
||||
return None
|
||||
if isinstance(v, (int, float)):
|
||||
return float(v)
|
||||
if isinstance(v, str):
|
||||
m = re.search(r"(-?\d+(?:\.\d+)?)\s*([a-zA-Z]+)?", v)
|
||||
if not m:
|
||||
return None
|
||||
x = float(m.group(1))
|
||||
unit = (m.group(2) or "").lower()
|
||||
if units and unit:
|
||||
factor = units.get(unit)
|
||||
if factor is None:
|
||||
return None # 未知单位不猜
|
||||
x *= factor
|
||||
return x
|
||||
return None
|
||||
|
||||
|
||||
def _to_bool(v: Any) -> bool | None:
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
if isinstance(v, (int, float)) and not isinstance(v, bool):
|
||||
if v in (0, 1):
|
||||
return bool(v)
|
||||
if isinstance(v, str):
|
||||
low = v.strip().lower()
|
||||
if low in ("true", "yes", "required"):
|
||||
return True
|
||||
if low in ("false", "no", "not required"):
|
||||
return False
|
||||
return None
|
||||
|
||||
|
||||
def _norm_refs(v: Any) -> str | None:
|
||||
if not isinstance(v, str) or not v.strip():
|
||||
return None
|
||||
low = v.strip().lower()
|
||||
low = _REFS_ALIAS.get(low, low)
|
||||
return low if low in _REFS_ALLOWED else "other"
|
||||
|
||||
|
||||
def _norm_conf(v: Any) -> float | None:
|
||||
if isinstance(v, bool) or not isinstance(v, (int, float)):
|
||||
return None
|
||||
return max(0.0, min(1.0, float(v)))
|
||||
|
||||
|
||||
def _normalize(raw: dict[str, Any]) -> dict[str, Any]:
|
||||
"""白名单归一:类型错的值置 null(宁缺勿错,raw 全量保留在结果里)。"""
|
||||
src = raw.get("rules")
|
||||
src = src if isinstance(src, dict) else {}
|
||||
rules: dict[str, Any] = {}
|
||||
for k in _RULE_INT_KEYS:
|
||||
rules[k] = _to_int(src.get(k))
|
||||
rules["min_font_pt"] = _to_float(src.get("min_font_pt"), _UNIT_TO_PT)
|
||||
rules["min_margin_mm"] = _to_float(src.get("min_margin_mm"), _UNIT_TO_MM)
|
||||
rules["min_line_spacing"] = _to_float(src.get("min_line_spacing"))
|
||||
rules["refs_style"] = _norm_refs(src.get("refs_style"))
|
||||
rules["anonymous"] = _to_bool(src.get("anonymous"))
|
||||
|
||||
sq_src = raw.get("source_quotes")
|
||||
sq_src = sq_src if isinstance(sq_src, dict) else {}
|
||||
quotes: dict[str, list[str]] = {}
|
||||
for key, val in sq_src.items():
|
||||
if key != "_general" and key not in rules:
|
||||
continue
|
||||
items = val if isinstance(val, list) else [val]
|
||||
cleaned = [str(x).strip() for x in items if isinstance(x, (str, int, float))
|
||||
and str(x).strip()]
|
||||
if cleaned:
|
||||
quotes[key] = cleaned
|
||||
|
||||
conf_src = raw.get("confidence")
|
||||
conf_src = conf_src if isinstance(conf_src, dict) else {}
|
||||
pf_src = conf_src.get("per_field")
|
||||
pf_src = pf_src if isinstance(pf_src, dict) else {}
|
||||
per_field: dict[str, float] = {}
|
||||
for k, v in pf_src.items():
|
||||
c = _norm_conf(v)
|
||||
if c is not None and (k in rules or k == "_general"):
|
||||
per_field[k] = c
|
||||
overall = _norm_conf(conf_src.get("overall"))
|
||||
if overall is None and per_field:
|
||||
overall = sum(per_field.values()) / len(per_field)
|
||||
confidence = {"overall": overall, "per_field": per_field}
|
||||
|
||||
notes = raw.get("notes")
|
||||
return {
|
||||
"rules": rules,
|
||||
"source_quotes": quotes,
|
||||
"confidence": confidence,
|
||||
"notes": str(notes) if notes else "",
|
||||
}
|
||||
|
||||
|
||||
def _validation_error(norm: dict[str, Any]) -> str | None:
|
||||
"""强制凭证:任一非 null 字段缺 source_quotes → 拒收(ok=False)。"""
|
||||
missing = [k for k, v in norm["rules"].items()
|
||||
if v is not None and k not in norm["source_quotes"]]
|
||||
if missing:
|
||||
return f"LLM 输出缺少非 null 字段的 source_quotes:{', '.join(missing)}"
|
||||
if not norm["source_quotes"].get("_general"):
|
||||
return "LLM 输出缺少 source_quotes._general 整体引证"
|
||||
if any(v is not None for v in norm["rules"].values()):
|
||||
return None
|
||||
return "LLM 未抽出任何非 null 字段(rules 全空)"
|
||||
|
||||
|
||||
async def extract_format_rules(markdown: str, journal_name: str) -> dict:
|
||||
"""markdown(≤30000 字符)+ 期刊名 → 抽取结果(schema 见模块 docstring)。"""
|
||||
if not markdown or not markdown.strip():
|
||||
return {"ok": False, "error": "markdown 为空", "rules": None,
|
||||
"source_quotes": None, "confidence": None, "notes": None, "raw": None}
|
||||
user = _USER_TMPL.format(journal_name=journal_name, markdown=markdown)
|
||||
try:
|
||||
raw = await complete_json(_SYSTEM, user, temperature=0.1, max_tokens=8000)
|
||||
except LLMError as exc:
|
||||
return {"ok": False, "error": f"LLM 调用失败:{exc}", "rules": None,
|
||||
"source_quotes": None, "confidence": None, "notes": None, "raw": None}
|
||||
if not isinstance(raw, dict):
|
||||
return {"ok": False, "error": f"LLM 输出非 JSON 对象:{type(raw).__name__}",
|
||||
"rules": None, "source_quotes": None, "confidence": None,
|
||||
"notes": None, "raw": None}
|
||||
|
||||
norm = _normalize(raw)
|
||||
err = _validation_error(norm)
|
||||
if err:
|
||||
logger.warning("[extract] %s 校验失败:%s", journal_name, err)
|
||||
return {"ok": False, "error": err, "rules": None,
|
||||
"source_quotes": norm["source_quotes"], "confidence": norm["confidence"],
|
||||
"notes": norm["notes"], "raw": raw}
|
||||
filled = sum(1 for v in norm["rules"].values() if v is not None)
|
||||
logger.info("[extract] %s → 8 字段中 %d 个非 null", journal_name, filled)
|
||||
return {"ok": True, "error": None, "rules": norm["rules"],
|
||||
"source_quotes": norm["source_quotes"],
|
||||
"confidence": norm["confidence"], "notes": norm["notes"], "raw": raw}
|
||||
@@ -1,356 +0,0 @@
|
||||
"""services/format_fetcher.py —— 投稿指南页面抓取(格式规则自动获取·第一批)。
|
||||
|
||||
约束(用户拍板 2026-10-10):
|
||||
- 零新依赖:httpx + beautifulsoup4(venv 已装 4.15.0,用户明确指定使用;
|
||||
lxml 6.1.2 在 requirements.lock 内,作 bs4 解析器)
|
||||
- 同域限速 ≥ 2s(robots.txt 与正文请求共用计时器)
|
||||
- 抓正文前先 GET /robots.txt,命中 Disallow → ok=False 跳过
|
||||
- UA: "PlantAI-RAG-Bot/1.0 (+contact: admin@example.com)"
|
||||
- 只提取正文(去导航/页脚/脚本),截断至 30000 字符
|
||||
|
||||
返回固定四键:{"ok": bool, "markdown": str, "error": str | None, "fetched_at": str}
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup, NavigableString
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
USER_AGENT = "PlantAI-RAG-Bot/1.0 (+contact: admin@example.com)"
|
||||
TIMEOUT = 30.0
|
||||
MAX_CHARS = 30_000
|
||||
MIN_INTERVAL_PER_HOST = 2.0 # 同域请求间隔 ≥ 2s(robots 与正文共用)
|
||||
_MIN_TEXT_CHARS = 200 # 正文短于此 → 疑似 JS 渲染/反爬页
|
||||
_HEADERS = {
|
||||
"User-Agent": USER_AGENT,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
}
|
||||
|
||||
# 同域限速:host → 上次请求完成时刻(monotonic)
|
||||
_last_hit: dict[str, float] = {}
|
||||
_throttle_lock = asyncio.Lock()
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def _fail(error: str) -> dict:
|
||||
return {"ok": False, "markdown": "", "error": error, "fetched_at": _now_iso()}
|
||||
|
||||
|
||||
async def _throttle(host: str) -> None:
|
||||
"""同域串行 + ≥2s 间隔(全局锁:本批抓取量小,简单可靠优先)。"""
|
||||
async with _throttle_lock:
|
||||
now = time.monotonic()
|
||||
wait = _last_hit.get(host, now) + MIN_INTERVAL_PER_HOST - now
|
||||
if wait > 0:
|
||||
await asyncio.sleep(wait)
|
||||
_last_hit[host] = time.monotonic()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- robots.txt
|
||||
|
||||
def _parse_robots(text: str) -> list[tuple[list[str], list[tuple[str, str]]]]:
|
||||
"""→ [(agents, [(kind, pattern)...]), ...](连续 UA 行同组;规则行后出现 UA=新组)。"""
|
||||
groups: list[tuple[list[str], list[tuple[str, str]]]] = []
|
||||
agents: list[str] = []
|
||||
rules: list[tuple[str, str]] = []
|
||||
for raw in text.splitlines():
|
||||
line = raw.split("#", 1)[0].strip()
|
||||
if not line or ":" not in line:
|
||||
continue
|
||||
key, _, value = line.partition(":")
|
||||
key, value = key.strip().lower(), value.strip()
|
||||
if key == "user-agent":
|
||||
if agents and rules: # 上一组已出规则 → 当前 UA 行开新组
|
||||
groups.append((agents, rules))
|
||||
agents, rules = [], []
|
||||
agents.append(value)
|
||||
elif key in ("allow", "disallow"):
|
||||
if not agents: # 无前置 UA 的规则按 * 组处理
|
||||
agents = ["*"]
|
||||
rules.append((key, value))
|
||||
if agents:
|
||||
groups.append((agents, rules))
|
||||
return groups
|
||||
|
||||
|
||||
def _robots_path_allowed(groups, path: str) -> tuple[bool, str]:
|
||||
"""最长匹配决胜(同长 Allow 优先;`$` 结尾=精确匹配;空 Disallow 跳过)。"""
|
||||
|
||||
def matched_by(pat: str) -> bool:
|
||||
if pat.endswith("$"):
|
||||
return path == pat[:-1]
|
||||
return path.startswith(pat)
|
||||
|
||||
token = "plantai-rag-bot"
|
||||
matched = [rules for agents, rules in groups
|
||||
if any(token in a.lower() for a in agents)]
|
||||
if not matched:
|
||||
matched = [rules for agents, rules in groups
|
||||
if any(a.lower() == "*" for a in agents)]
|
||||
if not matched:
|
||||
return True, "robots.txt 无匹配组(本 bot 不受限)"
|
||||
|
||||
best_len, best_allow, best_pat = -1, False, ""
|
||||
for rules in matched:
|
||||
for kind, pat in rules:
|
||||
if not pat: # "Disallow:" 空值 = 放行一切(该组无需处理)
|
||||
continue
|
||||
if not matched_by(pat):
|
||||
continue
|
||||
length = len(pat)
|
||||
allow = kind == "allow"
|
||||
if length > best_len or (length == best_len and allow and not best_allow):
|
||||
best_len, best_allow, best_pat = length, allow, pat
|
||||
if best_len < 0:
|
||||
return True, "robots.txt 允许(无路径匹配)"
|
||||
if best_allow:
|
||||
return True, f"robots.txt Allow 命中({best_pat})"
|
||||
return False, f"robots.txt Disallow 命中({best_pat})"
|
||||
|
||||
|
||||
async def _robots_allows(client: httpx.AsyncClient, url: str) -> tuple[bool, str]:
|
||||
p = urlparse(url)
|
||||
robots_url = f"{p.scheme}://{p.netloc}/robots.txt"
|
||||
await _throttle(p.netloc.lower())
|
||||
try:
|
||||
resp = await client.get(robots_url, headers=_HEADERS)
|
||||
except httpx.HTTPError as exc:
|
||||
return True, f"robots.txt 获取失败({type(exc).__name__}),按允许处理"
|
||||
if resp.status_code != 200:
|
||||
return True, f"robots.txt HTTP {resp.status_code},按允许处理"
|
||||
groups = _parse_robots(resp.text)
|
||||
return _robots_path_allowed(groups, p.path or "/")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 正文提取
|
||||
|
||||
_STRIP_TAGS = ("script", "style", "noscript", "iframe", "form", "nav",
|
||||
"footer", "aside", "button", "select", "textarea", "input", "svg")
|
||||
_STRIP_ROLES = ("navigation", "contentinfo", "complementary", "search")
|
||||
# 正文容器候选(按优先级;首个文本 ≥500 字符者胜出)
|
||||
_CONTENT_SELECTORS = (
|
||||
"main", "article", '[role="main"]', "#content", "#main", "#main-content",
|
||||
".article-body", "#article-body", ".article__body", ".hub-article-body",
|
||||
".page-content", ".main-content", ".content-area",
|
||||
)
|
||||
_CHALLENGE_MARKERS = ("just a moment", "attention required", "captcha",
|
||||
"access denied", "ddos protection", "verify you are human")
|
||||
|
||||
|
||||
def _decode_html(resp: httpx.Response) -> str:
|
||||
"""header charset → <meta charset> → utf-8(errors=replace),防 OUP/Wiley 无 header 乱码。"""
|
||||
enc = None
|
||||
m = re.search(r"charset=([\w.-]+)", resp.headers.get("content-type", ""), re.I)
|
||||
if m:
|
||||
enc = m.group(1)
|
||||
if not enc:
|
||||
m = re.search(rb"""charset\s*=\s*["']?([\w.-]+)""", resp.content[:4096], re.I)
|
||||
if m:
|
||||
enc = m.group(1).decode("ascii", "ignore")
|
||||
try:
|
||||
return resp.content.decode(enc or "utf-8", errors="replace")
|
||||
except LookupError:
|
||||
return resp.content.decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
def _squeeze(s: str) -> str:
|
||||
"""行内空白压缩 + 去空行(保留单换行)。"""
|
||||
lines = [re.sub(r"[ \t\r\f\v]+", " ", line).strip() for line in s.split("\n")]
|
||||
return "\n".join(line for line in lines if line)
|
||||
|
||||
|
||||
def _inline(el) -> str:
|
||||
"""行内渲染:a→md链接 / strong→** / em→* / code→` / br→换行;跳过列表与图片。"""
|
||||
parts: list[str] = []
|
||||
for child in el.children:
|
||||
name = getattr(child, "name", None)
|
||||
if name is None: # NavigableString
|
||||
parts.append(str(child))
|
||||
continue
|
||||
if name in ("ul", "ol", "img", "picture", "figure"):
|
||||
continue
|
||||
if name == "a":
|
||||
href = (child.get("href") or "").strip()
|
||||
txt = _inline(child).strip()
|
||||
if not txt:
|
||||
continue
|
||||
parts.append(f"[{txt}]({href})" if href.startswith(("http://", "https://")) else txt)
|
||||
elif name in ("strong", "b"):
|
||||
parts.append(f"**{_inline(child)}**")
|
||||
elif name in ("em", "i"):
|
||||
parts.append(f"*{_inline(child)}*")
|
||||
elif name == "code":
|
||||
parts.append(f"`{_inline(child).strip()}`")
|
||||
elif name == "br":
|
||||
parts.append("\n")
|
||||
else:
|
||||
parts.append(_inline(child))
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def _list_md(el, ordered: bool) -> str:
|
||||
lines: list[str] = []
|
||||
idx = 1
|
||||
for li in el.find_all("li", recursive=False):
|
||||
# li 文本不含嵌套列表(嵌套列表随后缩进输出)
|
||||
buf = []
|
||||
for child in li.children:
|
||||
if getattr(child, "name", None) in ("ul", "ol"):
|
||||
continue
|
||||
buf.append(_inline(child) if getattr(child, "name", None) else str(child))
|
||||
txt = " ".join("".join(buf).split())
|
||||
if txt:
|
||||
lines.append(f"{idx}. {txt}" if ordered else f"- {txt}")
|
||||
idx += 1
|
||||
for nested in li.find_all(("ul", "ol"), recursive=False):
|
||||
sub = _list_md(nested, nested.name == "ol")
|
||||
if sub:
|
||||
lines.extend(" " + line for line in sub.split("\n"))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _table_md(table) -> str | None:
|
||||
rows: list[list[str]] = []
|
||||
for tr in table.find_all("tr"):
|
||||
cells = [" ".join(_squeeze(_inline(c)).split())
|
||||
for c in tr.find_all(("td", "th"))]
|
||||
if cells:
|
||||
rows.append(cells)
|
||||
if not rows:
|
||||
return None
|
||||
out = ["| " + " | ".join(rows[0]) + " |"]
|
||||
if table.find("th"): # 首行含表头 → 补分隔行(合法 GFM 表)
|
||||
out.append("| " + " | ".join(["---"] * len(rows[0])) + " |")
|
||||
out += ["| " + " | ".join(r) + " |" for r in rows[1:]]
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def _blocks(el, out: list[str]) -> None:
|
||||
for child in el.children:
|
||||
name = getattr(child, "name", None)
|
||||
if name is None: # 裸文本
|
||||
txt = _squeeze(str(child))
|
||||
if txt:
|
||||
out.append(txt)
|
||||
continue
|
||||
if name in _STRIP_TAGS:
|
||||
continue
|
||||
if name in ("h1", "h2", "h3", "h4", "h5", "h6"):
|
||||
txt = " ".join(_squeeze(_inline(child)).split())
|
||||
if txt:
|
||||
out.append("#" * int(name[1]) + " " + txt)
|
||||
elif name == "p":
|
||||
txt = _squeeze(_inline(child))
|
||||
if txt:
|
||||
out.append(txt)
|
||||
elif name in ("ul", "ol"):
|
||||
md = _list_md(child, name == "ol")
|
||||
if md:
|
||||
out.append(md)
|
||||
elif name == "table":
|
||||
md = _table_md(child)
|
||||
if md:
|
||||
out.append(md)
|
||||
elif name == "blockquote":
|
||||
txt = _squeeze(_inline(child))
|
||||
if txt:
|
||||
out.append("\n".join("> " + line for line in txt.split("\n")))
|
||||
elif name == "pre":
|
||||
code = child.get_text().strip("\n")
|
||||
if code.strip():
|
||||
out.append("```\n" + code + "\n```")
|
||||
elif name == "hr":
|
||||
out.append("---")
|
||||
else: # div/section/article/main/figure/dl/details/… 容器递归
|
||||
_blocks(child, out)
|
||||
|
||||
|
||||
def _html_to_markdown(html: str) -> tuple[str, str | None]:
|
||||
"""→ (markdown, error)。选正文容器 → 去壳 → DOM 递归转 Markdown。"""
|
||||
soup = BeautifulSoup(html, "lxml") # lxml 在 lock 内;缺时 bs4 自动回退 html.parser
|
||||
|
||||
title = (soup.title.get_text() if soup.title else "") or ""
|
||||
for tag in soup.find_all(_STRIP_TAGS):
|
||||
tag.decompose()
|
||||
for tag in soup.find_all(attrs={"role": _STRIP_ROLES}):
|
||||
tag.decompose()
|
||||
|
||||
container = None
|
||||
fallback_hits = []
|
||||
for sel in _CONTENT_SELECTORS:
|
||||
for node in soup.select(sel):
|
||||
text_len = len(node.get_text(" ", strip=True))
|
||||
fallback_hits.append((text_len, node))
|
||||
if text_len >= 500:
|
||||
container = node
|
||||
break
|
||||
if container is not None:
|
||||
break
|
||||
if container is None:
|
||||
if fallback_hits:
|
||||
container = max(fallback_hits, key=lambda x: x[0])[1]
|
||||
else:
|
||||
container = soup.body or soup
|
||||
|
||||
blocks: list[str] = []
|
||||
_blocks(container, blocks)
|
||||
md = "\n\n".join(blocks)
|
||||
|
||||
probe = (title + " " + md[:500]).lower()
|
||||
marker = next((m for m in _CHALLENGE_MARKERS if m in probe), None)
|
||||
if marker:
|
||||
return "", f"疑似反爬/挑战页(命中 {marker!r},title={title[:80]!r})"
|
||||
if len(md) < _MIN_TEXT_CHARS:
|
||||
return "", f"正文提取过短({len(md)} < {_MIN_TEXT_CHARS} 字符),疑似 JS 渲染或空壳页"
|
||||
if len(md) > MAX_CHARS:
|
||||
md = md[:MAX_CHARS - 1] + "…"
|
||||
return md, None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 入口
|
||||
|
||||
async def fetch_submission_guidelines(url: str) -> dict:
|
||||
"""抓投稿指南页面 → 正文 Markdown。
|
||||
|
||||
流程:robots.txt 检查(≥2s 限速)→ 禁止即跳过 → GET 正文(≥2s 限速)
|
||||
→ 状态/类型校验 → 正文提取 → 截断 30000 字符。
|
||||
"""
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in ("http", "https") or not parsed.netloc:
|
||||
return _fail(f"非法 URL:{url!r}")
|
||||
host = parsed.netloc.lower()
|
||||
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=TIMEOUT) as client:
|
||||
allowed, robots_reason = await _robots_allows(client, url)
|
||||
if not allowed:
|
||||
logger.info("[fetch] robots 禁止,跳过 %s(%s)", url, robots_reason)
|
||||
return _fail(robots_reason)
|
||||
|
||||
await _throttle(host)
|
||||
try:
|
||||
resp = await client.get(url, headers=_HEADERS)
|
||||
except httpx.HTTPError as exc:
|
||||
return _fail(f"请求失败({type(exc).__name__}):{exc}")
|
||||
if resp.status_code != 200:
|
||||
return _fail(f"HTTP {resp.status_code}")
|
||||
ctype = resp.headers.get("content-type", "")
|
||||
if "html" not in ctype and "xhtml" not in ctype:
|
||||
return _fail(f"非 HTML 响应:{ctype or '无 Content-Type'}")
|
||||
|
||||
markdown, err = _html_to_markdown(_decode_html(resp))
|
||||
if err:
|
||||
return _fail(f"{err}(robots: {robots_reason})")
|
||||
logger.info("[fetch] %s → %d 字符(robots: %s)", url, len(markdown), robots_reason)
|
||||
return {"ok": True, "markdown": markdown, "error": None, "fetched_at": _now_iso()}
|
||||
@@ -1,223 +0,0 @@
|
||||
"""services/journal_kb.py —— 期刊 scope_desc → vstore 入库 + 向量匹配(只读消费 8094)。
|
||||
|
||||
链路(全部复用 vstore 既有 API,零改动):
|
||||
1. ensure_scope_docs:各期刊 scope_desc 以 .txt 入库(category=journal_scope,
|
||||
name=期刊名),document_id 回写 journals.vstore_doc_id(幂等:已有 id 不重灌)。
|
||||
2. match:GET /api/vstore/search-chunks?q=<摘要截断>&doc_ids=<id 白名单>&top_k=N
|
||||
→ 按 doc_id 映射回期刊,取该期刊最高 chunk 分数为 similarity。
|
||||
3. 自愈:白名单命中 0 条 → GET /documents/{id} 探活 → 失效的重灌(vstore 去重
|
||||
语义下同内容重灌会换 doc_id,需回写)→ 重试一次 → 仍 0 视为语义低相似。
|
||||
|
||||
错误语义:连通性/上游异常抛 MatchError(路由降级为全量候选);
|
||||
返回 ([], False) 表示检索成功但无有效匹配(路由同样降级)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from ..config import get_settings
|
||||
from ..db import set_vstore_doc_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SCOPE_CATEGORY = "journal_scope"
|
||||
|
||||
|
||||
class MatchError(Exception):
|
||||
"""vstore 向量匹配失败(不可达/响应异常)。"""
|
||||
|
||||
|
||||
def _timeout() -> httpx.Timeout:
|
||||
t = get_settings().vstore_match_timeout
|
||||
return httpx.Timeout(connect=5.0, read=t, write=t, pool=5.0)
|
||||
|
||||
|
||||
def _err_body(resp: httpx.Response) -> str:
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError:
|
||||
return (resp.text or "").strip()[:500] or "(空响应体)"
|
||||
if isinstance(payload, dict):
|
||||
err = payload.get("error")
|
||||
if isinstance(err, dict) and err.get("message"):
|
||||
return str(err["message"])
|
||||
if isinstance(err, str):
|
||||
return err
|
||||
return str(payload)[:500]
|
||||
|
||||
|
||||
async def _ingest_scope(client: httpx.AsyncClient, base: str, journal: dict[str, Any]) -> str:
|
||||
"""单本期刊 scope_desc 入库,返回 document_id。"""
|
||||
content = str(journal.get("scope_desc") or "").strip()
|
||||
if not content:
|
||||
raise MatchError(f"期刊 scope_desc 为空:{journal.get('id')}")
|
||||
jid = str(journal.get("id") or "")
|
||||
try:
|
||||
resp = await client.post(
|
||||
f"{base}/api/vstore/ingest",
|
||||
files={"file": (f"{jid}.txt", content.encode("utf-8"), "text/plain")},
|
||||
data={
|
||||
"name": str(journal.get("name") or jid),
|
||||
"category": SCOPE_CATEGORY,
|
||||
"description": "review_service 期刊范围种子(scope_desc 向量匹配)",
|
||||
},
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
raise MatchError(f"vstore 不可达({base}):{exc}") from exc
|
||||
if resp.status_code >= 400:
|
||||
raise MatchError(f"vstore ingest 失败(HTTP {resp.status_code}):{_err_body(resp)}")
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError as exc:
|
||||
raise MatchError(f"vstore ingest 响应非 JSON:{exc}") from exc
|
||||
if not isinstance(payload, dict) or not payload.get("ok"):
|
||||
raise MatchError(
|
||||
str((payload.get("error") or {}).get("message") if isinstance(payload, dict) else "") or "vstore ingest 失败"
|
||||
)
|
||||
doc_id = str((payload.get("item") or {}).get("document_id") or "")
|
||||
if not doc_id:
|
||||
raise MatchError("vstore ingest 响应缺少 document_id")
|
||||
return doc_id
|
||||
|
||||
|
||||
async def ensure_scope_docs(journals: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""补齐缺失的 scope 入库映射(就地回写 vstore_doc_id 并持久到 DB)。"""
|
||||
settings = get_settings()
|
||||
missing = [j for j in journals if not j.get("vstore_doc_id")]
|
||||
if not missing:
|
||||
return journals
|
||||
async with httpx.AsyncClient(timeout=_timeout()) as client:
|
||||
for j in missing:
|
||||
try:
|
||||
doc_id = await _ingest_scope(client, settings.vstore_base, j)
|
||||
except MatchError as exc:
|
||||
# 单本失败不阻断其余(该期刊将不进白名单,由路由降级兜底)
|
||||
logger.warning("journal_kb: scope 入库失败(%s):%s", j.get("id"), exc)
|
||||
continue
|
||||
j["vstore_doc_id"] = doc_id
|
||||
await set_vstore_doc_id(str(j["id"]), doc_id)
|
||||
return journals
|
||||
|
||||
|
||||
async def _search(
|
||||
client: httpx.AsyncClient, base: str, query: str, doc_ids: list[str]
|
||||
) -> list[dict[str, Any]]:
|
||||
"""chunk 级白名单检索。连通/上游错误抛 MatchError。"""
|
||||
if not doc_ids:
|
||||
# doc_ids 为空时 vstore 侧等于不过滤 → 会命中用户私有库,必须拦死
|
||||
raise MatchError("白名单为空,拒绝全库检索")
|
||||
try:
|
||||
resp = await client.get(
|
||||
f"{base}/api/vstore/search-chunks",
|
||||
params={
|
||||
"q": query,
|
||||
"top_k": get_settings().recommend_scope_top_k,
|
||||
"doc_ids": ",".join(doc_ids),
|
||||
},
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
raise MatchError(f"vstore search-chunks 不可达:{exc}") from exc
|
||||
if resp.status_code >= 400:
|
||||
raise MatchError(f"vstore search-chunks 失败(HTTP {resp.status_code}):{_err_body(resp)}")
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError as exc:
|
||||
raise MatchError(f"vstore search-chunks 响应非 JSON:{exc}") from exc
|
||||
if not isinstance(payload, dict) or not payload.get("ok"):
|
||||
raise MatchError(
|
||||
str((payload.get("error") or {}).get("message") if isinstance(payload, dict) else "") or "vstore search-chunks 失败"
|
||||
)
|
||||
items = payload.get("items")
|
||||
return items if isinstance(items, list) else []
|
||||
|
||||
|
||||
async def _doc_alive(client: httpx.AsyncClient, base: str, doc_id: str) -> bool:
|
||||
try:
|
||||
resp = await client.get(f"{base}/api/vstore/documents/{doc_id}")
|
||||
except httpx.HTTPError:
|
||||
return False
|
||||
if resp.status_code != 200:
|
||||
return False
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError:
|
||||
return False
|
||||
return bool(isinstance(payload, dict) and payload.get("ok"))
|
||||
|
||||
|
||||
async def _refresh_dead_docs(
|
||||
client: httpx.AsyncClient, base: str, journals: list[dict[str, Any]]
|
||||
) -> None:
|
||||
"""白名单 0 命中时探活:失效文档重灌并回写新 id。"""
|
||||
for j in journals:
|
||||
doc_id = str(j.get("vstore_doc_id") or "")
|
||||
if not doc_id or await _doc_alive(client, base, doc_id):
|
||||
continue
|
||||
try:
|
||||
new_id = await _ingest_scope(client, base, j)
|
||||
except MatchError as exc:
|
||||
logger.warning("journal_kb: scope 重灌失败(%s):%s", j.get("id"), exc)
|
||||
continue
|
||||
j["vstore_doc_id"] = new_id
|
||||
await set_vstore_doc_id(str(j["id"]), new_id)
|
||||
|
||||
|
||||
async def match(
|
||||
abstract: str, journals: list[dict[str, Any]]
|
||||
) -> tuple[list[dict[str, Any]], bool]:
|
||||
"""摘要 vs scope_desc 向量匹配。
|
||||
|
||||
返回 (candidates, vector_ok);candidates 元素:
|
||||
{"journal": <journals 行>, "similarity": float}
|
||||
按 similarity 降序。MatchError → 路由降级;([], False) → 语义低相似。
|
||||
"""
|
||||
settings = get_settings()
|
||||
query = abstract.strip()[: settings.recommend_query_chars]
|
||||
if not query:
|
||||
return [], False
|
||||
|
||||
await ensure_scope_docs(journals)
|
||||
whitelisted = [j for j in journals if j.get("vstore_doc_id")]
|
||||
if not whitelisted:
|
||||
raise MatchError("无任何期刊 scope 入库成功(vstore_doc_id 为空)")
|
||||
|
||||
doc_map = {str(j["vstore_doc_id"]): j for j in whitelisted}
|
||||
timeout = _timeout()
|
||||
async with httpx.AsyncClient(timeout=timeout) as client:
|
||||
items = await _search(client, settings.vstore_base, query, list(doc_map))
|
||||
if not items:
|
||||
await _refresh_dead_docs(client, settings.vstore_base, whitelisted)
|
||||
doc_map = {str(j["vstore_doc_id"]): j for j in whitelisted if j.get("vstore_doc_id")}
|
||||
if not doc_map:
|
||||
return [], False
|
||||
items = await _search(client, settings.vstore_base, query, list(doc_map))
|
||||
|
||||
# 聚合:每本期刊取最高 chunk 分
|
||||
best: dict[str, float] = {}
|
||||
for item in items:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
j = doc_map.get(str(item.get("doc_id") or ""))
|
||||
if j is None:
|
||||
continue
|
||||
try:
|
||||
score = float(item.get("score"))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
jid = str(j["id"])
|
||||
if jid not in best or score > best[jid]:
|
||||
best[jid] = score
|
||||
if not best:
|
||||
return [], False
|
||||
|
||||
by_id = {str(j["id"]): j for j in journals}
|
||||
candidates = [
|
||||
{"journal": by_id[jid], "similarity": score}
|
||||
for jid, score in best.items()
|
||||
if jid in by_id
|
||||
]
|
||||
candidates.sort(key=lambda c: c["similarity"], reverse=True)
|
||||
return candidates, bool(candidates)
|
||||
@@ -1,210 +0,0 @@
|
||||
"""services/journal_rank.py —— LLM 重排 + 幻觉防护 + 三档归类。
|
||||
|
||||
职责:
|
||||
1. rerank:把带语义分的候选期刊(来自 journal_kb)+ 用户偏好交给 LLM,
|
||||
要求按 id 白名单返回 {rankings:[{id, tier, match_reason, risk_note}]}。
|
||||
2. validate:硬校验——白名单外 id 丢弃(幻觉拦截);tier 非法/LLM 缺漏的
|
||||
期刊按规则补位(绝不因 LLM 输出丢失候选期刊);期刊名称/指标一律不采信
|
||||
LLM,最终展示全部用 DB 字段(防止改名型幻觉)。
|
||||
3. fallback_rankings:LLM 不可用时的确定性规则分档(按 impact_factor)。
|
||||
|
||||
三档语义:challenge 挑战型(高影响/慢审稿)、fit 适配型(均衡)、
|
||||
safe 稳妥型(快速/门槛低)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from ..config import get_settings
|
||||
from . import llm
|
||||
|
||||
TIERS = ("challenge", "fit", "safe")
|
||||
TIER_LABELS = {
|
||||
"challenge": "挑战型(高影响因子/慢审稿)",
|
||||
"fit": "适配型(中等影响/周期适中)",
|
||||
"safe": "稳妥型(快速发表/门槛较低)",
|
||||
}
|
||||
|
||||
# 规则分档阈值(近似值,可后续配置化)
|
||||
_CHALLENGE_MIN_IF = 9.0
|
||||
_SAFE_MAX_IF = 4.0
|
||||
|
||||
# tier 同义词归一(LLM 偶发中文标签)
|
||||
_TIER_ALIASES = {
|
||||
"挑战": "challenge",
|
||||
"挑战型": "challenge",
|
||||
"高风险": "challenge",
|
||||
"适配": "fit",
|
||||
"适配型": "fit",
|
||||
"稳妥": "safe",
|
||||
"稳妥型": "safe",
|
||||
"低风险": "safe",
|
||||
}
|
||||
|
||||
SYSTEM_PROMPT = """你是学术投稿推荐专家(植物基因组学方向)。根据给定候选期刊、论文摘要与用户偏好,为**每一个**候选期刊输出推荐分档与理由。
|
||||
|
||||
硬规则(违反即视为无效输出):
|
||||
1. id 白名单:只能使用输入 candidates 中出现的 id;绝对禁止添加、替换或臆造列表之外的任何期刊。
|
||||
2. 每个输入 id 恰好输出一次,不得遗漏、不得重复;期刊名称等元数据由系统填充,你只输出 id。
|
||||
3. tier 只能取 challenge | fit | safe:
|
||||
- challenge(挑战型):高影响因子、审稿较慢或竞争激烈的期刊
|
||||
- fit(适配型):影响力与发表周期均衡、与论文契合的期刊
|
||||
- safe(稳妥型):审稿较快、门槛较低、命中概率较高的期刊
|
||||
4. match_reason:中文,1-2 句,结合摘要的研究主题与该刊 scope_desc 及指标(如有 similarity 一并参考),说明为何匹配/不匹配。
|
||||
5. risk_note:中文,1-2 句,指出投稿风险(超出用户偏好、IF 门槛偏高、范围匹配偏弱、审稿周期长等);无明显风险写「无」。
|
||||
6. 用户 preferences 为软约束:不满足偏好的期刊仍必须输出,并在 risk_note 中明确说明。
|
||||
7. 不确定时宁可保守陈述,禁止编造统计数据(录用率、审稿天数等)。
|
||||
|
||||
输出:仅一个 JSON 对象,格式:
|
||||
{"rankings":[{"id":"...","tier":"challenge","match_reason":"...","risk_note":"..."}]}
|
||||
不要输出任何 JSON 之外的内容。"""
|
||||
|
||||
|
||||
class RankError(Exception):
|
||||
"""LLM 输出形态异常(无法按 JSON 解析/结构非法)。"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Ranking:
|
||||
journal_id: str
|
||||
tier: str
|
||||
match_reason: str
|
||||
risk_note: str
|
||||
|
||||
|
||||
def rule_tier(journal: dict[str, Any]) -> str:
|
||||
"""确定性分档规则(LLM 缺漏/不可用时兜底)。"""
|
||||
inf = journal.get("impact_factor")
|
||||
weeks = journal.get("review_time_weeks")
|
||||
if isinstance(inf, (int, float)):
|
||||
if inf >= _CHALLENGE_MIN_IF:
|
||||
return "challenge"
|
||||
if inf <= _SAFE_MAX_IF:
|
||||
return "safe"
|
||||
elif isinstance(weeks, (int, float)) and weeks <= 4:
|
||||
return "safe"
|
||||
return "fit"
|
||||
|
||||
|
||||
def build_user_prompt(
|
||||
abstract: str,
|
||||
candidates: list[dict[str, Any]],
|
||||
preferences: dict[str, Any],
|
||||
) -> str:
|
||||
"""候选 + 偏好 + 摘要 → 用户消息(JSON)。"""
|
||||
payload = {
|
||||
"abstract": abstract.strip()[:4000],
|
||||
"preferences": preferences or {},
|
||||
"candidates": [
|
||||
{
|
||||
"id": str(c["journal"]["id"]),
|
||||
"name": str(c["journal"].get("name") or ""),
|
||||
"impact_factor": c["journal"].get("impact_factor"),
|
||||
"review_time_weeks": c["journal"].get("review_time_weeks"),
|
||||
"acceptance_rate": c["journal"].get("acceptance_rate"),
|
||||
"similarity": c.get("similarity"),
|
||||
"scope_desc": str(c["journal"].get("scope_desc") or "")[:400],
|
||||
}
|
||||
for c in candidates
|
||||
],
|
||||
}
|
||||
return json.dumps(payload, ensure_ascii=False)
|
||||
|
||||
|
||||
def _normalize_tier(raw: Any) -> str | None:
|
||||
tier = str(raw or "").strip().lower()
|
||||
tier = _TIER_ALIASES.get(str(raw or "").strip(), tier)
|
||||
return tier if tier in TIERS else None
|
||||
|
||||
|
||||
def validate_rankings(
|
||||
payload: Any, candidates: list[dict[str, Any]]
|
||||
) -> tuple[list[Ranking], int, int]:
|
||||
"""校验 LLM 输出。
|
||||
|
||||
返回 (rankings, dropped_hallucinated, repaired):
|
||||
- 白名单外 id / 重复 id → 丢弃并计数(repaired 中重复不计);
|
||||
- tier 非法、LLM 缺漏的 id → 按规则补位(repaired 计数)。
|
||||
"""
|
||||
if not isinstance(payload, dict) or not isinstance(payload.get("rankings"), list):
|
||||
raise RankError("LLM 输出缺少 rankings 数组")
|
||||
|
||||
by_id = {str(c["journal"]["id"]): c["journal"] for c in candidates}
|
||||
input_ids = list(by_id)
|
||||
seen: set[str] = set()
|
||||
rankings: list[Ranking] = []
|
||||
dropped = 0
|
||||
repaired = 0
|
||||
|
||||
for entry in payload["rankings"]:
|
||||
if not isinstance(entry, dict):
|
||||
dropped += 1
|
||||
continue
|
||||
jid = str(entry.get("id") or "").strip()
|
||||
if jid in seen:
|
||||
continue
|
||||
if jid not in by_id:
|
||||
dropped += 1 # 幻觉期刊:绝不进输出
|
||||
continue
|
||||
seen.add(jid)
|
||||
tier = _normalize_tier(entry.get("tier"))
|
||||
if tier is None:
|
||||
tier = rule_tier(by_id[jid])
|
||||
repaired += 1
|
||||
reason = str(entry.get("match_reason") or "").strip() or "基于期刊范围与指标的匹配"
|
||||
risk = str(entry.get("risk_note") or "").strip() or "无"
|
||||
rankings.append(Ranking(journal_id=jid, tier=tier, match_reason=reason, risk_note=risk))
|
||||
|
||||
for jid in input_ids:
|
||||
if jid in seen:
|
||||
continue
|
||||
rankings.append(
|
||||
Ranking(
|
||||
journal_id=jid,
|
||||
tier=rule_tier(by_id[jid]),
|
||||
match_reason="LLM 未给出该期刊条目,已按影响因子/周期规则补位",
|
||||
risk_note="未提供(LLM 输出缺漏,规则兜底)",
|
||||
)
|
||||
)
|
||||
repaired += 1
|
||||
return rankings, dropped, repaired
|
||||
|
||||
|
||||
async def rerank(
|
||||
abstract: str,
|
||||
candidates: list[dict[str, Any]],
|
||||
preferences: dict[str, Any],
|
||||
) -> tuple[list[Ranking], int, int]:
|
||||
"""LLM 重排 + 校验。LLMError / RankError 上抛(路由降级为规则分档)。"""
|
||||
settings = get_settings()
|
||||
payload = await llm.complete_json(
|
||||
SYSTEM_PROMPT,
|
||||
build_user_prompt(abstract, candidates, preferences),
|
||||
temperature=0.2,
|
||||
max_tokens=3000,
|
||||
timeout=settings.recommend_read_timeout,
|
||||
)
|
||||
return validate_rankings(payload, candidates)
|
||||
|
||||
|
||||
def fallback_rankings(candidates: list[dict[str, Any]]) -> list[Ranking]:
|
||||
"""LLM 不可用时的确定性分档(不丢任何候选期刊)。"""
|
||||
rankings: list[Ranking] = []
|
||||
for c in candidates:
|
||||
j = c["journal"]
|
||||
inf = j.get("impact_factor")
|
||||
weeks = j.get("review_time_weeks")
|
||||
rankings.append(
|
||||
Ranking(
|
||||
journal_id=str(j["id"]),
|
||||
tier=rule_tier(j),
|
||||
match_reason=(
|
||||
f"规则分档:影响因子 {inf if inf is not None else '未收录'}、"
|
||||
f"审稿约 {weeks if weeks is not None else '未收录'} 周"
|
||||
),
|
||||
risk_note="LLM 不可用,本条按影响因子/周期规则估算",
|
||||
)
|
||||
)
|
||||
return rankings
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user