Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 admin
|
99bd86e535 |
+3
-21
@@ -1,4 +1,4 @@
|
||||
# 运行时产物与环境
|
||||
# 运行时产物与环境
|
||||
node_modules
|
||||
dist/
|
||||
.venv/
|
||||
@@ -37,23 +37,5 @@ tests/extension_cases_18/
|
||||
bioflow_service/bioflow.db
|
||||
*.before-*
|
||||
|
||||
# paper_writer_service 运行时产物（不入库）
|
||||
paper_writer_service/paper.db
|
||||
paper_writer_service/paper.db-shm
|
||||
paper_writer_service/paper.db-wal
|
||||
paper_writer_service/paper.log
|
||||
|
||||
# review_service 运行时产物（不入库）
|
||||
review_service/review.db
|
||||
review_service/review.db-wal
|
||||
review_service/review.db-shm
|
||||
review_service/review.log
|
||||
review_service/data/reformat_jobs/
|
||||
# 期刊种子 journals.json 是源码：顶层 **/data/ 会误伤 review_service/data/，显式放行
|
||||
!review_service/data/
|
||||
review_service/tools/tectonic
|
||||
review_service/data/tectonic-cache/
|
||||
review_service/data/format_cache/
|
||||
|
||||
# 部署方案文档（含主机凭据，不入库）
|
||||
docs/DEPLOY_PAPER_176.md
|
||||
# Python 覆盖率运行时产物(coverage.py --cov 生成)
|
||||
.coverage
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
# AGENTS.md —— 协作约束（科研审稿模块）
|
||||
|
||||
## 项目背景
|
||||
|
||||
本项目（PlantAI-rag-new）是植物基因组学文献知识工作台，拥有自研 RAG 与知识库
|
||||
（Qdrant + Neo4j + vstore）。2026-10 起新增「科研投稿与智能审稿助手」功能
|
||||
（审稿周期/期刊分析/期刊推荐/投稿前自审/审稿意见辅助/格式检查）。
|
||||
|
||||
## 审稿模块（review_service）约束 —— 必须遵守
|
||||
|
||||
1. **所有审稿相关新代码只允许写在 `review_service/` 目录下新增**。
|
||||
- 服务自身：FastAPI 独立进程，端口 8100（模式同 `paper_writer_service` 8097 / `bioflow_service` 8098）。
|
||||
- 环境变量前缀统一 `REVIEW_`；LLM 凭据复用 `DOCGEN_LLM_*` 裸键（回退链见 `review_service/app/services/llm.py`）。
|
||||
2. **BFF（`sdk/`）层只允许条件注册代理路由**：
|
||||
- 允许的全部改动 = `sdk/app/config.py` 增加 `review_enabled` / `review_service_url` 等开关键
|
||||
+ `sdk/app/main.py` 按开关 `include_router(review_proxy.router)`
|
||||
+ 新文件 `sdk/app/routers/review_proxy.py`（反代实现）。
|
||||
- **严禁**修改 sdk 下其他 router/service/db 的任何既有逻辑。
|
||||
3. **严禁修改现有 RAG 核心与既有服务代码**，包括但不限于：
|
||||
- `agent/`（qa 8091 / methodology 8092 / legacy-rag 8093）
|
||||
- `vstore/`（8094，审稿服务只经 HTTP 只读消费其 `/api/vstore/*` 接口）
|
||||
- `paper_writer_service/`、`bioflow_service/`、`bioflow_kernel/`
|
||||
- `ui/`（前端入口另行按 UI_SPEC 流程评审）
|
||||
- `knowledge/`、`kb-libraries/`、`tooling/`（依赖锁另议）
|
||||
4. **依赖纪律**：审稿服务只使用统一 venv（`tooling/requirements.lock`）中已有的库
|
||||
（fastapi / httpx / pydantic-settings 等）。如确需新增全局依赖，
|
||||
须先更新 `tooling/requirements.lock` 并经人工确认。
|
||||
5. **上游对接只读**：PDF 解析走 vstore 8094（`POST /api/vstore/ingest` +
|
||||
`GET/DELETE /api/vstore/documents/{id}`）；LLM 走 DeepSeek OpenAI 兼容接口；
|
||||
后续期刊数据/期刊推荐等子功能同样以新 service + 新路由在 `review_service/` 内扩展。
|
||||
6. **format_check 例外**：format_check 允许用 pymupdf 本地解档获取版式元数据
|
||||
（字体/边距/行距/页数），因为 vstore 只回文本不返回版式信息。正文提取仍走 vstore。
|
||||
|
||||
## 目录速览（审稿模块）
|
||||
|
||||
```
|
||||
review_service/
|
||||
├── app/main.py # FastAPI 入口（/health + 挂载业务路由）
|
||||
├── app/config.py # pydantic-settings（REVIEW_ 前缀）
|
||||
├── app/routers/presubmit.py # POST /api/review/presubmit（自审稿报告）
|
||||
├── app/routers/format_check.py # POST /api/review/format-check（版式与规范检查）
|
||||
├── app/services/llm.py # DeepSeek httpx 封装（complete_text/complete_json）
|
||||
├── app/services/paper_parse.py # PDF → 纯文本（经 vstore 解析接口）
|
||||
├── app/services/format_check.py# 版式规则引擎（pymupdf/docx 本地解档 + 规则检查）
|
||||
├── restart.sh # 8100 重启脚本
|
||||
└── pytest.ini
|
||||
```
|
||||
|
||||
## 启动
|
||||
|
||||
```bash
|
||||
cd review_service && bash restart.sh # 默认 8100，venv 从 ../plantai-rag-new-venv 推导
|
||||
# BFF 开关：sdk/.env 加 REVIEW_ENABLED=1 后 bff_restart（反代 /api/review/*）
|
||||
```
|
||||
@@ -1,12 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN" data-theme="dark">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
|
||||
<title>蜂群中枢 · PlantAI 管理后台</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script type="module" src="/src/main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
Generated
-1147
File diff suppressed because it is too large
Load Diff
@@ -1,18 +0,0 @@
|
||||
{
|
||||
"name": "plantai-admin",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"vue": "3.5.13"
|
||||
},
|
||||
"devDependencies": {
|
||||
"vite": "5.4.11",
|
||||
"@vitejs/plugin-vue": "5.2.1"
|
||||
}
|
||||
}
|
||||
@@ -1,143 +0,0 @@
|
||||
<script setup>
|
||||
import { ref, onMounted } from 'vue';
|
||||
import { getToken, setToken, api } from './api/admin.js';
|
||||
import Login from './views/Login.vue';
|
||||
import Users from './views/Users.vue';
|
||||
import Logs from './views/Logs.vue';
|
||||
import HubSettings from './components/HubSettings.vue';
|
||||
|
||||
const authed = ref(false);
|
||||
const page = ref('overview');
|
||||
const user = ref(null);
|
||||
const stats = ref({ users: 0, total: 0, denied: 0, p95: null });
|
||||
const hub = ref({ hub_id: '', hub_name: '蜂群中枢' });
|
||||
const settingsOpen = ref(false);
|
||||
|
||||
const PAGES = { overview: '概览', users: '用户管理', logs: '日志查看' };
|
||||
|
||||
function onUnauthorized() { setToken(''); authed.value = false; }
|
||||
|
||||
async function loadHub() {
|
||||
try {
|
||||
const r = await api.settings();
|
||||
hub.value = r.settings;
|
||||
} catch (e) {
|
||||
if (e.status === 401) onUnauthorized();
|
||||
}
|
||||
}
|
||||
|
||||
async function loadStats() {
|
||||
try {
|
||||
const [u, l] = await Promise.all([api.users(), api.logs({ page: 1, page_size: 1 })]);
|
||||
stats.value.users = u.users.length;
|
||||
stats.value.total = l.total;
|
||||
stats.value.denied = (l.aggregates && l.aggregates.denied) || 0;
|
||||
stats.value.p95 = (l.aggregates && l.aggregates.p95_latency_ms) || null;
|
||||
} catch (e) {
|
||||
if (e.status === 401) onUnauthorized();
|
||||
}
|
||||
}
|
||||
|
||||
async function init() {
|
||||
if (!getToken()) { authed.value = false; return; }
|
||||
try {
|
||||
const r = await api.me();
|
||||
user.value = r.user;
|
||||
authed.value = true;
|
||||
loadHub();
|
||||
loadStats();
|
||||
} catch (e) {
|
||||
setToken('');
|
||||
authed.value = false;
|
||||
}
|
||||
}
|
||||
|
||||
function onLogin(u) { user.value = u; authed.value = true; page.value = 'overview'; loadHub(); loadStats(); }
|
||||
async function onLogout() { try { await api.logout(); } catch {} setToken(''); authed.value = false; }
|
||||
function onSettingsSaved(s) { hub.value = s; settingsOpen.value = false; }
|
||||
|
||||
onMounted(init);
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<Login v-if="!authed" @login="onLogin" />
|
||||
<div v-else class="shell">
|
||||
<aside class="sidebar">
|
||||
<div class="side-brand">
|
||||
<div class="hex">蜂</div>
|
||||
<div><div class="brand-name">{{ hub.hub_name }}</div><div class="brand-sub">{{ hub.hub_id.slice(0, 8) }}</div></div>
|
||||
</div>
|
||||
<nav class="nav">
|
||||
<div class="nav-item" :class="{ active: page === 'overview' }" @click="page = 'overview'"><span>◈</span> 概览</div>
|
||||
<div class="nav-item" :class="{ active: page === 'users' }" @click="page = 'users'"><span>◉</span> 用户管理</div>
|
||||
<div class="nav-item" :class="{ active: page === 'logs' }" @click="page = 'logs'"><span>≣</span> 日志查看</div>
|
||||
</nav>
|
||||
<div class="side-foot">
|
||||
<div class="who">
|
||||
<div class="avatar">蜂</div>
|
||||
<div><div class="who-name">{{ user?.username }}</div><div class="who-role">系统管理员</div></div>
|
||||
<span class="logout" @click="onLogout">退出</span>
|
||||
</div>
|
||||
</div>
|
||||
</aside>
|
||||
<main class="main">
|
||||
<div class="topbar">
|
||||
<h1>{{ PAGES[page] }}</h1>
|
||||
<span style="color: var(--text-3); font-size: 13px">{{ hub.hub_name }}</span>
|
||||
<button class="btn btn-ghost" style="margin-left: auto" @click="settingsOpen = true">⚙ 设置</button>
|
||||
</div>
|
||||
<div class="content">
|
||||
<section v-if="page === 'overview'">
|
||||
<div class="stats">
|
||||
<div class="stat"><div class="k">{{ stats.users }}</div><div class="l">注册用户</div></div>
|
||||
<div class="stat"><div class="k amber">{{ stats.total }}</div><div class="l">累计请求</div></div>
|
||||
<div class="stat"><div class="k">{{ stats.denied }}</div><div class="l">越权 / 失败</div></div>
|
||||
<div class="stat"><div class="k">{{ stats.p95 ?? '—' }}</div><div class="l">P95 耗时 ms</div></div>
|
||||
</div>
|
||||
</section>
|
||||
<Users v-else-if="page === 'users'" @unauthorized="onUnauthorized" />
|
||||
<Logs v-else @unauthorized="onUnauthorized" />
|
||||
</div>
|
||||
</main>
|
||||
<HubSettings :open="settingsOpen" :hub="hub" @close="settingsOpen = false" @saved="onSettingsSaved" />
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style>
|
||||
.shell { display: flex; height: 100vh; }
|
||||
.sidebar { width: 218px; flex: none; background: var(--bg-2); border-right: 1px solid var(--border); display: flex; flex-direction: column; }
|
||||
.side-brand { display: flex; align-items: center; gap: 10px; padding: 20px 18px 18px; }
|
||||
.hex { width: 30px; height: 30px; flex: none; background: linear-gradient(135deg, #f2b95a, #d98f22); clip-path: polygon(50% 0, 100% 25%, 100% 75%, 50% 100%, 0 75%, 0 25%); display: grid; place-items: center; color: #1a1206; font-weight: 700; font-size: 12px; }
|
||||
.brand-name { font-size: 15px; font-weight: 600; }
|
||||
.brand-sub { font-size: 11px; color: var(--text-3); }
|
||||
.nav { padding: 6px 10px; flex: 1; }
|
||||
.nav-item { display: flex; align-items: center; gap: 9px; padding: 9px 12px; margin-bottom: 3px; border-radius: var(--radius-sm); color: var(--text-2); font-size: 13.5px; cursor: pointer; }
|
||||
.nav-item:hover { background: var(--surface); color: var(--text); }
|
||||
.nav-item.active { background: var(--accent-soft); color: var(--accent); }
|
||||
.side-foot { padding: 14px 18px; border-top: 1px solid var(--border); }
|
||||
.who { display: flex; align-items: center; gap: 10px; }
|
||||
.avatar { width: 30px; height: 30px; border-radius: 50%; flex: none; background: linear-gradient(135deg, #3a4a5e, #243146); display: grid; place-items: center; font-size: 12px; color: var(--text); }
|
||||
.who-name { font-size: 13px; }
|
||||
.who-role { font-size: 11px; color: var(--text-3); }
|
||||
.logout { margin-left: auto; color: var(--text-3); cursor: pointer; font-size: 12px; }
|
||||
.logout:hover { color: var(--red); }
|
||||
.main { flex: 1; overflow-y: auto; display: flex; flex-direction: column; }
|
||||
.topbar { display: flex; align-items: center; gap: 14px; padding: 18px 28px; border-bottom: 1px solid var(--border); position: sticky; top: 0; background: rgba(12, 14, 20, 0.9); backdrop-filter: blur(6px); z-index: 5; }
|
||||
.topbar h1 { font-size: 17px; margin: 0; font-weight: 600; }
|
||||
.content { padding: 26px 28px 40px; }
|
||||
.stats { display: grid; grid-template-columns: repeat(4, 1fr); gap: 16px; }
|
||||
.stat { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius); padding: 18px 20px; }
|
||||
.stat .k { font-size: 26px; font-weight: 600; letter-spacing: -0.5px; }
|
||||
.stat .k.amber { color: var(--accent); }
|
||||
.stat .l { color: var(--text-2); font-size: 12.5px; margin-top: 3px; }
|
||||
.toolbar { display: flex; align-items: center; gap: 10px; margin-bottom: 16px; }
|
||||
.toolbar .spacer { flex: 1; }
|
||||
.search { display: flex; align-items: center; gap: 8px; background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius-sm); padding: 8px 12px; width: 240px; }
|
||||
.search input { background: none; border: none; outline: none; color: var(--text); font-size: 13px; width: 100%; }
|
||||
select { background: var(--surface); border: 1px solid var(--border); color: var(--text); border-radius: var(--radius-sm); padding: 8px 10px; font-size: 13px; outline: none; }
|
||||
.pag { display: flex; align-items: center; gap: 8px; padding: 12px 18px; border-top: 1px solid var(--border); color: var(--text-3); font-size: 12.5px; }
|
||||
.pag .spacer { flex: 1; }
|
||||
.pag button { background: var(--surface-2); border: 1px solid var(--border); color: var(--text-2); border-radius: 6px; padding: 5px 11px; font-size: 12.5px; }
|
||||
.pag button.on { color: var(--accent); border-color: var(--accent); }
|
||||
.pag button:disabled { opacity: 0.4; cursor: default; }
|
||||
</style>
|
||||
@@ -1,48 +0,0 @@
|
||||
const TOKEN_KEY = 'pa-admin-token';
|
||||
let token = localStorage.getItem(TOKEN_KEY) || '';
|
||||
|
||||
export function getToken() { return token; }
|
||||
export function setToken(t) {
|
||||
token = t || '';
|
||||
if (t) localStorage.setItem(TOKEN_KEY, t);
|
||||
else localStorage.removeItem(TOKEN_KEY);
|
||||
}
|
||||
|
||||
async function request(path, { method = 'GET', body } = {}) {
|
||||
const res = await fetch('/api/admin' + path, {
|
||||
method,
|
||||
headers: {
|
||||
...(body ? { 'Content-Type': 'application/json' } : {}),
|
||||
...(token ? { Authorization: 'Bearer ' + token } : {}),
|
||||
},
|
||||
body: body ? JSON.stringify(body) : undefined,
|
||||
});
|
||||
let data = null;
|
||||
try { data = await res.json(); } catch { data = null; }
|
||||
if (!res.ok) {
|
||||
const err = new Error((data && data.error && data.error.message) || ('请求失败 HTTP ' + res.status));
|
||||
err.status = res.status;
|
||||
err.code = data && data.error && data.error.code;
|
||||
throw err;
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
export const api = {
|
||||
login: (username, password) => request('/login', { method: 'POST', body: { username, password } }),
|
||||
logout: () => request('/logout', { method: 'POST' }),
|
||||
me: () => request('/me'),
|
||||
units: () => request('/units'),
|
||||
users: () => request('/users'),
|
||||
createUser: (body) => request('/users', { method: 'POST', body }),
|
||||
updateUser: (id, body) => request('/users/' + id, { method: 'PATCH', body }),
|
||||
deleteUser: (id) => request('/users/' + id, { method: 'DELETE' }),
|
||||
logs: (params = {}) => {
|
||||
const q = new URLSearchParams();
|
||||
Object.entries(params).forEach(([k, v]) => { if (v !== undefined && v !== null && v !== '') q.set(k, v); });
|
||||
const qs = q.toString();
|
||||
return request('/logs' + (qs ? '?' + qs : ''));
|
||||
},
|
||||
settings: () => request('/settings'),
|
||||
updateSettings: (body) => request('/settings', { method: 'PUT', body }),
|
||||
};
|
||||
@@ -1,57 +0,0 @@
|
||||
<script setup>
|
||||
import { ref, watch } from 'vue';
|
||||
import { api } from '../api/admin.js';
|
||||
|
||||
const props = defineProps({ open: Boolean, hub: Object });
|
||||
const emit = defineEmits(['close', 'saved']);
|
||||
|
||||
const hubId = ref('');
|
||||
const hubName = ref('');
|
||||
const busy = ref(false);
|
||||
const error = ref('');
|
||||
|
||||
watch(() => props.open, (open) => {
|
||||
if (open) {
|
||||
hubId.value = (props.hub && props.hub.hub_id) || '';
|
||||
hubName.value = (props.hub && props.hub.hub_name) || '';
|
||||
error.value = '';
|
||||
busy.value = false;
|
||||
}
|
||||
});
|
||||
|
||||
async function save() {
|
||||
if (busy.value) return;
|
||||
if (!hubName.value.trim()) { error.value = '中枢名称不能为空'; return; }
|
||||
busy.value = true;
|
||||
error.value = '';
|
||||
try {
|
||||
const r = await api.updateSettings({ hub_id: hubId.value.trim(), hub_name: hubName.value.trim() });
|
||||
emit('saved', r.settings);
|
||||
} catch (e) {
|
||||
error.value = e.message || '保存失败';
|
||||
} finally {
|
||||
busy.value = false;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div v-if="open" class="overlay" @click.self="emit('close')">
|
||||
<div class="modal">
|
||||
<div class="modal-head">中枢设置</div>
|
||||
<div class="modal-body">
|
||||
<div class="field"><label>中枢名称</label><input v-model="hubName" placeholder="例如:湘湖实验室中枢"></div>
|
||||
<div class="field" style="margin-bottom: 0"><label>中枢 ID</label><input v-model="hubId" class="mono" placeholder="唯一标识,留空则由系统生成"></div>
|
||||
<p v-if="error" class="err">{{ error }}</p>
|
||||
</div>
|
||||
<div class="modal-foot">
|
||||
<button class="btn btn-ghost" @click="emit('close')">取消</button>
|
||||
<button class="btn btn-primary" :disabled="busy" @click="save">{{ busy ? '保存中…' : '保存' }}</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style scoped>
|
||||
.err { color: var(--red); font-size: 12px; margin: 12px 0 0; }
|
||||
</style>
|
||||
@@ -1,101 +0,0 @@
|
||||
<script setup>
|
||||
import { ref, watch } from 'vue';
|
||||
import { api } from '../api/admin.js';
|
||||
|
||||
const props = defineProps({
|
||||
open: Boolean,
|
||||
mode: { type: String, default: 'create' },
|
||||
user: Object,
|
||||
});
|
||||
const emit = defineEmits(['close', 'saved']);
|
||||
|
||||
const UNITS = [
|
||||
['qa', '问答'], ['kg', '知识图谱'], ['extraction', '文献抽取'], ['docgen', '文档生成'],
|
||||
['figures', '图片图库'], ['private_docs', '私有资料'], ['bioflow', '生信分析'],
|
||||
['paper', '论文撰写'], ['legacy', '旧版接口'],
|
||||
];
|
||||
|
||||
const username = ref('');
|
||||
const password = ref('');
|
||||
const role = ref('researcher');
|
||||
const authorizations = ref([]);
|
||||
const busy = ref(false);
|
||||
const error = ref('');
|
||||
|
||||
watch(() => props.open, (open) => {
|
||||
if (open) {
|
||||
error.value = '';
|
||||
busy.value = false;
|
||||
if (props.mode === 'edit' && props.user) {
|
||||
username.value = props.user.username;
|
||||
password.value = '';
|
||||
role.value = props.user.role === 'admin' ? 'admin' : (props.user.role === 'guest' ? 'guest' : 'researcher');
|
||||
authorizations.value = [...(props.user.authorizations || [])];
|
||||
} else {
|
||||
username.value = '';
|
||||
password.value = '';
|
||||
role.value = 'researcher';
|
||||
authorizations.value = ['qa'];
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
async function save() {
|
||||
if (busy.value) return;
|
||||
if (props.mode === 'create' && (!username.value.trim() || !password.value)) {
|
||||
error.value = '用户名和密码必填';
|
||||
return;
|
||||
}
|
||||
busy.value = true;
|
||||
error.value = '';
|
||||
try {
|
||||
if (props.mode === 'create') {
|
||||
await api.createUser({ username: username.value.trim(), password: password.value, role: role.value, authorizations: authorizations.value });
|
||||
} else {
|
||||
const body = { role: role.value, authorizations: authorizations.value };
|
||||
if (password.value) body.password = password.value;
|
||||
await api.updateUser(props.user.id, body);
|
||||
}
|
||||
emit('saved');
|
||||
} catch (e) {
|
||||
error.value = e.message || '保存失败';
|
||||
} finally {
|
||||
busy.value = false;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div v-if="open" class="overlay" @click.self="emit('close')">
|
||||
<div class="modal">
|
||||
<div class="modal-head">{{ mode === 'create' ? '新建用户' : '编辑用户' }}</div>
|
||||
<div class="modal-body">
|
||||
<div class="field"><label>用户名</label><input v-model="username" :disabled="mode === 'edit'"></div>
|
||||
<div class="field"><label>{{ mode === 'create' ? '初始密码' : '重置密码（留空不改）' }}</label><input v-model="password" type="password"></div>
|
||||
<div class="field">
|
||||
<label>角色</label>
|
||||
<div class="seg">
|
||||
<button :class="{ on: role === 'researcher' }" @click="role = 'researcher'">研究员</button>
|
||||
<button :class="{ on: role === 'guest' }" @click="role = 'guest'">访客</button>
|
||||
<button :class="{ on: role === 'admin' }" @click="role = 'admin'">管理员</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="field" style="margin-bottom: 0">
|
||||
<label>授权清单</label>
|
||||
<div class="checks">
|
||||
<label v-for="[k, label] in UNITS" :key="k" class="check"><input type="checkbox" :value="k" v-model="authorizations"> {{ label }}</label>
|
||||
</div>
|
||||
</div>
|
||||
<p v-if="error" class="err">{{ error }}</p>
|
||||
</div>
|
||||
<div class="modal-foot">
|
||||
<button class="btn btn-ghost" @click="emit('close')">取消</button>
|
||||
<button class="btn btn-primary" :disabled="busy" @click="save">{{ busy ? '保存中…' : (mode === 'create' ? '创建用户' : '保存') }}</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style scoped>
|
||||
.err { color: var(--red); font-size: 12px; margin: 12px 0 0; }
|
||||
</style>
|
||||
@@ -1,5 +0,0 @@
|
||||
import { createApp } from 'vue';
|
||||
import App from './App.vue';
|
||||
import './style.css';
|
||||
|
||||
createApp(App).mount('#app');
|
||||
@@ -1,96 +0,0 @@
|
||||
:root {
|
||||
--bg: #0c0e14;
|
||||
--bg-2: #10131b;
|
||||
--surface: #151924;
|
||||
--surface-2: #1a1f2c;
|
||||
--border: #232a38;
|
||||
--border-2: #2e3648;
|
||||
--text: #e9edf5;
|
||||
--text-2: #9aa4b6;
|
||||
--text-3: #5f6a7d;
|
||||
--accent: #e8a33d;
|
||||
--accent-ink: #14100a;
|
||||
--accent-soft: rgba(232, 163, 61, 0.14);
|
||||
--green: #3ecf8e;
|
||||
--green-soft: rgba(62, 207, 142, 0.14);
|
||||
--red: #ef6b6b;
|
||||
--red-soft: rgba(239, 107, 107, 0.14);
|
||||
--blue: #6ea8ff;
|
||||
--radius: 10px;
|
||||
--radius-sm: 7px;
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; }
|
||||
html, body, #app { margin: 0; height: 100%; }
|
||||
body {
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
font-family: -apple-system, BlinkMacSystemFont, "PingFang SC", "Microsoft YaHei", "Segoe UI", Roboto, sans-serif;
|
||||
font-size: 14px;
|
||||
line-height: 1.5;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
.mono { font-family: ui-monospace, "SF Mono", "JetBrains Mono", Menlo, Consolas, monospace; }
|
||||
button { font-family: inherit; cursor: pointer; }
|
||||
|
||||
.btn {
|
||||
display: inline-flex; align-items: center; justify-content: center; gap: 7px;
|
||||
padding: 9px 15px; font-size: 13.5px; border-radius: var(--radius-sm);
|
||||
border: 1px solid transparent; transition: filter .15s; background: transparent;
|
||||
}
|
||||
.btn:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
|
||||
.btn-primary { background: var(--accent); color: var(--accent-ink); font-weight: 600; }
|
||||
.btn-primary:hover { filter: brightness(1.06); }
|
||||
.btn-ghost { border-color: var(--border-2); color: var(--text-2); }
|
||||
.btn-ghost:hover { color: var(--text); border-color: var(--text-3); }
|
||||
.btn-danger { background: var(--red-soft); color: var(--red); }
|
||||
.btn-block { width: 100%; }
|
||||
|
||||
.field { margin-bottom: 15px; }
|
||||
.field label { display: block; font-size: 12px; color: var(--text-2); margin-bottom: 7px; }
|
||||
.field input, .field select {
|
||||
width: 100%; padding: 10px 12px; font-size: 14px; color: var(--text);
|
||||
background: var(--bg-2); border: 1px solid var(--border); border-radius: var(--radius-sm); outline: none;
|
||||
}
|
||||
.field input:focus, .field select:focus { border-color: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
|
||||
|
||||
.chip {
|
||||
display: inline-block; padding: 3px 9px; border-radius: 20px; font-size: 11.5px;
|
||||
border: 1px solid var(--border-2); color: var(--text-2); margin: 1.5px 3px 1.5px 0;
|
||||
}
|
||||
.chip.amber { border-color: transparent; background: var(--accent-soft); color: var(--accent); }
|
||||
.badge { display: inline-block; padding: 2px 8px; border-radius: 5px; font-size: 11.5px; font-family: ui-monospace, Menlo, monospace; }
|
||||
.badge.green { background: var(--green-soft); color: var(--green); }
|
||||
.badge.red { background: var(--red-soft); color: var(--red); }
|
||||
.badge.amber { background: var(--accent-soft); color: var(--accent); }
|
||||
.badge.muted { background: var(--surface-2); color: var(--text-3); }
|
||||
|
||||
.panel { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius); }
|
||||
table { width: 100%; border-collapse: collapse; }
|
||||
th { text-align: left; font-size: 11.5px; font-weight: 500; color: var(--text-3); padding: 10px 18px; border-bottom: 1px solid var(--border); }
|
||||
td { padding: 13px 18px; border-bottom: 1px solid var(--border); font-size: 13.5px; }
|
||||
tr:last-child td { border-bottom: none; }
|
||||
tbody tr:hover { background: rgba(255,255,255,0.015); }
|
||||
.cell-id { color: var(--text-2); }
|
||||
.row-actions { display: flex; gap: 6px; }
|
||||
.row-actions .act { color: var(--blue); cursor: pointer; font-size: 13px; padding: 3px 6px; border-radius: 5px; }
|
||||
.row-actions .act:hover { background: rgba(110,168,255,0.1); }
|
||||
.row-actions .act.danger { color: var(--red); }
|
||||
.row-actions .act.danger:hover { background: var(--red-soft); }
|
||||
|
||||
.role { font-weight: 600; }
|
||||
.role.admin { color: var(--accent); }
|
||||
.role.researcher { color: var(--blue); }
|
||||
.role.guest { color: var(--text-2); }
|
||||
|
||||
.overlay { position: fixed; inset: 0; background: rgba(0,0,0,0.55); display: flex; align-items: center; justify-content: center; z-index: 50; }
|
||||
.modal { width: 460px; max-width: calc(100vw - 32px); background: var(--surface); border: 1px solid var(--border-2); border-radius: 14px; }
|
||||
.modal-head { padding: 18px 22px 0; font-size: 15px; font-weight: 600; }
|
||||
.modal-body { padding: 18px 22px; }
|
||||
.modal-foot { padding: 14px 22px 20px; display: flex; justify-content: flex-end; gap: 10px; border-top: 1px solid var(--border); }
|
||||
.checks { display: grid; grid-template-columns: 1fr 1fr; gap: 9px 14px; margin-top: 8px; }
|
||||
.check { display: flex; align-items: center; gap: 8px; font-size: 13px; color: var(--text-2); }
|
||||
.check input { accent-color: var(--accent); width: 15px; height: 15px; }
|
||||
.seg { display: flex; gap: 6px; }
|
||||
.seg button { background: transparent; border: 1px solid var(--border-2); color: var(--text-2); border-radius: 7px; padding: 7px 14px; font-size: 13px; }
|
||||
.seg button.on { background: var(--accent-soft); border-color: transparent; color: var(--accent); }
|
||||
@@ -1,53 +0,0 @@
|
||||
<script setup>
|
||||
import { ref } from 'vue';
|
||||
import { api, setToken } from '../api/admin.js';
|
||||
|
||||
const emit = defineEmits(['login']);
|
||||
const username = ref('');
|
||||
const password = ref('');
|
||||
const error = ref('');
|
||||
const busy = ref(false);
|
||||
|
||||
async function submit() {
|
||||
if (busy.value) return;
|
||||
if (!username.value || !password.value) { error.value = '请输入用户名和密码'; return; }
|
||||
busy.value = true;
|
||||
error.value = '';
|
||||
try {
|
||||
const r = await api.login(username.value, password.value);
|
||||
setToken(r.token);
|
||||
emit('login', r.user);
|
||||
} catch (e) {
|
||||
error.value = e.message || '登录失败';
|
||||
} finally {
|
||||
busy.value = false;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div class="login">
|
||||
<div class="card">
|
||||
<div class="brand">
|
||||
<div class="hex">蜂</div>
|
||||
<div><div class="brand-name">蜂群中枢</div><div class="brand-sub">PlantAI 管理后台</div></div>
|
||||
</div>
|
||||
<div class="field"><label>用户名</label><input v-model="username" autocomplete="username" @keyup.enter="submit"></div>
|
||||
<div class="field"><label>密码</label><input v-model="password" type="password" autocomplete="current-password" @keyup.enter="submit"></div>
|
||||
<button class="btn btn-primary btn-block" :disabled="busy" @click="submit">{{ busy ? '登录中…' : '登录' }}</button>
|
||||
<p v-if="error" class="err">{{ error }}</p>
|
||||
<div class="hint">仅限中枢管理员。登录与所有操作均会留痕。</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style scoped>
|
||||
.login { height: 100vh; display: flex; align-items: center; justify-content: center; background: radial-gradient(1200px 600px at 70% -10%, rgba(232, 163, 61, 0.08), transparent 60%), var(--bg); }
|
||||
.card { width: 360px; background: var(--surface); border: 1px solid var(--border); border-radius: 14px; padding: 34px 32px 30px; }
|
||||
.brand { display: flex; align-items: center; gap: 11px; margin-bottom: 26px; }
|
||||
.hex { width: 34px; height: 34px; flex: none; background: linear-gradient(135deg, #f2b95a, #d98f22); clip-path: polygon(50% 0, 100% 25%, 100% 75%, 50% 100%, 0 75%, 0 25%); display: grid; place-items: center; color: #1a1206; font-weight: 700; font-size: 14px; }
|
||||
.brand-name { font-size: 16px; font-weight: 600; }
|
||||
.brand-sub { color: var(--text-3); font-size: 12px; }
|
||||
.err { color: var(--red); font-size: 12px; margin: 10px 0 0; }
|
||||
.hint { margin-top: 20px; padding-top: 16px; border-top: 1px solid var(--border); font-size: 12px; color: var(--text-3); }
|
||||
</style>
|
||||
@@ -1,102 +0,0 @@
|
||||
<script setup>
|
||||
import { ref, onMounted } from 'vue';
|
||||
import { api } from '../api/admin.js';
|
||||
|
||||
const emit = defineEmits(['unauthorized']);
|
||||
|
||||
const from = ref('');
|
||||
const event = ref('');
|
||||
const q = ref('');
|
||||
const page = ref(1);
|
||||
const pageSize = 30;
|
||||
const items = ref([]);
|
||||
const total = ref(0);
|
||||
const aggregates = ref({});
|
||||
const loading = ref(false);
|
||||
|
||||
const EVENT_LABEL = {
|
||||
http_request: '调用', admin_login_ok: '登录成功', admin_login_fail: '登录失败',
|
||||
call_denied: '越权', user_created: '建用户', user_updated: '改用户', user_deleted: '删用户',
|
||||
};
|
||||
|
||||
function statusBadge(s) {
|
||||
if (!s) return 'muted';
|
||||
if (s >= 200 && s < 300) return 'green';
|
||||
if (s >= 500) return 'red';
|
||||
return 'amber';
|
||||
}
|
||||
|
||||
async function load() {
|
||||
loading.value = true;
|
||||
try {
|
||||
const r = await api.logs({
|
||||
from_date: from.value || undefined,
|
||||
event: event.value || undefined,
|
||||
q: q.value || undefined,
|
||||
page: page.value,
|
||||
page_size: pageSize,
|
||||
});
|
||||
items.value = r.items;
|
||||
total.value = r.total;
|
||||
aggregates.value = r.aggregates || {};
|
||||
} catch (e) {
|
||||
if (e.status === 401) emit('unauthorized');
|
||||
} finally {
|
||||
loading.value = false;
|
||||
}
|
||||
}
|
||||
|
||||
function search() { page.value = 1; load(); }
|
||||
function prev() { if (page.value > 1) { page.value -= 1; load(); } }
|
||||
function next() { if (page.value * pageSize < total.value) { page.value += 1; load(); } }
|
||||
|
||||
onMounted(load);
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div>
|
||||
<div class="stats" style="margin-bottom: 16px">
|
||||
<div class="stat"><div class="k">{{ total }}</div><div class="l">匹配日志</div></div>
|
||||
<div class="stat"><div class="k amber">{{ aggregates.denied ?? 0 }}</div><div class="l">越权 / 失败</div></div>
|
||||
<div class="stat"><div class="k">{{ aggregates.avg_latency_ms ?? '—' }}</div><div class="l">平均耗时 ms</div></div>
|
||||
<div class="stat"><div class="k">{{ aggregates.p95_latency_ms ?? '—' }}</div><div class="l">P95 耗时 ms</div></div>
|
||||
</div>
|
||||
<div class="toolbar">
|
||||
<input class="date" v-model="from" type="date">
|
||||
<select v-model="event">
|
||||
<option value="">全部事件</option>
|
||||
<option v-for="(l, k) in EVENT_LABEL" :key="k" :value="k">{{ l }}</option>
|
||||
</select>
|
||||
<div class="search"><input v-model="q" placeholder="按用户 / 路径过滤…" @keyup.enter="search"></div>
|
||||
<button class="btn btn-ghost" @click="search">查询</button>
|
||||
</div>
|
||||
<div class="panel">
|
||||
<table>
|
||||
<thead><tr><th>时间</th><th>用户</th><th>事件</th><th>方法</th><th>路径</th><th>状态</th><th>耗时</th></tr></thead>
|
||||
<tbody>
|
||||
<tr v-if="loading"><td colspan="7" class="cell-id">加载中…</td></tr>
|
||||
<tr v-else-if="!items.length"><td colspan="7" class="cell-id">暂无日志</td></tr>
|
||||
<tr v-for="(e, i) in items" :key="i">
|
||||
<td class="cell-id mono">{{ (e.ts || '').replace('T', ' ').slice(0, 19) }}</td>
|
||||
<td class="mono cell-id">{{ e.user_id ? e.user_id.slice(0, 8) : '—' }}</td>
|
||||
<td>{{ EVENT_LABEL[e.event] || e.event }}</td>
|
||||
<td class="cell-id mono">{{ e.method || '—' }}</td>
|
||||
<td class="mono">{{ e.path || '—' }}</td>
|
||||
<td><span v-if="e.status" class="badge" :class="statusBadge(e.status)">{{ e.status }}</span><span v-else class="badge muted">—</span></td>
|
||||
<td class="cell-id mono">{{ e.latency_ms != null ? e.latency_ms + 'ms' : '—' }}</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<div class="pag">
|
||||
<span>共 {{ total }} 条</span><span class="spacer"></span>
|
||||
<button :disabled="page <= 1" @click="prev">上一页</button>
|
||||
<button class="on">{{ page }}</button>
|
||||
<button :disabled="page * pageSize >= total" @click="next">下一页</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<style scoped>
|
||||
input.date { background: var(--surface); border: 1px solid var(--border); color: var(--text); border-radius: var(--radius-sm); padding: 8px 10px; font-size: 13px; outline: none; }
|
||||
</style>
|
||||
@@ -1,72 +0,0 @@
|
||||
<script setup>
|
||||
import { ref, onMounted } from 'vue';
|
||||
import { api } from '../api/admin.js';
|
||||
import UserForm from '../components/UserForm.vue';
|
||||
|
||||
const emit = defineEmits(['unauthorized']);
|
||||
|
||||
const users = ref([]);
|
||||
const loading = ref(true);
|
||||
const formOpen = ref(false);
|
||||
const formMode = ref('create');
|
||||
const editing = ref(null);
|
||||
|
||||
const ROLE_LABEL = { admin: '管理员', researcher: '研究员', guest: '访客' };
|
||||
const UNIT_LABEL = { qa: '问答', kg: '知识图谱', extraction: '文献抽取', docgen: '文档生成', figures: '图片图库', private_docs: '私有资料', bioflow: '生信分析', paper: '论文撰写', legacy: '旧版接口' };
|
||||
|
||||
async function load() {
|
||||
loading.value = true;
|
||||
try {
|
||||
const r = await api.users();
|
||||
users.value = r.users;
|
||||
} catch (e) {
|
||||
if (e.status === 401) emit('unauthorized');
|
||||
} finally {
|
||||
loading.value = false;
|
||||
}
|
||||
}
|
||||
|
||||
function create() { formMode.value = 'create'; editing.value = null; formOpen.value = true; }
|
||||
function edit(u) { formMode.value = 'edit'; editing.value = u; formOpen.value = true; }
|
||||
async function del(u) {
|
||||
if (!confirm(`删除用户「${u.username}」？此操作不可撤销。`)) return;
|
||||
try { await api.deleteUser(u.id); load(); } catch (e) { alert(e.message || '删除失败'); }
|
||||
}
|
||||
function onSaved() { formOpen.value = false; load(); }
|
||||
|
||||
onMounted(load);
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div>
|
||||
<div class="toolbar">
|
||||
<div class="spacer"></div>
|
||||
<button class="btn btn-primary" @click="create">＋ 新建用户</button>
|
||||
</div>
|
||||
<div class="panel">
|
||||
<table>
|
||||
<thead><tr><th>用户</th><th>角色</th><th>授权范围</th><th>创建时间</th><th style="width: 120px"></th></tr></thead>
|
||||
<tbody>
|
||||
<tr v-if="loading"><td colspan="5" class="cell-id">加载中…</td></tr>
|
||||
<tr v-else-if="!users.length"><td colspan="5" class="cell-id">暂无用户</td></tr>
|
||||
<tr v-for="u in users" :key="u.id">
|
||||
<td>
|
||||
<div style="display: flex; align-items: center; gap: 10px">
|
||||
<div class="avatar">{{ u.username[0] }}</div>
|
||||
<div><div>{{ u.username }}</div><div class="cell-id" style="font-size: 12px">{{ u.id.slice(0, 8) }}</div></div>
|
||||
</div>
|
||||
</td>
|
||||
<td><span class="role" :class="u.role">{{ ROLE_LABEL[u.role] || u.role }}</span></td>
|
||||
<td>
|
||||
<span v-if="u.role === 'admin'" class="chip amber">全部权限</span>
|
||||
<template v-else><span v-for="a in u.authorizations" :key="a" class="chip">{{ UNIT_LABEL[a] || a }}</span></template>
|
||||
</td>
|
||||
<td class="cell-id mono">{{ (u.created_at || '').slice(0, 10) }}</td>
|
||||
<td><div class="row-actions"><span class="act" @click="edit(u)">编辑</span><span class="act danger" @click="del(u)">删除</span></div></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<UserForm :open="formOpen" :mode="formMode" :user="editing" @close="formOpen = false" @saved="onSaved" />
|
||||
</div>
|
||||
</template>
|
||||
@@ -1,15 +0,0 @@
|
||||
import { defineConfig } from 'vite';
|
||||
import vue from '@vitejs/plugin-vue';
|
||||
|
||||
export default defineConfig({
|
||||
base: '/admin/',
|
||||
plugins: [vue()],
|
||||
server: {
|
||||
host: '127.0.0.1',
|
||||
port: 5181,
|
||||
proxy: {
|
||||
'/api': { target: 'http://127.0.0.1:8090', changeOrigin: false },
|
||||
},
|
||||
},
|
||||
build: { outDir: 'dist', target: 'es2020' },
|
||||
});
|
||||
@@ -16,3 +16,70 @@ capabilities:
|
||||
endpoint: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi
|
||||
method: GET
|
||||
timeout: 8
|
||||
|
||||
# ---- 生信能力单元（S1）：两个 MCP 端点，各自独立装卸 ----
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 基因家族分析（路径规划/建任务/参数确认/文件获取与预处理/状态查询），自然语言触发
|
||||
session_default: true
|
||||
nl_triggers:
|
||||
- 基因家族
|
||||
- 家族成员
|
||||
- 亚家族
|
||||
- 系统发育
|
||||
- 建树
|
||||
- 进化树
|
||||
- 保守基序
|
||||
- motif
|
||||
- 结构域
|
||||
- 基因结构
|
||||
- 共线性
|
||||
- Ka/Ks
|
||||
- 选择压力
|
||||
- 组合证据
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/gene-family/
|
||||
prompt_pack:
|
||||
file: bioflow_gene_family_agent_rules.md
|
||||
|
||||
- name: bioflow-rnaseq
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 转录组分析（实验设计确认/样本配置/路径规划/建任务/差异表达/富集），自然语言触发
|
||||
session_default: true
|
||||
nl_triggers:
|
||||
- 转录组
|
||||
- RNA-seq
|
||||
- RNAseq
|
||||
- rnaseq
|
||||
- 差异表达
|
||||
- 表达定量
|
||||
- 富集
|
||||
- WGCNA
|
||||
- GSEA
|
||||
- 实验设计
|
||||
- fastq
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/rnaseq/
|
||||
prompt_pack:
|
||||
file: bioflow_rnaseq_agent_rules.md
|
||||
|
||||
# ---- 生信能力单元（S4）：第三个 MCP 端点，用户自定义扩展分析 ----
|
||||
- name: bioflow-extension
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 用户自定义扩展分析（创建/转换/试运行/校验/发布/运行），自然语言触发
|
||||
session_default: true
|
||||
nl_triggers:
|
||||
- 扩展
|
||||
- 拓展分析
|
||||
- 扩展分析
|
||||
- 自定义分析
|
||||
- 分析模板
|
||||
- 分析脚本
|
||||
- 分析工具
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/extension/
|
||||
prompt_pack:
|
||||
file: bioflow_extension_agent_rules.md
|
||||
|
||||
Vendored
+3
@@ -31,3 +31,6 @@ QA_V1_WEB_SEARCH_MAX_RESULTS=4
|
||||
QA_V1_WEB_SEARCH_TIMEOUT=8
|
||||
QA_V1_WEB_SEARCH_MAX_CHARS=1200
|
||||
QA_V1_ENABLE_SUPERPOWERS=1
|
||||
|
||||
# 生信对话式能力熔断开关（S1）：默认关（=0）。置 1 才启用 QA 前门 bioflow_agent。
|
||||
QA_V1_ENABLE_BIOFLOW_AGENT=0
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# 用户自定义扩展分析对话规范（bioflow_extension_agent_rules）
|
||||
|
||||
你是用户自定义扩展分析助手。引导用户把一段可运行的分析脚本/工具，通过「创建 → AI 转换 → 试运行 → 校验 → 确认 → 发布 → 运行」流程，做成 PlantAI 可自然语言触发的自定义扩展。
|
||||
|
||||
## 对话节奏（自然语言主导 + 卡片辅助）
|
||||
每次需要用户输入/选择时，遵循固定节奏：
|
||||
1. 先用自然语言说明并列表展示需要什么；
|
||||
2. 提示用户可用卡片快捷操作，也可直接打字（两条通道语义一致）；
|
||||
3. 用户回答后，用自然语言复述，确认无误再继续。
|
||||
|
||||
## 创建（create_extension）
|
||||
- 需要用户提供：显示名、运行命令、代码文件（可上传/粘贴文本内容）、运行时镜像、可选测试数据/金标。
|
||||
- 代码文件以「路径 + 文本内容」提交；引导用户说明脚本是做什么的、依赖什么。
|
||||
|
||||
## AI 转换与校验（build → trial_run → validate）
|
||||
- 创建后调用 build 做 AI 转换（原始命令 → command_template + 参数 + 依赖），失败时规则版兜底。
|
||||
- 转换结果先向用户说明（识别到的参数/依赖/输入角色），请用户确认参数名与默认值。
|
||||
- 试运行/校验的结果要转述给用户，失败时说明原因并引导修复（repair 循环）。
|
||||
|
||||
## 确认与发布（confirm_extension → publish_extension）
|
||||
- 确认前请用户确认触发词（nl_triggers）与参数——触发词决定用户以后说什么话能唤起这个扩展。
|
||||
- 发布后扩展才可被自然语言触发运行；发布是不可逆的对外可用状态，须经用户确认。
|
||||
|
||||
## 运行（run_extension）
|
||||
- 仅已发布扩展可运行；运行参数须在白名单内（参数 schema），未知参数明确追问。
|
||||
- 运行结果（run_result）转述给用户。
|
||||
|
||||
## 其它
|
||||
- 工具名/参数名/内部实现一律不外泄；
|
||||
- 状态不确定时主动查 extension_status 并向用户转述当前阶段与下一步；
|
||||
- 人工确认边界（确认触发词、发布）不可跳过、不可代用户拍板。
|
||||
@@ -0,0 +1,32 @@
|
||||
# 基因家族分析对话规范（bioflow_gene_family_agent_rules）
|
||||
|
||||
你是基因家族分析助手。用自然语言引导用户从需求到分析完成，全程对话式交互。
|
||||
|
||||
## 人工确认边界（不可跳过、不可代确认）
|
||||
- **subfamily_confirm（亚家族预测人工确认）是硬性人工边界**：系统预测的亚家族划分结果，
|
||||
必须停下来请用户查看并确认或修改。你绝不能代用户确认，也不能在用户未确认前继续下游分析。
|
||||
- 用户提出修改（如"把某成员从亚家族 A 移到亚家族 B"）时，先复述修改内容，再请用户做最终确认。
|
||||
|
||||
## 对话节奏（自然语言主导 + 卡片辅助）
|
||||
每次需要用户输入/选择时，遵循固定节奏：
|
||||
1. 先用自然语言说明并**列表展示**选项及含义；
|
||||
2. 提示用户可用卡片快捷操作，也可直接打字回复（两条通道语义一致）；
|
||||
3. 用户选择后，用自然语言**复述**其选择，确认无误再继续。
|
||||
|
||||
## 路径规划与建任务
|
||||
- 规划后列出"核心流程（必选）+ 可选分析"候选路径，说明每个可选分析的生物学意义；
|
||||
- **建任务前必须经用户确认阶段链**，收到确认后再创建任务，绝不擅自建任务。
|
||||
|
||||
## 文件获取与预处理
|
||||
- 源文件支持上传、http/https 链接、OSS 地址、局域网机器路径；
|
||||
- 文件到达后先探测格式，**格式异常（如 Excel/文本）时先向用户说明列结构并提问确认**
|
||||
（哪列是序列、是否去重、是否去表头等），再由模型整理成 fasta/gff 等标准格式；
|
||||
- 转换结果过格式验证，失败时说明问题并追问修正（循环不超过 3 次后建议人工整理）。
|
||||
|
||||
## 参数确认
|
||||
- 列"参数名（默认值，含义）"请用户确认或修改；未识别参数时明确指出并给出允许范围。
|
||||
|
||||
## 其它
|
||||
- 工具名、参数名、内部实现一律不外泄；
|
||||
- 任务运行中用户提问（流程/参数/生物学意义）随时回答，不打断任务；
|
||||
- 卡住或不确定时主动查询任务状态，向用户转述当前进度与下一步。
|
||||
@@ -0,0 +1,29 @@
|
||||
# 转录组分析对话规范（bioflow_rnaseq_agent_rules）
|
||||
|
||||
你是转录组分析助手。用自然语言引导用户从实验设计确认到差异表达/富集分析完成。
|
||||
|
||||
## 人工确认边界（不可跳过、不可代确认）
|
||||
- **实验设计确认（rnaseq_de 之前的必确认信息）是硬性边界**：开始分析前必须与用户确认——
|
||||
① 物种与参考基因组版本；② 每个样本的 FASTQ（上传/链接/局域网路径）及单端双端；
|
||||
③ 对照组/处理组各几个重复、有无时间点/批次；④ 建库是否链特异性。
|
||||
- 统计模型正确性依赖上述信息，未确认前不得创建差异表达任务，也不能代用户拍板基线/比较方向。
|
||||
|
||||
## 对话节奏（自然语言主导 + 卡片辅助）
|
||||
每次需要用户输入/选择时，遵循固定节奏：
|
||||
1. 先用自然语言说明并**列表展示**必确认信息与可选分析；
|
||||
2. 提示用户可用卡片快捷操作，也可直接打字回复（两条通道语义一致）；
|
||||
3. 用户回答后，用自然语言**复述**其样本设计与选择（如"干旱 3 vs 对照 3，双端，非链特异，对照为基线"），
|
||||
确认无误再继续。
|
||||
|
||||
## 样本配置与文件预处理
|
||||
- 用户用自然语言描述样本时，可解析生成 samples.yaml（样本名/对照处理组/重复/单双端）；
|
||||
- FASTQ 支持上传、http/https 链接、OSS 地址、局域网路径；核对成对完整；
|
||||
- 非 FASTQ 的输入文件（表格/文本等）先探测格式，**异常时先说明列结构并提问确认**，再整理为可用格式。
|
||||
|
||||
## 参数确认
|
||||
- 列"参数名（默认值，含义）"请用户确认或修改；未识别参数时明确指出并给出允许范围。
|
||||
|
||||
## 其它
|
||||
- 工具名、参数名、内部实现一律不外泄；
|
||||
- 任务运行中用户提问（流程/参数/实验设计意义）随时回答，不打断任务；
|
||||
- 卡住或不确定时主动查询任务状态，向用户转述当前进度与下一步（含样本质控 PCA、差异基因数目等结果转述）。
|
||||
@@ -13,6 +13,7 @@ from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from . import __version__
|
||||
from .bioflow_agent import try_bioflow_agent
|
||||
from .config import QAConfig
|
||||
from .pipeline import QAAnswerError, QAV1Pipeline
|
||||
|
||||
@@ -47,6 +48,9 @@ class QARequest(BaseModel):
|
||||
# ---- Capability Gateway（S1）：纯数据字段，无 validator（Cython 编译安全）----
|
||||
history: List[Dict[str, str]] = Field(default_factory=list)
|
||||
capabilities: Dict[str, Any] = Field(default_factory=dict)
|
||||
# ---- Bioflow 对话式编排（S2）：卡片回填协议 + session↔job 绑定 ----
|
||||
card_result: Optional[Dict[str, Any]] = None
|
||||
bioflow: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
@@ -150,6 +154,18 @@ async def chat(request: QARequest) -> Dict[str, Any]:
|
||||
detail={"error_code": "llm_unavailable", "answer": "", "message": "Final-answer LLM is not configured"},
|
||||
)
|
||||
try:
|
||||
# ---- Bioflow Agent 前门（S1）：命中则短路返回，否则走原 pipeline ----
|
||||
bioflow_result = await asyncio.to_thread(
|
||||
try_bioflow_agent,
|
||||
request.question,
|
||||
llm_client=pipeline.llm,
|
||||
history=request.history,
|
||||
capabilities=request.capabilities,
|
||||
card_result=request.card_result,
|
||||
bioflow=request.bioflow,
|
||||
)
|
||||
if bioflow_result is not None:
|
||||
return bioflow_result
|
||||
ablation = dict(request.ablation or {})
|
||||
ablation["use_final_llm"] = True
|
||||
return await asyncio.to_thread(
|
||||
@@ -216,6 +232,19 @@ async def chat_stream(request: QARequest) -> StreamingResponse:
|
||||
|
||||
def run() -> None:
|
||||
try:
|
||||
# ---- Bioflow Agent 前门（S1）：命中则短路，否则走原 pipeline ----
|
||||
bioflow_result = try_bioflow_agent(
|
||||
request.question,
|
||||
llm_client=pipeline.llm,
|
||||
history=request.history,
|
||||
capabilities=request.capabilities,
|
||||
card_result=request.card_result,
|
||||
bioflow=request.bioflow,
|
||||
progress_callback=emit,
|
||||
)
|
||||
if bioflow_result is not None:
|
||||
events.put(("complete", bioflow_result))
|
||||
return
|
||||
ablation = dict(request.ablation or {})
|
||||
ablation["use_final_llm"] = True
|
||||
result = pipeline.answer(
|
||||
|
||||
@@ -0,0 +1,684 @@
|
||||
from __future__ import annotations
|
||||
|
||||
"""bioflow_agent.py —— 生信 MCP 能力前门路由（S1 + S2）。
|
||||
|
||||
S1：三重门（会话开关/注册表/意图预筛）→ ToolRunner 工具循环 → 自然语言转述。
|
||||
S2：对话式编排闭环——card_result 卡片回填协议 + session↔job 绑定 + 多轮记忆
|
||||
+ 参数抽取闭环 + job_status 主动查询转述。
|
||||
|
||||
card_result 协议（前端卡片操作 → 结构化用户消息 → QA 流）：
|
||||
{card: path_selection|form_confirm|file_upload|archive_choice, payload: {...}}
|
||||
BioflowAgent 读取后经工具执行，再自然语言复述；卡片与自然语言两条通道在
|
||||
服务端语义一致（QA 对话状态机是唯一决策点）。
|
||||
|
||||
session↔job 绑定：BFF sessions.meta.bioflow = {active_job_id, chain}，经请求
|
||||
body["bioflow"] 注入（含 session_id）；job_status 工具是任务真相唯一权威。
|
||||
|
||||
金标路径零改动：任何不命中/异常都返回 None 回退原 pipeline。
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from .capabilities.manifests import CapabilityManifest
|
||||
from .capabilities.providers import PromptPackProvider, get_dispatcher
|
||||
from .capabilities.registry import get_registry
|
||||
from .tool_runner import ToolRunner, make_llm_complete
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
# 三个生信能力单元（与 bioflow_service 三个 MCP 端点一一对应）
|
||||
BIOFLOW_CAPABILITY_NAMES = ("bioflow-gene-family", "bioflow-rnaseq", "bioflow-extension")
|
||||
ENV_FUSE = "QA_V1_ENABLE_BIOFLOW_AGENT"
|
||||
MAX_ROUNDS = 5
|
||||
|
||||
# card_result 卡片类型（S2 前端卡片回填协议）
|
||||
CARD_PATH_SELECTION = "path_selection"
|
||||
CARD_FORM_CONFIRM = "form_confirm"
|
||||
CARD_FILE_UPLOAD = "file_upload"
|
||||
CARD_ARCHIVE_CHOICE = "archive_choice"
|
||||
CARD_EXTENSION_ACTION = "extension_action" # S4：扩展能力单元操作（build/trial/validate/confirm/publish/run）
|
||||
CARD_TYPES = (CARD_PATH_SELECTION, CARD_FORM_CONFIRM, CARD_FILE_UPLOAD, CARD_ARCHIVE_CHOICE, CARD_EXTENSION_ACTION)
|
||||
|
||||
# chain 名 → 能力名（建任务时选择哪条链的 create 工具）
|
||||
_CHAIN_TO_CAPABILITY = {
|
||||
"gene_family": "bioflow-gene-family",
|
||||
"rnaseq": "bioflow-rnaseq",
|
||||
}
|
||||
|
||||
|
||||
def _chain_to_capability(chain: str) -> Optional[str]:
|
||||
"""把 chain 名（gene_family/rnaseq）映射为能力名；未知返回 None。"""
|
||||
return _CHAIN_TO_CAPABILITY.get(str(chain or "").strip().lower())
|
||||
|
||||
_SYSTEM_BASE = (
|
||||
"你是 PlantAI 生信分析助手，用自然语言引导用户完成基因家族或转录组分析。\n"
|
||||
"你可以调用生信工具完成规划、建任务、确认参数、登记文件、查状态等操作。\n"
|
||||
"硬性规则：\n"
|
||||
"1. 回复只使用自然语言，绝不向用户泄露工具名、参数名或内部实现细节。\n"
|
||||
"2. 需要用户提供文件/参数/做选择时，先自然语言说明并列表展示选项与含义，再请用户回答，"
|
||||
"绝不擅自替用户做不可逆决策。\n"
|
||||
"3. 人工确认边界（基因家族 subfamily_confirm、转录组实验设计确认）不可跳过、不可代用户确认。\n"
|
||||
"4. 任务运行中可随时回答流程/参数/生物学意义类问题；卡住或不确定时主动查询任务状态，"
|
||||
"再向用户转述当前进度与下一步（等待文件、等待参数确认、失败可重跑等）。\n"
|
||||
"5. 文件格式异常时先向用户说明并提问确认，再由用户决定如何处理。\n"
|
||||
"6. 用户用自然语言给出参数（如「用 8 核、padj 用 0.01」）时，抽取为结构化参数后经确认步骤提交；"
|
||||
"只提交白名单内的参数名，未知参数明确指出并追问（最多 3 次）。\n"
|
||||
"7. 若用户消息是对上一轮卡片/提问的确认或选择，先复述用户的选择，确认无误再执行对应动作。\n"
|
||||
)
|
||||
|
||||
|
||||
def _env_enabled() -> bool:
|
||||
"""门 1：QA_V1_ENABLE_BIOFLOW_AGENT 熔断开关。默认关（未设置 = 关）。"""
|
||||
raw = os.environ.get(ENV_FUSE)
|
||||
return raw is not None and str(raw).strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _intent_match(question: str, triggers: List[str]) -> bool:
|
||||
"""门 3：意图预筛——问句是否命中任一触发词（大小写不敏感子串匹配）。"""
|
||||
if not triggers:
|
||||
return False
|
||||
q = (question or "").lower()
|
||||
return any(t and t.lower() in q for t in triggers)
|
||||
|
||||
|
||||
def _registered_capabilities(capabilities: Optional[Dict[str, Any]]) -> List[CapabilityManifest]:
|
||||
"""注册表 + 会话开关：返回所有已注册且会话开启的生信能力 manifest（不判意图）。"""
|
||||
try:
|
||||
registry = get_registry()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: registry 访问失败，回退原 pipeline: %s", exc)
|
||||
return []
|
||||
session_caps = capabilities or {}
|
||||
out: List[CapabilityManifest] = []
|
||||
for name in BIOFLOW_CAPABILITY_NAMES:
|
||||
try:
|
||||
spec = registry.get(name)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: registry.get(%s) 失败: %s", name, exc)
|
||||
continue
|
||||
if spec is None:
|
||||
continue # 未注册
|
||||
manifest = spec.manifest
|
||||
if not bool(session_caps.get(name, manifest.session_default)):
|
||||
continue # 会话开关关
|
||||
out.append(manifest)
|
||||
return out
|
||||
|
||||
|
||||
def _candidate_capabilities(
|
||||
question: str,
|
||||
capabilities: Optional[Dict[str, Any]],
|
||||
) -> List[CapabilityManifest]:
|
||||
"""门 2 + 门 3：注册表 + 会话开关 + 意图预筛，返回命中的候选能力清单。"""
|
||||
return [
|
||||
m for m in _registered_capabilities(capabilities)
|
||||
if _intent_match(question, m.nl_triggers)
|
||||
]
|
||||
|
||||
|
||||
def _build_tools(
|
||||
manifests: List[CapabilityManifest],
|
||||
) -> Tuple[List[Dict[str, Any]], Dict[str, CapabilityManifest]]:
|
||||
"""经 MCP list_tools 发现各能力工具，构造 LLM 工具表 + ToolRunner manifests。
|
||||
|
||||
返回 (tools, tool_manifests)：
|
||||
- tools: OpenAI function schema 列表（真实 server 工具名/描述/入参 schema）
|
||||
- tool_manifests: {真实工具名: manifest}，manifest.name=能力名（共享连接），
|
||||
mcp.tool=真实工具名
|
||||
|
||||
多能力同时命中时：专有工具名本就不同，各自保留；共享工具（两端点语义一致）
|
||||
按名去重，只保留首个能力的一份，避免 LLM tools 重名 + 同名覆盖。
|
||||
"""
|
||||
dispatcher = get_dispatcher()
|
||||
tools: List[Dict[str, Any]] = []
|
||||
tool_manifests: Dict[str, CapabilityManifest] = {}
|
||||
for cap_manifest in manifests:
|
||||
url = str((cap_manifest.config.get("mcp") or {}).get("url") or "").strip()
|
||||
if not url:
|
||||
LOGGER.warning("bioflow agent: 能力 %s 缺 mcp.url，跳过", cap_manifest.name)
|
||||
continue
|
||||
try:
|
||||
discovered = dispatcher.list_mcp_tools(cap_manifest)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: 能力 %s MCP 工具发现失败: %s", cap_manifest.name, exc)
|
||||
continue
|
||||
for item in discovered:
|
||||
tool_name = str(item.get("name") or "").strip()
|
||||
if not tool_name:
|
||||
continue
|
||||
if tool_name in tool_manifests:
|
||||
# 共享工具去重（见 S1 修复说明）
|
||||
continue
|
||||
input_schema = item.get("input_schema") or {}
|
||||
if not isinstance(input_schema, dict):
|
||||
input_schema = {"type": "object", "properties": {}}
|
||||
tools.append(
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": tool_name,
|
||||
"description": str(item.get("description") or "")[:1000],
|
||||
"parameters": input_schema,
|
||||
},
|
||||
}
|
||||
)
|
||||
tool_manifests[tool_name] = CapabilityManifest(
|
||||
name=cap_manifest.name,
|
||||
provider="mcp",
|
||||
permission="read_only",
|
||||
description=str(item.get("description") or ""),
|
||||
config={"mcp": {"url": url, "tool": tool_name}},
|
||||
)
|
||||
return tools, tool_manifests
|
||||
|
||||
|
||||
def _build_system(manifests: List[CapabilityManifest]) -> str:
|
||||
"""基础系统提示 + 各能力对话规范（prompt_pack 注入）。"""
|
||||
qa_root = Path(os.environ.get("GENE_FAMILY_ROOT", "/data/gene-family-mas-v2"))
|
||||
parts = [_SYSTEM_BASE]
|
||||
seen: set[str] = set()
|
||||
for manifest in manifests:
|
||||
file_name = str((manifest.config.get("prompt_pack") or {}).get("file") or "").strip()
|
||||
if not file_name or file_name in seen:
|
||||
continue
|
||||
seen.add(file_name)
|
||||
try:
|
||||
text = PromptPackProvider.load(manifest, qa_root)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: prompt_pack 加载失败 %s: %s", file_name, exc)
|
||||
text = ""
|
||||
if text:
|
||||
parts.append(text)
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# S2 工具执行 / 转述 / 参数抽取 辅助
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _execute_tool(
|
||||
tool_manifests: Dict[str, CapabilityManifest], tool_name: str, arguments: Dict[str, Any]
|
||||
) -> str:
|
||||
"""执行单个工具，返回结果文本（异常转文本，不炸主流程）。"""
|
||||
manifest = tool_manifests.get(tool_name)
|
||||
if manifest is None:
|
||||
return f"工具 {tool_name} 未注册"
|
||||
try:
|
||||
return get_dispatcher().execute(manifest, arguments)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return f"工具执行失败: {type(exc).__name__}: {exc}"
|
||||
|
||||
|
||||
def _paraphrase(llm_client: Any, user_text: str, system: Optional[str] = None) -> str:
|
||||
"""把工具执行结果转述为自然语言（LLM 优先，失败回退原文，绝不抛异常）。"""
|
||||
system = system or (
|
||||
"你是 PlantAI 生信分析助手。把下面的工具执行结果转述为对用户的自然语言回复，"
|
||||
"绝不提及工具名/参数名/内部实现；需要用户下一步操作时明确说明。"
|
||||
)
|
||||
try:
|
||||
text = llm_client.complete(system, user_text, temperature=0.2, max_tokens=600)
|
||||
if text and str(text).strip():
|
||||
return str(text).strip()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: paraphrase LLM 失败，回退原文: %s", exc)
|
||||
return str(user_text or "").strip()
|
||||
|
||||
|
||||
def _extract_params(
|
||||
llm_client: Any, user_text: str, form_defaults: Dict[str, Any]
|
||||
) -> Tuple[Dict[str, Any], List[str]]:
|
||||
"""LLM 抽取用户自然语言中的阶段确认参数 → (已识别参数, 未识别参数列表)。
|
||||
|
||||
白名单 = form_defaults 的键；未识别参数（不在白名单）单独返回供追问（≤3 次）。
|
||||
参照 extension_llm 的「LLM 出 JSON → 白名单净化」模式，LLM 失败/坏 JSON 返回空。
|
||||
"""
|
||||
whitelist = {str(k) for k in (form_defaults or {}).keys()}
|
||||
if not whitelist:
|
||||
return {}, []
|
||||
system = (
|
||||
"你是参数抽取助手。从用户的话中抽取阶段确认参数，只输出 JSON。\n"
|
||||
'输出形态：{"params": {"<参数名>": <值>}, "unknown": ["<未识别参数名>"]}。\n'
|
||||
f"允许的参数名（白名单）：{', '.join(sorted(whitelist))}。\n"
|
||||
"只抽取用户明确提到的参数；不在白名单里的参数名放进 unknown。"
|
||||
)
|
||||
try:
|
||||
raw = llm_client.complete_json(system, user_text, max_tokens=500)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: 参数抽取 LLM 失败: %s", exc)
|
||||
return {}, []
|
||||
if not isinstance(raw, dict):
|
||||
return {}, []
|
||||
params = raw.get("params") if isinstance(raw.get("params"), dict) else {}
|
||||
known = {str(k): v for k, v in params.items() if str(k) in whitelist}
|
||||
unknown_raw = raw.get("unknown") or []
|
||||
if isinstance(unknown_raw, str):
|
||||
unknown_raw = [unknown_raw]
|
||||
unknown = [str(x).strip() for x in unknown_raw if isinstance(x, str) and str(x).strip()]
|
||||
return known, unknown
|
||||
|
||||
|
||||
def _try_parse_json(text: str) -> Optional[Any]:
|
||||
if not text:
|
||||
return None
|
||||
try:
|
||||
return json.loads(text)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _extract_job_id(result_text: str) -> Optional[str]:
|
||||
"""从 create_job 工具结果文本里提取 job_id（结果形如 {"ok":true,"data":{"id":...}}）。"""
|
||||
data = _try_parse_json(result_text)
|
||||
if isinstance(data, dict):
|
||||
inner = data.get("data") if isinstance(data.get("data"), dict) else data
|
||||
jid = inner.get("id") or inner.get("job_id")
|
||||
if jid:
|
||||
return str(jid)
|
||||
return None
|
||||
|
||||
|
||||
def _query_job_status(job_id: str, tool_manifests: Dict[str, CapabilityManifest], llm_client: Any) -> str:
|
||||
"""主动查 job_status 并转述（任务真相唯一权威）。"""
|
||||
result = _execute_tool(tool_manifests, "job_status", {"job_id": job_id})
|
||||
return _paraphrase(llm_client, f"查询任务状态的结果：\n{result}")
|
||||
|
||||
|
||||
def _empty_audit() -> Dict[str, Any]:
|
||||
return {"rounds": 0, "tool_calls": 0, "used_json_fallback": False, "tool_names": [], "errors": []}
|
||||
|
||||
|
||||
def _make_payload(
|
||||
question: str,
|
||||
answer: str,
|
||||
*,
|
||||
audit: Optional[Dict[str, Any]] = None,
|
||||
capabilities: Optional[List[str]] = None,
|
||||
active_job_id: Optional[str] = None,
|
||||
chain: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
payload: Dict[str, Any] = {
|
||||
"success": True,
|
||||
"version": "bioflow-agent-s2",
|
||||
"question": question,
|
||||
"answer": answer,
|
||||
"sources": [],
|
||||
"source_count": 0,
|
||||
"citation_validation": {"valid": True, "cited_pmids": [], "unknown_pmids": []},
|
||||
"bioflow": {
|
||||
"handled": True,
|
||||
"audit": audit or _empty_audit(),
|
||||
"capabilities": capabilities or [],
|
||||
},
|
||||
}
|
||||
if active_job_id:
|
||||
payload["bioflow"]["active_job_id"] = str(active_job_id)
|
||||
if chain:
|
||||
payload["bioflow"]["chain"] = str(chain)
|
||||
return payload
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# card_result 回填处理（S2）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _card_path_selection(
|
||||
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""路径卡确认 → create_*_job → 复述；带 reuse_archive 时先复用存档的阶段链/参数再建任务。"""
|
||||
chain = str(payload.get("chain") or (bioflow or {}).get("chain") or "").strip()
|
||||
stages = payload.get("stages")
|
||||
reuse_name = str(payload.get("reuse_archive") or payload.get("reuse_name") or "").strip()
|
||||
has_stages = isinstance(stages, list) and len(stages) > 0
|
||||
if not has_stages and not reuse_name:
|
||||
# 既无显式阶段链也无存档复用来源 → 无法建任务，回退原 pipeline
|
||||
return None
|
||||
cap_name = _chain_to_capability(chain)
|
||||
if cap_name is None:
|
||||
# 无法确定能力（chain 空/未知）：不能默认走某条链，回退原 pipeline
|
||||
return None
|
||||
tool = "create_gene_family_job" if cap_name == "bioflow-gene-family" else "create_rnaseq_job"
|
||||
if tool not in tool_manifests:
|
||||
return None
|
||||
args: Dict[str, Any] = {
|
||||
"session_id": str((bioflow or {}).get("session_id") or ""),
|
||||
"stages": [str(s) for s in stages] if isinstance(stages, list) and stages else [],
|
||||
"title": str(payload.get("title") or ""),
|
||||
"auto_continue": bool(payload.get("auto_continue", False)),
|
||||
}
|
||||
tool_names: List[str] = []
|
||||
# 存档复用（FR-BIO-023）：先 reuse_archive 取阶段链/参数/archive_id，再以其发起
|
||||
if reuse_name and "reuse_archive" in tool_manifests:
|
||||
tool_names.append("reuse_archive") # 实际先执行 reuse_archive（audit 顺序与实际一致）
|
||||
reuse_text = _execute_tool(tool_manifests, "reuse_archive", {"name": reuse_name})
|
||||
arch = _try_parse_json(reuse_text)
|
||||
if isinstance(arch, dict) and arch.get("ok"):
|
||||
data = arch.get("data") if isinstance(arch.get("data"), dict) else {}
|
||||
if data.get("id"):
|
||||
args["archive_id"] = str(data["id"])
|
||||
if not args["stages"] and isinstance(data.get("stage_chain"), list):
|
||||
args["stages"] = [str(s) for s in data["stage_chain"]]
|
||||
if isinstance(data.get("params"), dict):
|
||||
merged = dict(data["params"])
|
||||
if isinstance(payload.get("params"), dict):
|
||||
merged.update(payload["params"])
|
||||
args["params"] = merged
|
||||
elif not args["stages"]:
|
||||
# 复用失败（非 ok）且无显式阶段链：不能空链建任务，回退原 pipeline
|
||||
return None
|
||||
tool_names.append(tool) # create_*_job 最后执行
|
||||
if "params" not in args and isinstance(payload.get("params"), dict):
|
||||
args["params"] = payload["params"]
|
||||
result = _execute_tool(tool_manifests, tool, args)
|
||||
job_id = _extract_job_id(result)
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户确认了分析路径并创建任务（{'基因家族' if cap_name == 'bioflow-gene-family' else '转录组'}链"
|
||||
f"{'，复用存档 ' + reuse_name if reuse_name else ''}）。工具结果：\n{result}",
|
||||
)
|
||||
return _make_payload(
|
||||
"确认路径并创建任务", answer,
|
||||
audit={"rounds": 1, "tool_calls": len(tool_names), "used_json_fallback": False, "tool_names": tool_names, "errors": []},
|
||||
capabilities=[cap_name] if cap_name else [],
|
||||
active_job_id=job_id,
|
||||
chain=chain or None,
|
||||
)
|
||||
|
||||
|
||||
def _card_form_confirm(
|
||||
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""参数卡确认 → confirm_stage（白名单校验）→ 复述；rerun:true 走 rerun_stage（改参数重跑）。"""
|
||||
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
|
||||
stage = str(payload.get("stage") or "").strip()
|
||||
params = payload.get("params") if isinstance(payload.get("params"), dict) else {}
|
||||
rerun = bool(payload.get("rerun", False))
|
||||
if not job_id or not stage:
|
||||
return None
|
||||
tool = "rerun_stage" if rerun else "confirm_stage"
|
||||
result = _execute_tool(tool_manifests, tool, {"job_id": job_id, "stage": stage, "params": params})
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户{'重跑并确认' if rerun else '确认'}了阶段参数（任务 {job_id} 的 {stage}）。工具结果：\n{result}",
|
||||
)
|
||||
return _make_payload(
|
||||
f"{'重跑' if rerun else '确认'}阶段参数：{stage}", answer,
|
||||
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": [tool], "errors": []},
|
||||
active_job_id=job_id,
|
||||
)
|
||||
|
||||
|
||||
def _card_file_upload(
|
||||
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""文件卡 → 网络源 fetch_source_file / 服务器·局域网路径 register_job_files → 复述。"""
|
||||
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
|
||||
raw_files = payload.get("paths") or payload.get("files") or []
|
||||
if isinstance(raw_files, str):
|
||||
raw_files = [raw_files]
|
||||
files = [str(f) for f in raw_files if str(f).strip()] if isinstance(raw_files, list) else []
|
||||
if not job_id or not files:
|
||||
return None
|
||||
remote = [f for f in files if f.startswith(("http://", "https://", "oss://"))]
|
||||
local = [f for f in files if f not in remote]
|
||||
results: List[str] = []
|
||||
tool_names: List[str] = []
|
||||
for src in remote:
|
||||
r = _execute_tool(tool_manifests, "fetch_source_file", {"source": src, "job_id": job_id})
|
||||
results.append(r)
|
||||
tool_names.append("fetch_source_file")
|
||||
if local:
|
||||
r = _execute_tool(tool_manifests, "register_job_files", {"job_id": job_id, "paths": local})
|
||||
results.append(r)
|
||||
tool_names.append("register_job_files")
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户提供了文件（{'远程 ' + str(len(remote)) + ' 个' if remote else ''}"
|
||||
f"{'本地 ' + str(len(local)) + ' 个' if local else ''}）。工具结果：\n" + "\n".join(results),
|
||||
)
|
||||
return _make_payload(
|
||||
"登记任务文件", answer,
|
||||
audit={"rounds": 1, "tool_calls": len(tool_names), "used_json_fallback": False, "tool_names": tool_names, "errors": []},
|
||||
active_job_id=job_id,
|
||||
)
|
||||
|
||||
|
||||
def _card_archive_choice(
|
||||
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""存档卡 → archive_job 工具执行实际存档（或跳过）→ 复述。"""
|
||||
job_id = str(payload.get("job_id") or (bioflow or {}).get("active_job_id") or "").strip()
|
||||
do_archive = bool(payload.get("archive", False))
|
||||
if not job_id:
|
||||
return None
|
||||
if do_archive:
|
||||
name = str(payload.get("name") or "")
|
||||
description = str(payload.get("description") or "")
|
||||
result = _execute_tool(
|
||||
tool_manifests, "archive_job",
|
||||
{"job_id": job_id, "name": name, "description": description},
|
||||
)
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户选择把任务 {job_id} 存档（名称：{name or '未命名'}）。工具结果：\n{result}",
|
||||
)
|
||||
return _make_payload(
|
||||
"存档选择", answer,
|
||||
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": ["archive_job"], "errors": []},
|
||||
active_job_id=job_id,
|
||||
)
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户选择不存档任务 {job_id}。请用自然语言告知用户：已跳过存档，"
|
||||
"本次分析结果仍可在报告/结果中查看。",
|
||||
)
|
||||
return _make_payload(
|
||||
"存档选择", answer,
|
||||
audit=_empty_audit(),
|
||||
active_job_id=job_id,
|
||||
)
|
||||
|
||||
|
||||
def _card_extension_action(
|
||||
payload: Dict[str, Any], llm_client: Any, bioflow: Dict[str, Any], tool_manifests: Dict[str, CapabilityManifest]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""扩展操作卡（S4）：action ∈ build|trial_run|validate|confirm|publish|run → 对应扩展工具。"""
|
||||
ext_id = str(payload.get("ext_id") or "").strip()
|
||||
action = str(payload.get("action") or "").strip()
|
||||
if not ext_id or not action:
|
||||
return None
|
||||
tool = {
|
||||
"build": "build_extension",
|
||||
"trial_run": "trial_run_extension",
|
||||
"validate": "validate_extension",
|
||||
"confirm": "confirm_extension",
|
||||
"publish": "publish_extension",
|
||||
"run": "run_extension",
|
||||
"repair": "repair_extension",
|
||||
}.get(action)
|
||||
if tool is None or tool not in tool_manifests:
|
||||
return None
|
||||
args: Dict[str, Any] = {"ext_id": ext_id}
|
||||
if action == "confirm":
|
||||
if isinstance(payload.get("nl_triggers"), list):
|
||||
args["nl_triggers"] = payload["nl_triggers"]
|
||||
if isinstance(payload.get("parameters"), list):
|
||||
args["parameters"] = payload["parameters"]
|
||||
if payload.get("runtime_image"):
|
||||
args["runtime_image"] = str(payload["runtime_image"])
|
||||
if action == "run":
|
||||
if isinstance(payload.get("params"), dict):
|
||||
args["params"] = payload["params"]
|
||||
if isinstance(payload.get("files"), list):
|
||||
args["files"] = payload["files"]
|
||||
result = _execute_tool(tool_manifests, tool, args)
|
||||
if action == "confirm" and "EXT_STATUS" in result:
|
||||
# 状态降级话术：confirm 要求 validated/trial_passed 前置状态，
|
||||
# 编辑修正若发生在校验前，明确提示先试运行/校验，不报 500。
|
||||
answer = (
|
||||
"该扩展当前状态还不能确认。请先完成 AI 转换与试运行、通过校验后，"
|
||||
"再确认参数与触发词。"
|
||||
)
|
||||
else:
|
||||
answer = _paraphrase(
|
||||
llm_client,
|
||||
f"用户对扩展 {ext_id} 执行了操作 {action}。工具结果：\n{result}",
|
||||
)
|
||||
return _make_payload(
|
||||
f"扩展操作：{action}", answer,
|
||||
audit={"rounds": 1, "tool_calls": 1, "used_json_fallback": False, "tool_names": [tool], "errors": []},
|
||||
)
|
||||
|
||||
|
||||
def _handle_card_result(
|
||||
card_result: Dict[str, Any],
|
||||
llm_client: Any,
|
||||
bioflow: Optional[Dict[str, Any]],
|
||||
tool_manifests: Dict[str, CapabilityManifest],
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
card = str(card_result.get("card") or "").strip()
|
||||
payload = card_result.get("payload") if isinstance(card_result.get("payload"), dict) else {}
|
||||
ctx = bioflow or {}
|
||||
if card == CARD_PATH_SELECTION:
|
||||
return _card_path_selection(payload, llm_client, ctx, tool_manifests)
|
||||
if card == CARD_FORM_CONFIRM:
|
||||
return _card_form_confirm(payload, llm_client, ctx, tool_manifests)
|
||||
if card == CARD_FILE_UPLOAD:
|
||||
return _card_file_upload(payload, llm_client, ctx, tool_manifests)
|
||||
if card == CARD_ARCHIVE_CHOICE:
|
||||
return _card_archive_choice(payload, llm_client, ctx, tool_manifests)
|
||||
if card == CARD_EXTENSION_ACTION:
|
||||
return _card_extension_action(payload, llm_client, ctx, tool_manifests)
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 前门路由
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def try_bioflow_agent(
|
||||
question: str,
|
||||
*,
|
||||
llm_client: Any,
|
||||
history: Optional[List[Dict[str, str]]] = None,
|
||||
capabilities: Optional[Dict[str, Any]] = None,
|
||||
card_result: Optional[Dict[str, Any]] = None,
|
||||
bioflow: Optional[Dict[str, Any]] = None,
|
||||
progress_callback: Optional[Any] = None,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""前门路由入口。命中生信能力则返回答案 payload，否则返回 None（走原 pipeline）。
|
||||
|
||||
- card_result：非空时走卡片回填协议（直接路由到对应工具，不做意图预筛）。
|
||||
- 任何异常都不上抛——记日志后返回 None，保证金标路径永不因本模块失败。
|
||||
"""
|
||||
try:
|
||||
return _try_bioflow_agent_impl(
|
||||
question,
|
||||
llm_client=llm_client,
|
||||
history=history,
|
||||
capabilities=capabilities,
|
||||
card_result=card_result,
|
||||
bioflow=bioflow,
|
||||
progress_callback=progress_callback,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
LOGGER.warning("bioflow agent: 执行失败，回退原 pipeline: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
def _try_bioflow_agent_impl(
|
||||
question: str,
|
||||
*,
|
||||
llm_client: Any,
|
||||
history: Optional[List[Dict[str, str]]],
|
||||
capabilities: Optional[Dict[str, Any]],
|
||||
card_result: Optional[Dict[str, Any]],
|
||||
bioflow: Optional[Dict[str, Any]],
|
||||
progress_callback: Optional[Any],
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
# 门 1：会话开关（熔断）
|
||||
if not _env_enabled():
|
||||
return None
|
||||
|
||||
# card_result 卡片操作优先：用户已在卡片上明确选择，不做意图预筛，直接路由
|
||||
if isinstance(card_result, dict) and str(card_result.get("card") or "").strip() in CARD_TYPES:
|
||||
return _route_card_result(card_result, llm_client, capabilities, bioflow, progress_callback)
|
||||
|
||||
# NL 路径：门 2 + 门 3
|
||||
manifests = _candidate_capabilities(question, capabilities)
|
||||
if not manifests:
|
||||
return None
|
||||
|
||||
tools, tool_manifests = _build_tools(manifests)
|
||||
if not tools:
|
||||
LOGGER.warning("bioflow agent: 意图命中但 MCP 工具发现为空，回退原 pipeline")
|
||||
return None
|
||||
|
||||
if progress_callback is not None:
|
||||
try:
|
||||
progress_callback(
|
||||
{
|
||||
"stage": "bioflow_agent_started",
|
||||
"capabilities": [m.name for m in manifests],
|
||||
"tool_count": len(tools),
|
||||
}
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
system = _build_system(manifests)
|
||||
runner = ToolRunner(make_llm_complete(llm_client), max_rounds=MAX_ROUNDS)
|
||||
content, audit = runner.run(system, question, tools, tool_manifests, history=history)
|
||||
|
||||
answer = (content or "").strip()
|
||||
if not answer:
|
||||
return None
|
||||
|
||||
return _make_payload(
|
||||
question,
|
||||
answer,
|
||||
audit=audit,
|
||||
capabilities=[m.name for m in manifests],
|
||||
active_job_id=(bioflow or {}).get("active_job_id"),
|
||||
chain=(bioflow or {}).get("chain"),
|
||||
)
|
||||
|
||||
|
||||
def _route_card_result(
|
||||
card_result: Dict[str, Any],
|
||||
llm_client: Any,
|
||||
capabilities: Optional[Dict[str, Any]],
|
||||
bioflow: Optional[Dict[str, Any]],
|
||||
progress_callback: Optional[Any],
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""卡片操作路由：确定相关能力 → 构建工具 → 分发到对应 card handler。"""
|
||||
card = str(card_result.get("card") or "").strip()
|
||||
payload = card_result.get("payload") if isinstance(card_result.get("payload"), dict) else {}
|
||||
chain = str(payload.get("chain") or (bioflow or {}).get("chain") or "").strip()
|
||||
if card == CARD_EXTENSION_ACTION:
|
||||
# 扩展操作只依赖 bioflow-extension 能力，避免对三个能力各做一次 list_tools
|
||||
cap_name = "bioflow-extension"
|
||||
else:
|
||||
cap_name = _chain_to_capability(chain)
|
||||
registered = _registered_capabilities(capabilities)
|
||||
if cap_name:
|
||||
manifests = [m for m in registered if m.name == cap_name]
|
||||
else:
|
||||
manifests = registered
|
||||
if not manifests:
|
||||
return None
|
||||
tools, tool_manifests = _build_tools(manifests)
|
||||
if not tools:
|
||||
LOGGER.warning("bioflow agent: card_result 但 MCP 工具发现为空，回退原 pipeline")
|
||||
return None
|
||||
if progress_callback is not None:
|
||||
try:
|
||||
progress_callback({"stage": "bioflow_card", "card": card_result.get("card")})
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return _handle_card_result(card_result, llm_client, bioflow, tool_manifests)
|
||||
@@ -128,6 +128,8 @@ class InstallManager:
|
||||
"allowed_llm_phases": manifest.allowed_llm_phases,
|
||||
"session_default": manifest.session_default,
|
||||
}
|
||||
if manifest.nl_triggers:
|
||||
raw["nl_triggers"] = manifest.nl_triggers
|
||||
if manifest.schema:
|
||||
raw["schema"] = manifest.schema
|
||||
for section in ("http", "mcp", "python", "prompt_pack"):
|
||||
|
||||
@@ -37,6 +37,7 @@ class CapabilityManifest:
|
||||
session_default: bool = False
|
||||
schema: Dict[str, Any] = field(default_factory=dict)
|
||||
config: Dict[str, Any] = field(default_factory=dict) # provider 专属段（http:/mcp:/python:/prompt_pack: 归并）
|
||||
nl_triggers: List[str] = field(default_factory=list) # 意图预筛触发词（S1，生信能力路由用）
|
||||
|
||||
def to_tool_spec(self) -> Dict[str, Any]:
|
||||
"""转 OpenAI function schema（tools 参数形态）。"""
|
||||
@@ -110,6 +111,19 @@ def _parse_entry(raw: Any, index: int, load_warnings: List[str]) -> CapabilityMa
|
||||
if isinstance(raw.get(section), dict):
|
||||
config[section] = raw[section]
|
||||
config["session_default"] = bool(raw.get("session_default", False))
|
||||
# nl_triggers（S1）：意图预筛触发词，只收字符串、去空、去重、限长
|
||||
nl_triggers_raw = raw.get("nl_triggers") or []
|
||||
if isinstance(nl_triggers_raw, str):
|
||||
nl_triggers_raw = [nl_triggers_raw]
|
||||
nl_triggers: List[str] = []
|
||||
if isinstance(nl_triggers_raw, list):
|
||||
for trigger in nl_triggers_raw:
|
||||
if not isinstance(trigger, str):
|
||||
continue # 触发词只收字符串，非字符串（数字等）丢弃
|
||||
cleaned = _clean_text(trigger, 64)
|
||||
if cleaned and cleaned not in nl_triggers:
|
||||
nl_triggers.append(cleaned)
|
||||
nl_triggers = nl_triggers[:64]
|
||||
manifest = CapabilityManifest(
|
||||
name=name,
|
||||
provider=provider,
|
||||
@@ -119,6 +133,7 @@ def _parse_entry(raw: Any, index: int, load_warnings: List[str]) -> CapabilityMa
|
||||
session_default=bool(raw.get("session_default", False)),
|
||||
schema=schema,
|
||||
config=config,
|
||||
nl_triggers=nl_triggers,
|
||||
)
|
||||
# 大小上限：单条目序列化后 ≤8KB
|
||||
import json
|
||||
|
||||
@@ -130,6 +130,7 @@ class _McpConnection:
|
||||
self._thread: Optional[threading.Thread] = None
|
||||
self._session: Any = None
|
||||
self._server_tools: List[str] = []
|
||||
self._server_tool_schemas: List[Dict[str, Any]] = []
|
||||
self._started_at = 0.0
|
||||
self._last_used = 0.0
|
||||
self._restarts = 0
|
||||
@@ -200,6 +201,18 @@ class _McpConnection:
|
||||
raise
|
||||
self._ctx = ctx
|
||||
self._server_tools = [t.name for t in tools_result.tools]
|
||||
self._server_tool_schemas = [
|
||||
{
|
||||
"name": t.name,
|
||||
"description": (getattr(t, "description", None) or ""),
|
||||
"input_schema": (
|
||||
getattr(t, "inputSchema", None)
|
||||
or getattr(t, "input_schema", None)
|
||||
or {}
|
||||
),
|
||||
}
|
||||
for t in tools_result.tools
|
||||
]
|
||||
self._session = session
|
||||
self._started_at = time.time()
|
||||
|
||||
@@ -225,6 +238,7 @@ class _McpConnection:
|
||||
ctx = getattr(self, "_ctx", None)
|
||||
self._ctx = None
|
||||
self._server_tools = []
|
||||
self._server_tool_schemas = []
|
||||
if session is not None:
|
||||
try:
|
||||
await session.__aexit__(None, None, None)
|
||||
@@ -274,6 +288,14 @@ class _McpConnection:
|
||||
return []
|
||||
return list(self._server_tools)
|
||||
|
||||
def list_tool_schemas(self) -> List[Dict[str, Any]]:
|
||||
"""返回 MCP server 全量工具 schema（name/description/input_schema）。连接失败返回空。"""
|
||||
try:
|
||||
self._ensure_session()
|
||||
except ToolExecutionError:
|
||||
return []
|
||||
return list(self._server_tool_schemas)
|
||||
|
||||
def call(self, tool_name: str, arguments: Dict[str, Any]) -> str:
|
||||
self._ensure_session()
|
||||
try:
|
||||
@@ -337,6 +359,11 @@ class McpToolProvider:
|
||||
return tools + [f"<缺失声明工具 {tool_name}>"]
|
||||
return tools
|
||||
|
||||
def list_tool_schemas(self, manifest: CapabilityManifest) -> List[Dict[str, Any]]:
|
||||
"""返回该能力 MCP server 的全量工具 schema（供前门 agent 构建 LLM 工具表）。"""
|
||||
conn = self._connection(manifest)
|
||||
return conn.list_tool_schemas()
|
||||
|
||||
def reap_idle(self) -> int:
|
||||
reaped = 0
|
||||
with self._lock:
|
||||
@@ -448,6 +475,10 @@ class ToolDispatcher:
|
||||
return PythonToolProvider.execute(manifest, arguments)
|
||||
raise ToolExecutionError(f"工具 {manifest.name} provider={provider} 不支持调用")
|
||||
|
||||
def list_mcp_tools(self, manifest: CapabilityManifest) -> List[Dict[str, Any]]:
|
||||
"""MCP 能力全量工具 schema（bioflow_agent 构建 LLM 工具表用）。"""
|
||||
return self._mcp.list_tool_schemas(manifest)
|
||||
|
||||
def reap_idle(self) -> int:
|
||||
return self._mcp.reap_idle()
|
||||
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
"""test_bioflow_agent.py —— bioflow_agent 前门三重门 + nl_triggers 解析单测。
|
||||
|
||||
覆盖 S1 验收要点：
|
||||
- 熔断开关（QA_V1_ENABLE_BIOFLOW_AGENT）默认关 → 返回 None（走原 pipeline）
|
||||
- 意图预筛（nl_triggers）各自评估，未命中返回 None
|
||||
- 注册表未注册 / 会话开关关 → 返回 None
|
||||
- 命中后短路返回自然语言 payload（bioflow.handled=True）
|
||||
- 只装一个能力时，另一个能力的触发语不响应（隔离）
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.manifests import parse_capabilities_text # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
|
||||
|
||||
BIOFLOW_YAML = """
|
||||
capabilities:
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 基因家族分析
|
||||
session_default: true
|
||||
nl_triggers: [基因家族, 家族成员, 亚家族, 建树]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/gene-family/
|
||||
prompt_pack:
|
||||
file: bioflow_gene_family_agent_rules.md
|
||||
- name: bioflow-rnaseq
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 转录组分析
|
||||
session_default: true
|
||||
nl_triggers: [转录组, RNA-seq, 差异表达, 富集]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/rnaseq/
|
||||
prompt_pack:
|
||||
file: bioflow_rnaseq_agent_rules.md
|
||||
"""
|
||||
|
||||
|
||||
def _make_registry(tmp_path: Path) -> CapabilityRegistry:
|
||||
path = tmp_path / "capabilities.yaml"
|
||||
path.write_text(BIOFLOW_YAML, encoding="utf-8")
|
||||
registry = CapabilityRegistry(path)
|
||||
registry.load()
|
||||
return registry
|
||||
|
||||
|
||||
class FakeLLMClient:
|
||||
def __init__(self, content: str = "已为您规划分析路径。") -> None:
|
||||
self.content = content
|
||||
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kwargs):
|
||||
return {"content": self.content, "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
class TestNlTriggers:
|
||||
def test_parse_nl_triggers(self):
|
||||
manifests, warnings = parse_capabilities_text(BIOFLOW_YAML)
|
||||
assert warnings == []
|
||||
gf = next(m for m in manifests if m.name == "bioflow-gene-family")
|
||||
assert gf.nl_triggers == ["基因家族", "家族成员", "亚家族", "建树"]
|
||||
assert gf.config["mcp"]["url"].endswith("/mcp/gene-family/")
|
||||
assert gf.config["prompt_pack"]["file"] == "bioflow_gene_family_agent_rules.md"
|
||||
|
||||
def test_nl_triggers_dedupe_and_clean(self):
|
||||
text = """
|
||||
capabilities:
|
||||
- name: t
|
||||
provider: mcp
|
||||
nl_triggers: [基因家族, 基因家族, "", " 亚家族 ", 123]
|
||||
"""
|
||||
manifests, _ = parse_capabilities_text(text)
|
||||
assert manifests[0].nl_triggers == ["基因家族", "亚家族"]
|
||||
|
||||
|
||||
class TestIntentMatch:
|
||||
def test_match(self):
|
||||
assert ba._intent_match("帮我做水稻SPL基因家族分析", ["基因家族"]) is True
|
||||
assert ba._intent_match("转录组差异表达分析", ["转录组", "RNA-seq"]) is True
|
||||
assert ba._intent_match("什么是光合作用", ["基因家族", "转录组"]) is False
|
||||
|
||||
|
||||
class TestGateLogic:
|
||||
def test_fuse_off_returns_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict("os.environ", {}, clear=False):
|
||||
ba.os.environ.pop(ba.ENV_FUSE, None)
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_no_intent_match_returns_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"光合作用的电子传递链是什么", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_not_registered_returns_none(self, tmp_path):
|
||||
empty = CapabilityRegistry(tmp_path / "empty.yaml")
|
||||
empty.load()
|
||||
with mock.patch.object(ba, "get_registry", return_value=empty):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_session_switch_off_returns_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析",
|
||||
llm_client=FakeLLMClient(),
|
||||
capabilities={"bioflow-gene-family": False},
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_isolation_only_installed_responds(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
caps = ba._candidate_capabilities("帮我做基因家族分析", None)
|
||||
assert [m.name for m in caps] == ["bioflow-gene-family"]
|
||||
caps = ba._candidate_capabilities("转录组差异表达", None)
|
||||
assert [m.name for m in caps] == ["bioflow-rnaseq"]
|
||||
|
||||
def test_full_flow_short_circuits(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
fake_tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "plan_gene_family",
|
||||
"description": "规划基因家族路径",
|
||||
"parameters": {"type": "object", "properties": {}},
|
||||
},
|
||||
}
|
||||
]
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {"plan_gene_family": mock.ANY})):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is not None
|
||||
assert result["success"] is True
|
||||
assert result["answer"] == "已为您规划分析路径。"
|
||||
assert result["bioflow"]["handled"] is True
|
||||
assert result["bioflow"]["capabilities"] == ["bioflow-gene-family"]
|
||||
|
||||
def test_exception_falls_back_to_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.object(ba, "_candidate_capabilities", side_effect=RuntimeError("boom")):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
@@ -0,0 +1,255 @@
|
||||
"""test_bioflow_agent_s2.py —— S2 对话式编排闭环单测。
|
||||
|
||||
覆盖：card_result 回填协议（四类卡片）、参数抽取（白名单）、多轮记忆注入、
|
||||
session↔job 绑定上下文（active_job_id/chain）、job_status 主动查询。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest # noqa: E402
|
||||
from rag.qa_v1_1_73.tool_runner import ToolRunner # noqa: E402
|
||||
|
||||
|
||||
def _manifest(name="bioflow-gene-family", url="http://x/mcp/gene-family/"):
|
||||
return CapabilityManifest(name=name, provider="mcp", config={"mcp": {"url": url, "tool": name}})
|
||||
|
||||
|
||||
def _tool_manifests():
|
||||
"""覆盖四类卡片用到的工具 + job_status。"""
|
||||
names = ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
|
||||
"register_job_files", "fetch_source_file", "job_status")
|
||||
out = {}
|
||||
for t in names:
|
||||
out[t] = CapabilityManifest(
|
||||
name="bioflow-gene-family", provider="mcp",
|
||||
config={"mcp": {"url": "http://x/mcp/gene-family/", "tool": t}},
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
class FakeDispatcher:
|
||||
def __init__(self, results=None):
|
||||
self.results = results or {}
|
||||
self.calls = []
|
||||
|
||||
def execute(self, manifest, arguments):
|
||||
tool = manifest.config.get("mcp", {}).get("tool")
|
||||
self.calls.append((tool, arguments))
|
||||
return self.results.get(tool, '{"ok": true, "data": {"id": "job-1"}}')
|
||||
|
||||
|
||||
class FakeLLM:
|
||||
def __init__(self, complete_text="已收到，已为您处理。", json_payload=None):
|
||||
self.complete_text = complete_text
|
||||
self.json_payload = json_payload or {}
|
||||
|
||||
def complete(self, system, user, **kw):
|
||||
return self.complete_text
|
||||
|
||||
def complete_json(self, system, user, **kw):
|
||||
return self.json_payload
|
||||
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 参数抽取
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestExtractParams:
|
||||
def test_extract_known_and_unknown(self):
|
||||
llm = FakeLLM(json_payload={"params": {"threads": 8, "padj": 0.01}, "unknown": ["bogus"]})
|
||||
form_defaults = {"threads": {"default": 4}, "padj": {"default": 0.05}}
|
||||
known, unknown = ba._extract_params(llm, "用 8 核，padj 用 0.01", form_defaults)
|
||||
assert known == {"threads": 8, "padj": 0.01}
|
||||
assert unknown == ["bogus"]
|
||||
|
||||
def test_non_whitelist_param_filtered(self):
|
||||
llm = FakeLLM(json_payload={"params": {"evil": 1, "threads": 4}, "unknown": []})
|
||||
form_defaults = {"threads": {"default": 4}}
|
||||
known, unknown = ba._extract_params(llm, "x", form_defaults)
|
||||
assert known == {"threads": 4}
|
||||
|
||||
def test_bad_json_returns_empty(self):
|
||||
llm = FakeLLM(json_payload="not-a-dict")
|
||||
form_defaults = {"threads": {}}
|
||||
known, unknown = ba._extract_params(llm, "x", form_defaults)
|
||||
assert known == {} and unknown == []
|
||||
|
||||
def test_empty_whitelist(self):
|
||||
llm = FakeLLM()
|
||||
known, unknown = ba._extract_params(llm, "x", {})
|
||||
assert known == {} and unknown == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# card_result 回填协议
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestCardResult:
|
||||
def test_path_selection_creates_job_and_extracts_job_id(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
llm = FakeLLM()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "gene_family", "stages": ["identify", "tree"], "title": "水稻SPL"}},
|
||||
llm, {"session_id": "s1"}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert out["bioflow"]["active_job_id"] == "job-1"
|
||||
assert out["bioflow"]["chain"] == "gene_family"
|
||||
assert dispatcher.calls[0][0] == "create_gene_family_job"
|
||||
assert dispatcher.calls[0][1]["stages"] == ["identify", "tree"]
|
||||
assert dispatcher.calls[0][1]["session_id"] == "s1"
|
||||
|
||||
def test_path_selection_missing_stages_returns_none(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection", "payload": {"chain": "gene_family"}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_form_confirm_calls_confirm_stage(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "form_confirm",
|
||||
"payload": {"job_id": "job-9", "stage": "identify", "params": {"evalue": 1e-5}}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "confirm_stage"
|
||||
assert dispatcher.calls[0][1] == {"job_id": "job-9", "stage": "identify", "params": {"evalue": 1e-5}}
|
||||
|
||||
def test_form_confirm_uses_active_job_id_fallback(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
ba._handle_card_result(
|
||||
{"card": "form_confirm", "payload": {"stage": "identify", "params": {}}},
|
||||
FakeLLM(), {"active_job_id": "job-7"}, _tool_manifests(),
|
||||
)
|
||||
assert dispatcher.calls[0][1]["job_id"] == "job-7"
|
||||
|
||||
def test_file_upload_splits_remote_and_local(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "file_upload",
|
||||
"payload": {"job_id": "j1", "paths": ["https://x/a.fa", "/data/b.fa"]}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
tools = [c[0] for c in dispatcher.calls]
|
||||
assert "fetch_source_file" in tools and "register_job_files" in tools
|
||||
|
||||
def test_archive_choice_paraphrases(self):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "archive_choice", "payload": {"job_id": "j1", "archive": True}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert out["answer"] == "已收到，已为您处理。"
|
||||
|
||||
def test_unknown_card_returns_none(self):
|
||||
out = ba._handle_card_result({"card": "bogus", "payload": {}}, FakeLLM(), {}, _tool_manifests())
|
||||
assert out is None
|
||||
|
||||
def test_path_selection_reuse_failure_no_stages_returns_none(self):
|
||||
"""bug ②：复用失败且无显式 stages → 返回 None（不空链建任务）。"""
|
||||
dispatcher = FakeDispatcher({"reuse_archive": '{"ok": false, "code": "ARCHIVE_NOT_FOUND"}'})
|
||||
tm = _tool_manifests()
|
||||
tm["reuse_archive"] = CapabilityManifest(
|
||||
name="bioflow-gene-family", provider="mcp",
|
||||
config={"mcp": {"url": "http://x", "tool": "reuse_archive"}},
|
||||
)
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "gene_family", "reuse_archive": "不存在的存档"}},
|
||||
FakeLLM(), {}, tm,
|
||||
)
|
||||
assert out is None
|
||||
# 只执行了 reuse_archive，未调用 create_gene_family_job
|
||||
assert [c[0] for c in dispatcher.calls] == ["reuse_archive"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 前门路由（card_result / bioflow 上下文）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestRouteCardResult:
|
||||
def test_try_bioflow_agent_card_result_short_circuits(self):
|
||||
registry_manifest = _manifest()
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[registry_manifest]):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([{"type": "function", "function": {"name": "confirm_stage", "description": "", "parameters": {"type": "object", "properties": {}}}}], {"confirm_stage": registry_manifest})):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba.try_bioflow_agent(
|
||||
"确认", llm_client=FakeLLM(),
|
||||
card_result={"card": "form_confirm", "payload": {"job_id": "j1", "stage": "s", "params": {}}},
|
||||
)
|
||||
assert out is not None
|
||||
assert out["bioflow"]["handled"] is True
|
||||
|
||||
def test_try_bioflow_agent_fuse_off_ignores_card_result(self):
|
||||
with mock.patch.dict("os.environ", {}, clear=False):
|
||||
ba.os.environ.pop(ba.ENV_FUSE, None)
|
||||
out = ba.try_bioflow_agent(
|
||||
"确认", llm_client=FakeLLM(),
|
||||
card_result={"card": "form_confirm", "payload": {}},
|
||||
)
|
||||
assert out is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 多轮记忆注入（ToolRunner.run history）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestMultiTurnHistory:
|
||||
def test_history_injected_into_messages(self):
|
||||
class RecordingLLM:
|
||||
def __init__(self):
|
||||
self.captured_messages = None
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
self.captured_messages = list(messages)
|
||||
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
from rag.qa_v1_1_73.tool_runner import make_llm_complete
|
||||
|
||||
llm = RecordingLLM()
|
||||
runner = ToolRunner(make_llm_complete(llm), max_rounds=1)
|
||||
history = [{"role": "user", "content": "上一轮问题"}, {"role": "assistant", "content": "上一轮答案"}]
|
||||
runner.run("sys", "当前问题", [], {}, history=history)
|
||||
msgs = llm.captured_messages
|
||||
assert msgs[0] == {"role": "user", "content": "上一轮问题"}
|
||||
assert msgs[1] == {"role": "assistant", "content": "上一轮答案"}
|
||||
assert msgs[-1] == {"role": "user", "content": "当前问题"}
|
||||
|
||||
def test_history_filters_bad_roles(self):
|
||||
class RecordingLLM:
|
||||
def __init__(self):
|
||||
self.captured_messages = None
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
self.captured_messages = list(messages)
|
||||
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
from rag.qa_v1_1_73.tool_runner import make_llm_complete
|
||||
|
||||
llm = RecordingLLM()
|
||||
runner = ToolRunner(make_llm_complete(llm), max_rounds=1)
|
||||
history = [{"role": "system", "content": "evil"}, {"role": "user", "content": ""}, {"role": "user", "content": "ok"}]
|
||||
runner.run("sys", "q", [], {}, history=history)
|
||||
roles = [m["role"] for m in llm.captured_messages]
|
||||
assert "system" not in roles
|
||||
assert llm.captured_messages[-1] == {"role": "user", "content": "q"}
|
||||
@@ -0,0 +1,511 @@
|
||||
"""test_bioflow_agent_s2_supplement.py —— S2 覆盖清单补强测试。
|
||||
|
||||
补强 dev 的 test_bioflow_agent_s2.py 未覆盖的分支：
|
||||
- _chain_to_capability / _extract_job_id / _execute_tool / _paraphrase / _query_job_status
|
||||
- card 四类 handler 缺必需字段回退 None、files 键、archive=False、非 dict payload
|
||||
- _route_card_result：chain 过滤 / 未注册能力→None / tools 空→None
|
||||
- NL 路径注入 bioflow 上下文（active_job_id/chain）
|
||||
- system prompt 硬规则在位（追问 ≤3 / 不可代确认 / 工具名不外泄）
|
||||
- ToolRunner.run(history) 边界（non-dict 条目 / history=None / 空内容剔除）
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest # noqa: E402
|
||||
from rag.qa_v1_1_73.tool_runner import ToolRunner, make_llm_complete # noqa: E402
|
||||
|
||||
|
||||
def _manifest(name="bioflow-gene-family", url="http://x/mcp/gene-family/"):
|
||||
return CapabilityManifest(name=name, provider="mcp", config={"mcp": {"url": url, "tool": name}})
|
||||
|
||||
|
||||
def _tool_manifests():
|
||||
names = ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
|
||||
"register_job_files", "fetch_source_file", "job_status",
|
||||
"rerun_stage", "reuse_archive", "archive_job")
|
||||
out = {}
|
||||
for t in names:
|
||||
out[t] = CapabilityManifest(
|
||||
name="bioflow-gene-family", provider="mcp",
|
||||
config={"mcp": {"url": "http://x/mcp/gene-family/", "tool": t}},
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
class FakeLLM:
|
||||
def __init__(self, complete_text="已收到。", json_payload=None, complete_error=None):
|
||||
self.complete_text = complete_text
|
||||
self.json_payload = json_payload or {}
|
||||
self.complete_error = complete_error
|
||||
|
||||
def complete(self, system, user, **kw):
|
||||
if self.complete_error:
|
||||
raise self.complete_error
|
||||
return self.complete_text
|
||||
|
||||
def complete_json(self, system, user, **kw):
|
||||
return self.json_payload
|
||||
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
class FakeDispatcher:
|
||||
def __init__(self, results=None):
|
||||
self.results = results or {}
|
||||
self.calls = []
|
||||
|
||||
def execute(self, manifest, arguments):
|
||||
tool = manifest.config.get("mcp", {}).get("tool")
|
||||
self.calls.append((tool, arguments))
|
||||
return self.results.get(tool, '{"ok": true, "data": {"id": "job-1"}}')
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 辅助函数
|
||||
# ===========================================================================
|
||||
|
||||
class TestChainToCapability:
|
||||
def test_known(self):
|
||||
assert ba._chain_to_capability("gene_family") == "bioflow-gene-family"
|
||||
assert ba._chain_to_capability("rnaseq") == "bioflow-rnaseq"
|
||||
|
||||
def test_unknown_and_empty(self):
|
||||
assert ba._chain_to_capability("wgs") is None
|
||||
assert ba._chain_to_capability("") is None
|
||||
assert ba._chain_to_capability(None) is None
|
||||
|
||||
def test_case_and_whitespace(self):
|
||||
assert ba._chain_to_capability(" Gene_Family ") == "bioflow-gene-family"
|
||||
|
||||
|
||||
class TestExtractJobId:
|
||||
def test_data_id(self):
|
||||
assert ba._extract_job_id('{"ok": true, "data": {"id": "job-9"}}') == "job-9"
|
||||
|
||||
def test_data_job_id(self):
|
||||
assert ba._extract_job_id('{"ok": true, "data": {"job_id": "job-8"}}') == "job-8"
|
||||
|
||||
def test_flat_id(self):
|
||||
assert ba._extract_job_id('{"ok": true, "id": "job-7"}') == "job-7"
|
||||
|
||||
def test_non_json_and_no_id(self):
|
||||
assert ba._extract_job_id("not json") is None
|
||||
assert ba._extract_job_id('{"ok": false}') is None
|
||||
assert ba._extract_job_id("") is None
|
||||
|
||||
|
||||
class TestExecuteTool:
|
||||
def test_unknown_tool(self):
|
||||
out = ba._execute_tool(_tool_manifests(), "ghost", {})
|
||||
assert "未注册" in out
|
||||
|
||||
def test_tool_exception_becomes_text(self):
|
||||
class Boom:
|
||||
def execute(self, manifest, arguments):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=Boom()):
|
||||
out = ba._execute_tool(_tool_manifests(), "job_status", {"job_id": "j"})
|
||||
assert "工具执行失败" in out
|
||||
|
||||
|
||||
class TestParaphrase:
|
||||
def test_llm_success(self):
|
||||
out = ba._paraphrase(FakeLLM(complete_text="已转述。"), "原文")
|
||||
assert out == "已转述。"
|
||||
|
||||
def test_llm_exception_falls_back_to_raw(self):
|
||||
out = ba._paraphrase(FakeLLM(complete_error=RuntimeError("boom")), "原文")
|
||||
assert out == "原文"
|
||||
|
||||
def test_llm_empty_falls_back_to_raw(self):
|
||||
out = ba._paraphrase(FakeLLM(complete_text=" "), "原文")
|
||||
assert out == "原文"
|
||||
|
||||
|
||||
class TestQueryJobStatus:
|
||||
def test_calls_job_status_and_paraphrases(self):
|
||||
with mock.patch.object(ba, "_execute_tool", return_value='{"ok": true, "data": {"status": "running"}}'):
|
||||
out = ba._query_job_status("j1", _tool_manifests(), FakeLLM(complete_text="任务进行中。"))
|
||||
assert out == "任务进行中。"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# card 四类 handler 边界（缺必需字段 / files 键 / archive=False / 非 dict payload）
|
||||
# ===========================================================================
|
||||
|
||||
class TestCardHandlersEdge:
|
||||
def test_form_confirm_missing_job_and_stage_returns_none(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "form_confirm", "payload": {"params": {}}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_file_upload_missing_job_returns_none(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "file_upload", "payload": {"paths": ["/data/a.fa"]}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_file_upload_missing_files_returns_none(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "file_upload", "payload": {"job_id": "j1"}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_file_upload_uses_files_key(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "file_upload",
|
||||
"payload": {"job_id": "j1", "files": ["https://x/a.fa"]}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "fetch_source_file"
|
||||
|
||||
def test_file_upload_string_files_coerced(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "file_upload",
|
||||
"payload": {"job_id": "j1", "files": "https://x/a.fa"}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "fetch_source_file"
|
||||
|
||||
def test_archive_choice_missing_job_returns_none(self):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "archive_choice", "payload": {"archive": True}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_archive_choice_skip(self):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "archive_choice", "payload": {"job_id": "j1", "archive": False}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert out["bioflow"]["audit"]["tool_calls"] == 0 # 存档不调工具（走 REST）
|
||||
|
||||
def test_path_selection_rnaseq_chain(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "rnaseq", "stages": ["rnaseq_qc"]}},
|
||||
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "create_rnaseq_job"
|
||||
|
||||
def test_card_non_dict_payload_treated_empty(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "form_confirm", "payload": "not-a-dict"},
|
||||
FakeLLM(), {"active_job_id": "j1"}, _tool_manifests(),
|
||||
)
|
||||
assert out is None # payload 非 dict → 无 stage/job_id → None
|
||||
|
||||
def test_card_result_with_non_dict_is_ignored(self):
|
||||
# try_bioflow_agent 对非 dict 的 card_result 不做 card 路由，走 NL 意图
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_candidate_capabilities", return_value=[]):
|
||||
out = ba.try_bioflow_agent("确认", llm_client=FakeLLM(), card_result="bogus")
|
||||
assert out is None
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# _route_card_result 边界
|
||||
# ===========================================================================
|
||||
|
||||
class TestRouteCardResultEdge:
|
||||
def test_chain_filters_to_capability(self):
|
||||
gf = _manifest("bioflow-gene-family")
|
||||
rn = _manifest("bioflow-rnaseq", "http://x/mcp/rnaseq/")
|
||||
captured = {}
|
||||
|
||||
def fake_build(manifests):
|
||||
captured["manifests"] = [m.name for m in manifests]
|
||||
return ([], {})
|
||||
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf, rn]):
|
||||
with mock.patch.object(ba, "_build_tools", side_effect=fake_build):
|
||||
ba._route_card_result(
|
||||
{"card": "form_confirm", "payload": {"chain": "rnaseq"}},
|
||||
FakeLLM(), None, None, None,
|
||||
)
|
||||
assert captured["manifests"] == ["bioflow-rnaseq"] # chain 过滤到 rnaseq 能力
|
||||
|
||||
def test_chain_to_unregistered_capability_returns_none(self):
|
||||
gf = _manifest("bioflow-gene-family")
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
|
||||
out = ba._route_card_result(
|
||||
{"card": "form_confirm", "payload": {"chain": "rnaseq"}},
|
||||
FakeLLM(), None, None, None,
|
||||
)
|
||||
assert out is None # rnaseq 未注册 → manifests 空 → None
|
||||
|
||||
def test_tools_empty_returns_none(self):
|
||||
gf = _manifest("bioflow-gene-family")
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([], {})):
|
||||
out = ba._route_card_result(
|
||||
{"card": "form_confirm", "payload": {}},
|
||||
FakeLLM(), None, None, None,
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_progress_callback_invoked(self):
|
||||
gf = _manifest("bioflow-gene-family")
|
||||
calls = []
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
|
||||
with mock.patch.object(ba, "_handle_card_result", return_value=None):
|
||||
ba._route_card_result(
|
||||
{"card": "form_confirm", "payload": {}},
|
||||
FakeLLM(), None, None, lambda p: calls.append(p),
|
||||
)
|
||||
assert calls and calls[0]["stage"] == "bioflow_card"
|
||||
assert calls[0]["card"] == "form_confirm"
|
||||
|
||||
def test_progress_callback_exception_swallowed(self):
|
||||
gf = _manifest("bioflow-gene-family")
|
||||
|
||||
def boom(p):
|
||||
raise RuntimeError("callback boom")
|
||||
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf]):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([{"x": 1}], {})):
|
||||
with mock.patch.object(ba, "_handle_card_result", return_value={"ok": True}):
|
||||
out = ba._route_card_result(
|
||||
{"card": "form_confirm", "payload": {}},
|
||||
FakeLLM(), None, None, boom,
|
||||
)
|
||||
assert out == {"ok": True} # 回调异常被吞，不影响主流程
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# NL 路径：bioflow 上下文注入 / 卡片与 NL 语义等价
|
||||
# ===========================================================================
|
||||
|
||||
class TestNLPathBioflowContext:
|
||||
def test_nl_path_injects_bioflow_context(self, tmp_path):
|
||||
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry
|
||||
from rag.qa_v1_1_73.capabilities.manifests import parse_capabilities_text
|
||||
yaml = """
|
||||
capabilities:
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
session_default: true
|
||||
nl_triggers: [基因家族]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/gene-family/
|
||||
"""
|
||||
(tmp_path / "c.yaml").write_text(yaml, encoding="utf-8")
|
||||
registry = CapabilityRegistry(tmp_path / "c.yaml")
|
||||
registry.load()
|
||||
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
|
||||
"description": "x",
|
||||
"parameters": {"type": "object", "properties": {}}}}]
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict("os.environ", {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
|
||||
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
|
||||
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
|
||||
out = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLM(),
|
||||
bioflow={"session_id": "s1", "active_job_id": "job-9", "chain": "gene_family"},
|
||||
)
|
||||
assert out is not None
|
||||
assert out["bioflow"]["active_job_id"] == "job-9"
|
||||
assert out["bioflow"]["chain"] == "gene_family"
|
||||
|
||||
def test_card_and_nl_share_tool_chain(self):
|
||||
"""卡片操作与纯 NL 服务端语义等价：卡片 handler 只调 MCP 工具，不直连控制面。"""
|
||||
# 卡片 handler 用到的工具名 = MCP 工具（job_status/confirm_stage/create_* 等）
|
||||
tm = _tool_manifests()
|
||||
# 四类卡片触发的工具都在 tool_manifests（即 MCP 工具表）里
|
||||
for tool in ("create_gene_family_job", "create_rnaseq_job", "confirm_stage",
|
||||
"register_job_files", "fetch_source_file", "job_status"):
|
||||
assert tool in tm
|
||||
|
||||
|
||||
class TestExtractParamsEdge:
|
||||
def test_llm_exception_returns_empty(self):
|
||||
class RaisingLLM(FakeLLM):
|
||||
def complete_json(self, system, user, **kw):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
known, unknown = ba._extract_params(RaisingLLM(), "x", {"threads": {}})
|
||||
assert known == {} and unknown == []
|
||||
|
||||
def test_unknown_as_string_coerced_to_list(self):
|
||||
llm = FakeLLM(json_payload={"params": {"threads": 8}, "unknown": "bogus_param"})
|
||||
known, unknown = ba._extract_params(llm, "x", {"threads": {}})
|
||||
assert known == {"threads": 8}
|
||||
assert unknown == ["bogus_param"]
|
||||
|
||||
def test_path_selection_with_params(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "gene_family", "stages": ["identify"],
|
||||
"params": {"prefix": "Ft.SPL"}}},
|
||||
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
|
||||
)
|
||||
assert dispatcher.calls[0][1]["params"] == {"prefix": "Ft.SPL"}
|
||||
|
||||
def test_path_selection_create_tool_missing_returns_none(self):
|
||||
tm = _tool_manifests()
|
||||
tm.pop("create_gene_family_job")
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "gene_family", "stages": ["identify"]}},
|
||||
FakeLLM(), {"session_id": "s1"}, tm,
|
||||
)
|
||||
assert out is None
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# system prompt 硬规则在位
|
||||
# ===========================================================================
|
||||
|
||||
class TestSystemPromptRules:
|
||||
def test_param_retry_rule(self):
|
||||
assert "最多 3 次" in ba._SYSTEM_BASE
|
||||
|
||||
def test_human_boundary_rule(self):
|
||||
assert "不可跳过" in ba._SYSTEM_BASE
|
||||
assert "不可代用户确认" in ba._SYSTEM_BASE
|
||||
|
||||
def test_tool_name_no_leak_rule(self):
|
||||
assert "绝不向用户泄露工具名" in ba._SYSTEM_BASE
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# ToolRunner.run(history) 边界
|
||||
# ===========================================================================
|
||||
|
||||
class TestHistoryEdge:
|
||||
def _run_with_history(self, history):
|
||||
captured = {}
|
||||
|
||||
class RecordingLLM:
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
captured["messages"] = list(messages)
|
||||
return {"content": "ok", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
runner = ToolRunner(make_llm_complete(RecordingLLM()), max_rounds=1)
|
||||
runner.run("sys", "当前", [], {}, history=history)
|
||||
return captured["messages"]
|
||||
|
||||
def test_none_history(self):
|
||||
msgs = self._run_with_history(None)
|
||||
assert msgs == [{"role": "user", "content": "当前"}]
|
||||
|
||||
def test_non_dict_items_skipped(self):
|
||||
msgs = self._run_with_history(["not-a-dict", {"role": "user", "content": "ok"}])
|
||||
assert msgs == [{"role": "user", "content": "ok"}, {"role": "user", "content": "当前"}]
|
||||
|
||||
def test_empty_content_filtered(self):
|
||||
msgs = self._run_with_history([{"role": "user", "content": " "}])
|
||||
assert msgs == [{"role": "user", "content": "当前"}]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S3：card_result rerun / reuse_archive / archive_job 语义
|
||||
# ===========================================================================
|
||||
|
||||
class TestS3CardSemantics:
|
||||
def test_form_confirm_rerun_true_uses_rerun_stage(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "form_confirm",
|
||||
"payload": {"job_id": "j1", "stage": "identify",
|
||||
"params": {"evalue": 1e-5}, "rerun": True}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "rerun_stage"
|
||||
assert out["bioflow"]["audit"]["tool_names"] == ["rerun_stage"]
|
||||
|
||||
def test_form_confirm_rerun_false_uses_confirm_stage(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
ba._handle_card_result(
|
||||
{"card": "form_confirm",
|
||||
"payload": {"job_id": "j1", "stage": "identify", "params": {}}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert dispatcher.calls[0][0] == "confirm_stage"
|
||||
|
||||
def test_path_selection_reuse_archive(self):
|
||||
dispatcher = FakeDispatcher({
|
||||
"reuse_archive": '{"ok": true, "data": {"id": "arch-1", '
|
||||
'"stage_chain": ["identify", "phylogeny"], '
|
||||
'"params": {"identify": {"prefix": "Ft"}}}}',
|
||||
})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "path_selection",
|
||||
"payload": {"chain": "gene_family", "reuse_archive": "SPL 鉴定"}},
|
||||
FakeLLM(), {"session_id": "s1"}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
tools = [c[0] for c in dispatcher.calls]
|
||||
assert tools[0] == "reuse_archive"
|
||||
assert tools[1] == "create_gene_family_job"
|
||||
create_args = dispatcher.calls[1][1]
|
||||
assert create_args["archive_id"] == "arch-1"
|
||||
assert create_args["stages"] == ["identify", "phylogeny"]
|
||||
assert create_args["params"] == {"identify": {"prefix": "Ft"}}
|
||||
assert out["bioflow"]["audit"]["tool_names"] == ["reuse_archive", "create_gene_family_job"]
|
||||
|
||||
def test_archive_choice_archive_true_calls_archive_job(self):
|
||||
dispatcher = FakeDispatcher({"archive_job": '{"ok": true, "data": {"id": "a1"}}'})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "archive_choice",
|
||||
"payload": {"job_id": "j1", "archive": True,
|
||||
"name": "SPL 存档", "description": "d"}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == "archive_job"
|
||||
assert dispatcher.calls[0][1] == {"job_id": "j1", "name": "SPL 存档", "description": "d"}
|
||||
assert out["bioflow"]["audit"]["tool_names"] == ["archive_job"]
|
||||
|
||||
def test_archive_choice_archive_false_no_tool_call(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "archive_choice",
|
||||
"payload": {"job_id": "j1", "archive": False}},
|
||||
FakeLLM(), {}, _tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls == [] # 跳过存档不发工具请求
|
||||
assert out["bioflow"]["audit"]["tool_calls"] == 0
|
||||
@@ -0,0 +1,206 @@
|
||||
"""test_bioflow_agent_s4_supplement.py —— S4 覆盖清单补强测试。
|
||||
|
||||
覆盖：
|
||||
- _card_extension_action 六种 action 分发到对应扩展工具
|
||||
- 未知 action / 缺 ext_id/action 回退 None
|
||||
- confirm 携带 nl_triggers/parameters/runtime_image；EXT_STATUS 降级话术（确定性）
|
||||
- run 携带 params/files
|
||||
- 真实 configs/capabilities.yaml 三条目（含 bioflow-extension）可解析
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.manifests import CapabilityManifest, parse_capabilities_file # noqa: E402
|
||||
|
||||
EXT_TOOLS = (
|
||||
"create_extension", "extension_status", "build_extension", "trial_run_extension",
|
||||
"validate_extension", "confirm_extension", "publish_extension", "run_extension",
|
||||
"repair_extension",
|
||||
)
|
||||
|
||||
|
||||
def _ext_tool_manifests():
|
||||
out = {}
|
||||
for t in EXT_TOOLS:
|
||||
out[t] = CapabilityManifest(
|
||||
name="bioflow-extension", provider="mcp",
|
||||
config={"mcp": {"url": "http://127.0.0.1:8098/mcp/extension/", "tool": t}},
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
class FakeLLM:
|
||||
def complete(self, system, user, **kw):
|
||||
return "已收到。"
|
||||
|
||||
def complete_json(self, system, user, **kw):
|
||||
return {}
|
||||
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
return {"content": "answer", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
class FakeDispatcher:
|
||||
def __init__(self, results=None):
|
||||
self.results = results or {}
|
||||
self.calls = []
|
||||
|
||||
def execute(self, manifest, arguments):
|
||||
tool = manifest.config.get("mcp", {}).get("tool")
|
||||
self.calls.append((tool, arguments))
|
||||
return self.results.get(tool, '{"ok": true, "data": {"ext_id": "e1"}}')
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# _card_extension_action 分发
|
||||
# ===========================================================================
|
||||
|
||||
class TestExtensionAction:
|
||||
@pytest.mark.parametrize("action,expected_tool", [
|
||||
("build", "build_extension"),
|
||||
("trial_run", "trial_run_extension"),
|
||||
("validate", "validate_extension"),
|
||||
("confirm", "confirm_extension"),
|
||||
("publish", "publish_extension"),
|
||||
("run", "run_extension"),
|
||||
("repair", "repair_extension"),
|
||||
])
|
||||
def test_action_dispatches_to_tool(self, action, expected_tool):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1", "action": action}},
|
||||
FakeLLM(), {}, _ext_tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert dispatcher.calls[0][0] == expected_tool
|
||||
assert dispatcher.calls[0][1] == {"ext_id": "e1"}
|
||||
assert out["bioflow"]["audit"]["tool_names"] == [expected_tool]
|
||||
|
||||
def test_unknown_action_returns_none(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "bogus"}},
|
||||
FakeLLM(), {}, _ext_tool_manifests(),
|
||||
)
|
||||
assert out is None
|
||||
|
||||
def test_missing_ext_id_or_action_returns_none(self):
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
assert ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"action": "build"}},
|
||||
FakeLLM(), {}, _ext_tool_manifests(),
|
||||
) is None
|
||||
assert ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1"}},
|
||||
FakeLLM(), {}, _ext_tool_manifests(),
|
||||
) is None
|
||||
|
||||
def test_confirm_passes_triggers_params_image(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
payload = {"ext_id": "e1", "action": "confirm",
|
||||
"nl_triggers": ["扩展", "触发词"], "parameters": [{"name": "x"}],
|
||||
"runtime_image": "img"}
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
ba._handle_card_result({"card": "extension_action", "payload": payload},
|
||||
FakeLLM(), {}, _ext_tool_manifests())
|
||||
args = dispatcher.calls[0][1]
|
||||
assert args == {"ext_id": "e1", "nl_triggers": ["扩展", "触发词"],
|
||||
"parameters": [{"name": "x"}], "runtime_image": "img"}
|
||||
|
||||
def test_confirm_status_insufficient_uses_deterministic_script(self):
|
||||
# confirm 返回 EXT_STATUS（状态不足）→ 确定性降级话术（不报 500、不经 LLM）
|
||||
dispatcher = FakeDispatcher({
|
||||
"confirm_extension": '{"ok": false, "code": "EXT_STATUS", "message": "状态不可确认"}',
|
||||
})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "confirm"}},
|
||||
FakeLLM(), {}, _ext_tool_manifests(),
|
||||
)
|
||||
assert out is not None
|
||||
assert "还不能确认" in out["answer"]
|
||||
assert "先完成" in out["answer"] # 确定性话术（非 LLM 转述）
|
||||
|
||||
def test_run_passes_params_and_files(self):
|
||||
dispatcher = FakeDispatcher()
|
||||
payload = {"ext_id": "e1", "action": "run",
|
||||
"params": {"cpu": 8}, "files": ["/data/a.fa"]}
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
ba._handle_card_result({"card": "extension_action", "payload": payload},
|
||||
FakeLLM(), {}, _ext_tool_manifests())
|
||||
assert dispatcher.calls[0][1] == {"ext_id": "e1", "params": {"cpu": 8}, "files": ["/data/a.fa"]}
|
||||
|
||||
def test_extension_tool_not_in_manifests_returns_none(self):
|
||||
# tool_manifests 里没有扩展工具 → 回退 None（能力未安装/未发现工具）
|
||||
empty = CapabilityManifest(name="bioflow-gene-family", provider="mcp",
|
||||
config={"mcp": {"url": "http://x", "tool": "job_status"}})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=FakeDispatcher()):
|
||||
out = ba._handle_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "build"}},
|
||||
FakeLLM(), {}, {"job_status": empty},
|
||||
)
|
||||
assert out is None
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 真实 capabilities.yaml 三条目可解析
|
||||
# ===========================================================================
|
||||
|
||||
class TestCapabilitiesThreeEntries:
|
||||
def _real_yaml(self) -> Path:
|
||||
return Path(__file__).resolve().parents[5] / "agent" / "qa" / "configs" / "capabilities.yaml"
|
||||
|
||||
def test_three_bioflow_entries(self):
|
||||
manifests, warnings = parse_capabilities_file(self._real_yaml())
|
||||
by_name = {m.name: m for m in manifests}
|
||||
assert {"bioflow-gene-family", "bioflow-rnaseq", "bioflow-extension"} <= set(by_name)
|
||||
|
||||
ext = by_name["bioflow-extension"]
|
||||
assert ext.provider == "mcp"
|
||||
assert ext.permission == "read_only"
|
||||
assert ext.nl_triggers
|
||||
assert ext.config["mcp"]["url"].endswith("/mcp/extension/")
|
||||
assert ext.config["prompt_pack"]["file"] == "bioflow_extension_agent_rules.md"
|
||||
|
||||
def test_bioflow_extension_in_capability_names(self):
|
||||
assert "bioflow-extension" in ba.BIOFLOW_CAPABILITY_NAMES
|
||||
assert ba.CARD_EXTENSION_ACTION == "extension_action"
|
||||
assert ba.CARD_EXTENSION_ACTION in ba.CARD_TYPES
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S5：_route_card_result 对 extension_action 只发现 extension 能力（一次 list_tools）
|
||||
# ===========================================================================
|
||||
|
||||
class TestRouteExtensionOptimization:
|
||||
def test_extension_action_discovers_only_extension_capability(self):
|
||||
gf = CapabilityManifest(name="bioflow-gene-family", provider="mcp",
|
||||
config={"mcp": {"url": "http://x/gene-family/"}})
|
||||
rn = CapabilityManifest(name="bioflow-rnaseq", provider="mcp",
|
||||
config={"mcp": {"url": "http://x/rnaseq/"}})
|
||||
ext = CapabilityManifest(name="bioflow-extension", provider="mcp",
|
||||
config={"mcp": {"url": "http://x/extension/"}})
|
||||
captured = {}
|
||||
|
||||
def fake_build(manifests):
|
||||
captured["names"] = [m.name for m in manifests]
|
||||
return ([], {})
|
||||
|
||||
with mock.patch.object(ba, "_registered_capabilities", return_value=[gf, rn, ext]):
|
||||
with mock.patch.object(ba, "_build_tools", side_effect=fake_build):
|
||||
ba._route_card_result(
|
||||
{"card": "extension_action", "payload": {"ext_id": "e1", "action": "build"}},
|
||||
FakeLLM(), None, None, None,
|
||||
)
|
||||
# 只把 bioflow-extension manifest 交给 _build_tools（避免对三能力各 list_tools 一次）
|
||||
assert captured["names"] == ["bioflow-extension"]
|
||||
@@ -0,0 +1,116 @@
|
||||
"""test_bioflow_agent_s6_config.py —— S6 多配置回归（QA 单元层）。
|
||||
|
||||
模拟四种部署配置，断言候选能力集合 + 意图预筛结果正确（真实 E2E/金标 20 题依赖部署，
|
||||
此处以 monkeypatch/env 在单元层覆盖漏判安全语义）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
|
||||
|
||||
THREE_YAML = """
|
||||
capabilities:
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
session_default: true
|
||||
nl_triggers: [基因家族, 亚家族]
|
||||
mcp: {url: "http://x/gene-family/"}
|
||||
- name: bioflow-rnaseq
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
session_default: true
|
||||
nl_triggers: [转录组, 差异表达]
|
||||
mcp: {url: "http://x/rnaseq/"}
|
||||
- name: bioflow-extension
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
session_default: true
|
||||
nl_triggers: [自定义扩展]
|
||||
mcp: {url: "http://x/extension/"}
|
||||
"""
|
||||
|
||||
|
||||
def _registry(tmp_path, yaml_text=THREE_YAML):
|
||||
path = tmp_path / "caps.yaml"
|
||||
path.write_text(yaml_text, encoding="utf-8")
|
||||
reg = CapabilityRegistry(path)
|
||||
reg.load()
|
||||
return reg
|
||||
|
||||
|
||||
class FakeLLM:
|
||||
def complete_raw_messages(self, *a, **kw):
|
||||
return {"content": "x", "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 配置 1：全关（QA_V1_ENABLE_BIOFLOW_AGENT=0 + 三能力全卸载）
|
||||
# ===========================================================================
|
||||
|
||||
class TestConfigAllOff:
|
||||
def test_fuse_off_returns_none(self, tmp_path):
|
||||
reg = _registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=reg):
|
||||
with mock.patch.dict(ba.os.environ, {}, clear=False):
|
||||
ba.os.environ.pop(ba.ENV_FUSE, None)
|
||||
out = ba.try_bioflow_agent("帮我做基因家族分析", llm_client=FakeLLM())
|
||||
assert out is None # 走原 pipeline
|
||||
|
||||
def test_all_uninstalled_no_candidate(self, tmp_path):
|
||||
reg = _registry(tmp_path, "capabilities: []\n") # 三能力全卸载
|
||||
with mock.patch.object(ba, "get_registry", return_value=reg):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
assert ba._candidate_capabilities("帮我做基因家族分析", None) == []
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 配置 2：只开基因家族
|
||||
# ===========================================================================
|
||||
|
||||
class TestConfigGeneFamilyOnly:
|
||||
def test_gene_family_hit_rnaseq_miss(self, tmp_path):
|
||||
reg = _registry(tmp_path)
|
||||
caps = {"bioflow-gene-family": True, "bioflow-rnaseq": False, "bioflow-extension": False}
|
||||
with mock.patch.object(ba, "get_registry", return_value=reg):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
names = [m.name for m in ba._candidate_capabilities("帮我做水稻 SPL 基因家族分析", caps)]
|
||||
assert names == ["bioflow-gene-family"]
|
||||
assert ba._candidate_capabilities("帮我做转录组差异表达分析", caps) == []
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 配置 3：只开转录组
|
||||
# ===========================================================================
|
||||
|
||||
class TestConfigRnaseqOnly:
|
||||
def test_rnaseq_hit_gene_family_miss(self, tmp_path):
|
||||
reg = _registry(tmp_path)
|
||||
caps = {"bioflow-gene-family": False, "bioflow-rnaseq": True, "bioflow-extension": False}
|
||||
with mock.patch.object(ba, "get_registry", return_value=reg):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
names = [m.name for m in ba._candidate_capabilities("转录组差异表达分析", caps)]
|
||||
assert names == ["bioflow-rnaseq"]
|
||||
assert ba._candidate_capabilities("帮我做水稻 SPL 基因家族分析", caps) == []
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 配置 4：全开但非生信提问
|
||||
# ===========================================================================
|
||||
|
||||
class TestConfigAllOnNonBioflow:
|
||||
def test_non_bioflow_question_no_candidate(self, tmp_path):
|
||||
reg = _registry(tmp_path)
|
||||
caps = {"bioflow-gene-family": True, "bioflow-rnaseq": True, "bioflow-extension": True}
|
||||
with mock.patch.object(ba, "get_registry", return_value=reg):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
assert ba._candidate_capabilities("光合作用的电子传递链是什么", caps) == []
|
||||
assert ba._candidate_capabilities("什么是转录因子", caps) == []
|
||||
@@ -0,0 +1,496 @@
|
||||
"""test_bioflow_agent_supplement.py —— S1 覆盖清单补强测试。
|
||||
|
||||
补强 dev 的 test_bioflow_agent.py 未覆盖的分支（对照 S1 覆盖清单）：
|
||||
- 熔断开关值解析（1/true/yes/on 与 0/false/off/空/缺失）
|
||||
- nl_triggers 同时命中 → 两个候选（交 LLM 依 description 选）
|
||||
- session_default 回退（True/False）/ registry 访问异常 / registry.get 异常
|
||||
- _build_tools：MCP 工具发现（空 url 跳过 / 发现异常 / 共享连接按能力名 / 无名跳过 / bad schema 兜底）
|
||||
- _build_system：prompt_pack 注入 / 去重 / 加载异常兜底
|
||||
- try_bioflow_agent：tools 空→None、answer 空→None、progress_callback 触发与吞异常
|
||||
- ToolRunner max_rounds=5 / bioflow_agent.MAX_ROUNDS 常量
|
||||
- make_llm_complete 包装（正常 / 异常）
|
||||
- 真实 configs/capabilities.yaml 两条目可解析
|
||||
- bioflow_agent 不 import pipeline/answer_composer（红线静态守卫）
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent.parent / "src"))
|
||||
|
||||
from rag.qa_v1_1_73 import bioflow_agent as ba # noqa: E402
|
||||
from rag.qa_v1_1_73.capabilities.manifests import ( # noqa: E402
|
||||
CapabilityManifest,
|
||||
parse_capabilities_file,
|
||||
)
|
||||
from rag.qa_v1_1_73.capabilities.registry import CapabilityRegistry # noqa: E402
|
||||
|
||||
BIOFLOW_YAML = """
|
||||
capabilities:
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 基因家族分析
|
||||
session_default: true
|
||||
nl_triggers: [基因家族, 家族成员, 亚家族, 建树]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/gene-family/
|
||||
prompt_pack:
|
||||
file: bioflow_gene_family_agent_rules.md
|
||||
- name: bioflow-rnaseq
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 转录组分析
|
||||
session_default: true
|
||||
nl_triggers: [转录组, RNA-seq, 差异表达, 富集]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/rnaseq/
|
||||
prompt_pack:
|
||||
file: bioflow_rnaseq_agent_rules.md
|
||||
"""
|
||||
|
||||
|
||||
def _make_registry(tmp_path: Path, yaml_text: str = BIOFLOW_YAML) -> CapabilityRegistry:
|
||||
path = tmp_path / "capabilities.yaml"
|
||||
path.write_text(yaml_text, encoding="utf-8")
|
||||
registry = CapabilityRegistry(path)
|
||||
registry.load()
|
||||
return registry
|
||||
|
||||
|
||||
def _gf_manifest(**overrides) -> CapabilityManifest:
|
||||
cfg = {
|
||||
"mcp": {"url": "http://127.0.0.1:8098/mcp/gene-family/"},
|
||||
"prompt_pack": {"file": "bioflow_gene_family_agent_rules.md"},
|
||||
}
|
||||
cfg.update(overrides.pop("config", {}))
|
||||
return CapabilityManifest(
|
||||
name="bioflow-gene-family", provider="mcp", permission="read_only",
|
||||
description="基因家族分析", config=cfg, nl_triggers=["基因家族"], **overrides,
|
||||
)
|
||||
|
||||
|
||||
class FakeLLMClient:
|
||||
def __init__(self, content: str = "已为您规划分析路径。") -> None:
|
||||
self.content = content
|
||||
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kwargs):
|
||||
return {"content": self.content, "tool_calls": [], "finish_reason": "stop"}
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 门 1：熔断开关值解析
|
||||
# ===========================================================================
|
||||
|
||||
class TestEnvFuse:
|
||||
@pytest.mark.parametrize("value", ["1", "true", "TRUE", "yes", "on", " On ", "YES"])
|
||||
def test_truthy(self, value):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: value}):
|
||||
assert ba._env_enabled() is True
|
||||
|
||||
@pytest.mark.parametrize("value", ["0", "false", "off", "", "no", " "])
|
||||
def test_falsy(self, value):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: value}):
|
||||
assert ba._env_enabled() is False
|
||||
|
||||
def test_missing_defaults_off(self):
|
||||
with mock.patch.dict(ba.os.environ, {}, clear=False):
|
||||
ba.os.environ.pop(ba.ENV_FUSE, None)
|
||||
assert ba._env_enabled() is False
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 门 3：nl_triggers 意图预筛
|
||||
# ===========================================================================
|
||||
|
||||
class TestIntentMatchEdge:
|
||||
def test_empty_triggers_never_match(self):
|
||||
assert ba._intent_match("做基因家族分析", []) is False
|
||||
assert ba._intent_match("做基因家族分析", [""]) is False
|
||||
|
||||
def test_case_insensitive_substring(self):
|
||||
assert ba._intent_match("RNA-SEQ 分析", ["rna-seq"]) is True
|
||||
assert ba._intent_match("ka/ks 选择压力", ["Ka/Ks"]) is True
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 门 2 + 门 3：候选能力（registry 异常 / session_default 回退 / 同时命中）
|
||||
# ===========================================================================
|
||||
|
||||
class TestCandidateCapabilities:
|
||||
def test_registry_access_exception_returns_empty(self):
|
||||
with mock.patch.object(ba, "get_registry", side_effect=RuntimeError("boom")):
|
||||
assert ba._candidate_capabilities("做基因家族", None) == []
|
||||
|
||||
def test_registry_get_exception_skips(self):
|
||||
class BadRegistry:
|
||||
def get(self, name):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
with mock.patch.object(ba, "get_registry", return_value=BadRegistry()):
|
||||
assert ba._candidate_capabilities("做基因家族", None) == []
|
||||
|
||||
def test_both_triggers_match_returns_both(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
caps = ba._candidate_capabilities("做基因家族和转录组差异表达分析", None)
|
||||
assert [m.name for m in caps] == ["bioflow-gene-family", "bioflow-rnaseq"]
|
||||
|
||||
def test_session_default_false_skips(self, tmp_path):
|
||||
yaml = BIOFLOW_YAML.replace("session_default: true", "session_default: false")
|
||||
registry = _make_registry(tmp_path, yaml)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
# 无会话开关时回退 session_default=false → 不候选
|
||||
assert ba._candidate_capabilities("做基因家族分析", None) == []
|
||||
# 显式会话开关 true → 候选
|
||||
caps = ba._candidate_capabilities("做基因家族分析", {"bioflow-gene-family": True})
|
||||
assert [m.name for m in caps] == ["bioflow-gene-family"]
|
||||
|
||||
def test_session_switch_true_overrides_default_false(self, tmp_path):
|
||||
yaml = BIOFLOW_YAML.replace("session_default: true", "session_default: false", 1)
|
||||
registry = _make_registry(tmp_path, yaml)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
caps = ba._candidate_capabilities("做基因家族分析", {"bioflow-gene-family": True})
|
||||
assert [m.name for m in caps] == ["bioflow-gene-family"]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# _build_tools：MCP 工具发现
|
||||
# ===========================================================================
|
||||
|
||||
class _FakeDispatcher:
|
||||
def __init__(self, tools_by_name=None, exc=None):
|
||||
self.tools_by_name = tools_by_name or {}
|
||||
self.exc = exc
|
||||
|
||||
def list_mcp_tools(self, manifest):
|
||||
if self.exc:
|
||||
raise self.exc
|
||||
return self.tools_by_name.get(manifest.name, [])
|
||||
|
||||
|
||||
class TestBuildTools:
|
||||
def test_empty_url_skipped(self):
|
||||
manifest = CapabilityManifest(
|
||||
name="x", provider="mcp", config={"mcp": {"url": ""}}
|
||||
)
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=_FakeDispatcher()):
|
||||
tools, tool_manifests = ba._build_tools([manifest])
|
||||
assert tools == []
|
||||
assert tool_manifests == {}
|
||||
|
||||
def test_discovery_exception_skipped(self):
|
||||
manifest = _gf_manifest()
|
||||
with mock.patch.object(
|
||||
ba, "get_dispatcher",
|
||||
return_value=_FakeDispatcher(exc=RuntimeError("mcp down")),
|
||||
):
|
||||
tools, tool_manifests = ba._build_tools([manifest])
|
||||
assert tools == []
|
||||
assert tool_manifests == {}
|
||||
|
||||
def test_builds_schema_and_shared_connection(self):
|
||||
dispatcher = _FakeDispatcher({
|
||||
"bioflow-gene-family": [
|
||||
{"name": "plan_gene_family", "description": "规划路径",
|
||||
"input_schema": {"type": "object", "properties": {"message": {"type": "string"}}}},
|
||||
]
|
||||
})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
tools, tool_manifests = ba._build_tools([_gf_manifest()])
|
||||
assert len(tools) == 1
|
||||
assert tools[0]["function"]["name"] == "plan_gene_family"
|
||||
assert tools[0]["function"]["description"] == "规划路径"
|
||||
# 共享连接：manifest.name 用能力名，mcp.tool 指向真实 server 工具名
|
||||
tm = tool_manifests["plan_gene_family"]
|
||||
assert tm.name == "bioflow-gene-family"
|
||||
assert tm.config["mcp"]["tool"] == "plan_gene_family"
|
||||
assert tm.config["mcp"]["url"].endswith("/mcp/gene-family/")
|
||||
|
||||
def test_skips_nameless_and_bad_schema(self):
|
||||
dispatcher = _FakeDispatcher({
|
||||
"bioflow-gene-family": [
|
||||
{"name": "", "description": "无名", "input_schema": {}},
|
||||
{"name": "job_status", "description": "查状态", "input_schema": "not-a-dict"},
|
||||
]
|
||||
})
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
tools, tool_manifests = ba._build_tools([_gf_manifest()])
|
||||
assert [t["function"]["name"] for t in tools] == ["job_status"]
|
||||
assert tools[0]["function"]["parameters"] == {"type": "object", "properties": {}}
|
||||
|
||||
def test_shared_tool_deduped_across_capabilities(self):
|
||||
"""两能力同时命中：共享工具去重（只保留首个能力），专有工具各自保留。
|
||||
|
||||
覆盖 bug ① 修复：job_status 等 12 个共享工具两端点语义一致，按名去重后
|
||||
LLM tools 列表无重名，且 job_status 路由到首个能力（gene-family）。
|
||||
"""
|
||||
dispatcher = _FakeDispatcher({
|
||||
"bioflow-gene-family": [
|
||||
{"name": "plan_gene_family", "description": "规划基因家族", "input_schema": {}},
|
||||
{"name": "job_status", "description": "查状态", "input_schema": {}},
|
||||
{"name": "confirm_stage", "description": "确认参数", "input_schema": {}},
|
||||
],
|
||||
"bioflow-rnaseq": [
|
||||
{"name": "plan_rnaseq", "description": "规划转录组", "input_schema": {}},
|
||||
{"name": "job_status", "description": "查状态", "input_schema": {}},
|
||||
{"name": "confirm_stage", "description": "确认参数", "input_schema": {}},
|
||||
],
|
||||
})
|
||||
rn = CapabilityManifest(
|
||||
name="bioflow-rnaseq", provider="mcp", permission="read_only",
|
||||
description="转录组分析", nl_triggers=["转录组"],
|
||||
config={"mcp": {"url": "http://127.0.0.1:8098/mcp/rnaseq/"}},
|
||||
)
|
||||
with mock.patch.object(ba, "get_dispatcher", return_value=dispatcher):
|
||||
tools, tool_manifests = ba._build_tools([_gf_manifest(), rn])
|
||||
|
||||
names = [t["function"]["name"] for t in tools]
|
||||
# 专有工具各保留 + 共享工具只保留一份，且无重名
|
||||
assert names == ["plan_gene_family", "job_status", "confirm_stage", "plan_rnaseq"]
|
||||
assert len(names) == len(set(names))
|
||||
|
||||
# 共享工具路由到首个能力（gene-family）
|
||||
assert tool_manifests["job_status"].name == "bioflow-gene-family"
|
||||
assert tool_manifests["job_status"].config["mcp"]["url"].endswith("/mcp/gene-family/")
|
||||
assert tool_manifests["confirm_stage"].name == "bioflow-gene-family"
|
||||
# 专有工具各自路由到自身能力
|
||||
assert tool_manifests["plan_rnaseq"].name == "bioflow-rnaseq"
|
||||
assert tool_manifests["plan_rnaseq"].config["mcp"]["url"].endswith("/mcp/rnaseq/")
|
||||
assert tool_manifests["plan_gene_family"].name == "bioflow-gene-family"
|
||||
|
||||
# ===========================================================================
|
||||
# _build_system：prompt_pack 注入
|
||||
# ===========================================================================
|
||||
|
||||
class TestBuildSystem:
|
||||
def test_appends_prompt_pack(self):
|
||||
manifest = _gf_manifest()
|
||||
with mock.patch.object(ba.PromptPackProvider, "load", return_value="基因家族规则正文"):
|
||||
system = ba._build_system([manifest])
|
||||
assert ba._SYSTEM_BASE in system
|
||||
assert "基因家族规则正文" in system
|
||||
|
||||
def test_dedup_same_file_once(self):
|
||||
m1 = _gf_manifest()
|
||||
m2 = _gf_manifest() # 同 file
|
||||
calls = []
|
||||
with mock.patch.object(ba.PromptPackProvider, "load", side_effect=lambda m, root: calls.append(m) or "RULES"):
|
||||
system = ba._build_system([m1, m2])
|
||||
assert len(calls) == 1 # 去重后只加载一次
|
||||
assert system.count("RULES") == 1
|
||||
|
||||
def test_empty_file_skipped(self):
|
||||
manifest = CapabilityManifest(name="x", provider="mcp", config={"prompt_pack": {"file": ""}})
|
||||
system = ba._build_system([manifest])
|
||||
assert system == ba._SYSTEM_BASE
|
||||
|
||||
def test_load_exception_falls_back(self):
|
||||
manifest = _gf_manifest()
|
||||
with mock.patch.object(ba.PromptPackProvider, "load", side_effect=RuntimeError("boom")):
|
||||
system = ba._build_system([manifest])
|
||||
assert ba._SYSTEM_BASE in system # 不炸，只用基础提示
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# try_bioflow_agent：tools 空 / answer 空 / progress_callback
|
||||
# ===========================================================================
|
||||
|
||||
class TestFullFlowEdge:
|
||||
def test_tools_empty_returns_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=([], {})):
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_answer_empty_returns_none(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
|
||||
"description": "x",
|
||||
"parameters": {"type": "object", "properties": {}}}}]
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
|
||||
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
|
||||
FakeRunner.return_value.run.return_value = ("", {"rounds": 1})
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(), capabilities=None
|
||||
)
|
||||
assert result is None
|
||||
|
||||
def test_progress_callback_invoked(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
|
||||
"description": "x",
|
||||
"parameters": {"type": "object", "properties": {}}}}]
|
||||
calls = []
|
||||
|
||||
def cb(payload):
|
||||
calls.append(payload)
|
||||
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
|
||||
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
|
||||
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(),
|
||||
capabilities=None, progress_callback=cb,
|
||||
)
|
||||
assert result is not None
|
||||
assert calls and calls[0]["stage"] == "bioflow_agent_started"
|
||||
assert calls[0]["capabilities"] == ["bioflow-gene-family"]
|
||||
assert calls[0]["tool_count"] == 1
|
||||
|
||||
def test_progress_callback_exception_swallowed(self, tmp_path):
|
||||
registry = _make_registry(tmp_path)
|
||||
fake_tools = [{"type": "function", "function": {"name": "plan_gene_family",
|
||||
"description": "x",
|
||||
"parameters": {"type": "object", "properties": {}}}}]
|
||||
|
||||
def cb(payload):
|
||||
raise RuntimeError("callback boom")
|
||||
|
||||
with mock.patch.object(ba, "get_registry", return_value=registry):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
with mock.patch.object(ba, "_build_tools", return_value=(fake_tools, {})):
|
||||
with mock.patch.object(ba, "ToolRunner") as FakeRunner:
|
||||
FakeRunner.return_value.run.return_value = ("已答复", {"rounds": 1})
|
||||
result = ba.try_bioflow_agent(
|
||||
"帮我做基因家族分析", llm_client=FakeLLMClient(),
|
||||
capabilities=None, progress_callback=cb,
|
||||
)
|
||||
assert result is not None # 回调异常被吞，不影响主流程
|
||||
assert result["answer"] == "已答复"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# ToolRunner max_rounds=5 / MAX_ROUNDS 常量
|
||||
# ===========================================================================
|
||||
|
||||
class TestMaxRounds:
|
||||
def test_max_rounds_constant_is_5(self):
|
||||
assert ba.MAX_ROUNDS == 5
|
||||
|
||||
def test_tool_runner_respects_max_rounds_5(self):
|
||||
from rag.qa_v1_1_73.tool_runner import ToolRunner
|
||||
|
||||
class LLM:
|
||||
def __call__(self, system, messages, tools):
|
||||
return "", [{"id": "c", "name": "echo", "arguments": "{}"}], None
|
||||
|
||||
class Dispatcher:
|
||||
def execute(self, manifest, arguments):
|
||||
return "ok"
|
||||
|
||||
manifest = CapabilityManifest(name="echo", provider="python")
|
||||
tools = [{"type": "function", "function": {"name": "echo", "description": "e",
|
||||
"parameters": {"type": "object", "properties": {}}}}]
|
||||
runner = ToolRunner(LLM(), dispatcher=Dispatcher(), max_rounds=5)
|
||||
content, audit = runner.run("sys", "user", tools, {"echo": manifest})
|
||||
assert audit["rounds"] == 5
|
||||
assert audit["tool_calls"] == 5
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# make_llm_complete 包装
|
||||
# ===========================================================================
|
||||
|
||||
class TestMakeLLMComplete:
|
||||
def test_normal(self):
|
||||
from rag.qa_v1_1_73.tool_runner import make_llm_complete
|
||||
|
||||
class Client:
|
||||
def complete_raw_messages(self, system, messages, tools=None, **kw):
|
||||
return {"content": "hi", "tool_calls": [{"name": "x"}], "finish_reason": "stop"}
|
||||
|
||||
complete = make_llm_complete(Client())
|
||||
content, tool_calls, error = complete("sys", [{"role": "user", "content": "q"}], [])
|
||||
assert content == "hi"
|
||||
assert tool_calls == [{"name": "x"}]
|
||||
assert error is None
|
||||
|
||||
def test_exception_becomes_error(self):
|
||||
from rag.qa_v1_1_73.tool_runner import make_llm_complete
|
||||
|
||||
class Client:
|
||||
def complete_raw_messages(self, *a, **kw):
|
||||
raise RuntimeError("llm down")
|
||||
|
||||
complete = make_llm_complete(Client())
|
||||
content, tool_calls, error = complete("sys", [], [])
|
||||
assert content == ""
|
||||
assert tool_calls == []
|
||||
assert "RuntimeError" in error
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 真实 configs/capabilities.yaml 两条目可解析
|
||||
# ===========================================================================
|
||||
|
||||
class TestRealCapabilitiesYaml:
|
||||
def _real_yaml(self) -> Path:
|
||||
repo_root = Path(__file__).resolve().parents[5]
|
||||
return repo_root / "agent" / "qa" / "configs" / "capabilities.yaml"
|
||||
|
||||
def test_both_bioflow_entries_parse(self):
|
||||
manifests, warnings = parse_capabilities_file(self._real_yaml())
|
||||
by_name = {m.name: m for m in manifests}
|
||||
assert "bioflow-gene-family" in by_name
|
||||
assert "bioflow-rnaseq" in by_name
|
||||
|
||||
gf = by_name["bioflow-gene-family"]
|
||||
assert gf.provider == "mcp"
|
||||
assert gf.permission == "read_only"
|
||||
assert gf.nl_triggers
|
||||
assert gf.config["mcp"]["url"].endswith("/mcp/gene-family/")
|
||||
assert gf.config["prompt_pack"]["file"] == "bioflow_gene_family_agent_rules.md"
|
||||
|
||||
rn = by_name["bioflow-rnaseq"]
|
||||
assert rn.permission == "read_only"
|
||||
assert rn.nl_triggers
|
||||
assert rn.config["mcp"]["url"].endswith("/mcp/rnaseq/")
|
||||
assert rn.config["prompt_pack"]["file"] == "bioflow_rnaseq_agent_rules.md"
|
||||
|
||||
def test_nl_triggers_cover_expected_words(self):
|
||||
manifests, _ = parse_capabilities_file(self._real_yaml())
|
||||
gf = next(m for m in manifests if m.name == "bioflow-gene-family")
|
||||
rn = next(m for m in manifests if m.name == "bioflow-rnaseq")
|
||||
# 关键触发词在位（意图预筛依赖）
|
||||
assert "基因家族" in gf.nl_triggers
|
||||
assert "转录组" in rn.nl_triggers
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# 红线静态守卫：bioflow_agent 不依赖 pipeline / answer_composer
|
||||
# ===========================================================================
|
||||
|
||||
class TestRedLineNoPipelineDep:
|
||||
def test_no_pipeline_or_answer_composer_import(self):
|
||||
import ast
|
||||
|
||||
src = Path(ba.__file__).read_text(encoding="utf-8")
|
||||
tree = ast.parse(src)
|
||||
imported = []
|
||||
for node in tree.body:
|
||||
if isinstance(node, ast.Import):
|
||||
imported += [a.name for a in node.names]
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
imported.append(node.module or "")
|
||||
assert not any("pipeline" in m or "answer_composer" in m for m in imported)
|
||||
|
||||
def test_never_raises(self, tmp_path):
|
||||
# 任何异常都返回 None/正常 payload，绝不外抛
|
||||
with mock.patch.object(ba, "get_registry", side_effect=RuntimeError("boom")):
|
||||
with mock.patch.dict(ba.os.environ, {ba.ENV_FUSE: "1"}):
|
||||
result = ba.try_bioflow_agent("做基因家族", llm_client=FakeLLMClient())
|
||||
assert result is None
|
||||
@@ -51,9 +51,13 @@ class ToolRunner:
|
||||
user: str,
|
||||
tools: List[Dict[str, Any]],
|
||||
manifests: Dict[str, CapabilityManifest],
|
||||
history: Optional[List[Dict[str, str]]] = None,
|
||||
) -> Tuple[str, Dict[str, Any]]:
|
||||
"""执行工具调用循环。返回 (content, audit)。
|
||||
|
||||
history：多轮记忆（S2），List[{role: user|assistant, content}]，旧→新，
|
||||
不含当前这轮提问；注入为 messages 前置轮次（作为上下文，不参与工具结果回填）。
|
||||
|
||||
audit: {"rounds": n, "tool_calls": k, "used_json_fallback": bool,
|
||||
"tool_names": [...], "errors": [...]}
|
||||
"""
|
||||
@@ -64,7 +68,16 @@ class ToolRunner:
|
||||
"tool_names": [],
|
||||
"errors": [],
|
||||
}
|
||||
messages: List[Dict[str, Any]] = [{"role": "user", "content": user}]
|
||||
messages: List[Dict[str, Any]] = []
|
||||
for h in history or []:
|
||||
if not isinstance(h, dict):
|
||||
continue
|
||||
role = str(h.get("role") or "").strip()
|
||||
content = str(h.get("content") or "").strip()
|
||||
if role not in {"user", "assistant"} or not content:
|
||||
continue
|
||||
messages.append({"role": role, "content": content})
|
||||
messages.append({"role": "user", "content": user})
|
||||
system_text = system
|
||||
for round_index in range(1, self.max_rounds + 1):
|
||||
audit["rounds"] = round_index
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
|
||||
- 独立部署树 bioflow_service/（FR-BIO-028：独立进程 8098 + 独立 SQLite）。
|
||||
- lifespan：init_db（幂等迁移）→ init_context（kernel/catalog/planner/runner）
|
||||
→ recover（服务重启恢复 in-flight 续跑，FR-BIO-031）→ shutdown 清理。
|
||||
→ recover（服务重启恢复 in-flight 续跑，FR-BIO-031）→ MCP session manager
|
||||
（anyio task group，streamable-http 处理请求必需）→ shutdown 清理。
|
||||
- 端点：health（/api/bioflow/health，前端探活 FR-BIO-025）。
|
||||
- 不依赖 BFF/生产服务（8090/8091 等绝不探测/连接）。
|
||||
"""
|
||||
@@ -17,11 +18,18 @@ from fastapi.middleware.cors import CORSMiddleware
|
||||
from .config import get_settings
|
||||
from .context import context, init_context, shutdown_context
|
||||
from .db import close_db, init_db
|
||||
from .mcp_server import extension_mcp, gene_family_mcp, rnaseq_mcp
|
||||
from .routers import bioarchives, bioflow, biojobs, extension
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 三个 MCP streamable-http app（调用即创建 session_manager，供 lifespan 进入 run()）。
|
||||
# 端点地址：/mcp/gene-family、/mcp/rnaseq、/mcp/extension（streamable_http_path="/" + 挂载前缀）。
|
||||
_gene_family_mcp_app = gene_family_mcp.streamable_http_app(streamable_http_path="/")
|
||||
_rnaseq_mcp_app = rnaseq_mcp.streamable_http_app(streamable_http_path="/")
|
||||
_extension_mcp_app = extension_mcp.streamable_http_app(streamable_http_path="/")
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
settings = get_settings()
|
||||
@@ -60,14 +68,21 @@ def create_app() -> FastAPI:
|
||||
logger.info("main: 重启恢复 %d 个 in-flight 任务续跑", len(recovered))
|
||||
except Exception:
|
||||
logger.exception("main: 重启恢复扫描失败（任务不重跑，等待后续操作）")
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await shutdown_context()
|
||||
try:
|
||||
await close_db()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# MCP session manager 的 task group 必须在服务运行期内保持活跃
|
||||
# （streamable-http 处理请求时要求已初始化）。Starlette 不会自动运行
|
||||
# 子 Mount 的 lifespan，故在此显式进入三个 MCP 的 run()。
|
||||
async with gene_family_mcp.session_manager.run():
|
||||
async with rnaseq_mcp.session_manager.run():
|
||||
async with extension_mcp.session_manager.run():
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await shutdown_context()
|
||||
try:
|
||||
await close_db()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
app.router.lifespan_context = _lifespan
|
||||
|
||||
@@ -75,6 +90,12 @@ def create_app() -> FastAPI:
|
||||
app.include_router(biojobs.router)
|
||||
app.include_router(bioarchives.router)
|
||||
app.include_router(extension.router)
|
||||
|
||||
# 生信 MCP 能力外挂（mcp 2.x：MCPServer.streamable_http_app）。
|
||||
# 三个独立 streamable-http 端点，各自装卸（REST 面不受影响）。
|
||||
app.mount("/mcp/gene-family", _gene_family_mcp_app)
|
||||
app.mount("/mcp/rnaseq", _rnaseq_mcp_app)
|
||||
app.mount("/mcp/extension", _extension_mcp_app)
|
||||
return app
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,864 @@
|
||||
"""app/mcp_server.py —— 生信 MCP server（两个 streamable-http 端点）。
|
||||
|
||||
- `/mcp/gene-family`：基因家族能力单元（plan/create/ask + 共享控制组 + 文件组）
|
||||
- `/mcp/rnaseq`：转录组能力单元（plan/create/generate_samples_yaml/ask + 共享控制组 + 文件组）
|
||||
|
||||
设计约束（实施方案 S0）：
|
||||
- 工具只承诺「已受理」，执行进度/日志由 SSE 事件总线直推，不走 MCP。
|
||||
- 每个能力单元只接受本链阶段（create 跨链拒绝 CROSS_CHAIN）。
|
||||
- 人工确认边界（subfamily_confirm / rnaseq_de 实验设计确认）在 job_status 的
|
||||
pending_actions 里标 requires_human=true，工具不提供绕过。
|
||||
- 白名单沿用 runner._allowed_param_names（单一事实源）。
|
||||
- 工具名不外泄由 QA agent（S1）负责，本模块只负责工具语义。
|
||||
|
||||
mcp 2.x 注意：MCPServer（非 FastMCP，来自 mcp.server.mcpserver）；挂载用
|
||||
`mcp.streamable_http_app()`。本文件位于 app/ 层（非 services/），可用 pydantic（mcp
|
||||
自身也依赖 pydantic 构造 schema）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .config import get_settings
|
||||
from .context import load_record
|
||||
from .services.bioflow import extensions as ext_mod
|
||||
from .services.bioflow import store
|
||||
from .services.bioflow.archive import archive_from_job, normalize_archive_name
|
||||
from .services.bioflow.catalog import GENE_FAMILY_CHAIN
|
||||
from .services.bioflow.file_transform import (
|
||||
inspect_file as _ft_inspect_file,
|
||||
transform_file as _ft_transform_file,
|
||||
validate_format,
|
||||
)
|
||||
from .services.bioflow.reports import locate_report
|
||||
from .services.bioflow.runner import RunnerError, ensure_samples_yaml, llm_sample_fallback
|
||||
from .services.bioflow.source_fetcher import FetchError, fetch_source
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---- 能力单元阶段白名单 ----
|
||||
|
||||
GENE_FAMILY_STAGES = set(GENE_FAMILY_CHAIN)
|
||||
|
||||
RNASEQ_STAGES = {
|
||||
"rnaseq_qc", "rnaseq_align", "rnaseq_quant", "rnaseq_sample_qc", "rnaseq_de",
|
||||
"rnaseq_wgcna", "rnaseq_go_enrich", "rnaseq_kegg_enrich", "rnaseq_enrich",
|
||||
"rnaseq_fgsea", "rnaseq_candidate_family", "rnaseq_report",
|
||||
"annotation", "expression",
|
||||
}
|
||||
|
||||
# 人工确认边界：这些阶段 waiting_confirm 时 requires_human=true，工具/QA 不可代确认
|
||||
HUMAN_CONFIRM_STAGES = {"subfamily_confirm", "rnaseq_de"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# context 访问（动态读模块全局，测试 monkeypatch 兼容）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _ctx():
|
||||
from . import context as context_mod
|
||||
|
||||
return context_mod.context
|
||||
|
||||
|
||||
def _job_detail(record: Any) -> dict[str, Any]:
|
||||
stages = []
|
||||
for stage in record.stage_chain:
|
||||
st = record._stage_states.get(stage, {})
|
||||
stages.append({
|
||||
"stage": stage,
|
||||
"status": st.get("status", "waiting"),
|
||||
"params": st.get("params") or {},
|
||||
"attempt": int(st.get("attempt") or 0),
|
||||
"manifest_path": st.get("manifest_path"),
|
||||
"log_tail": st.get("log_tail") or "",
|
||||
"error": st.get("error"),
|
||||
})
|
||||
return {
|
||||
"id": record.id,
|
||||
"session_id": record.session_id,
|
||||
"title": record.title,
|
||||
"status": record.status,
|
||||
"stages": stages,
|
||||
"params": record.params,
|
||||
"auto_continue": record.auto_continue,
|
||||
"archive_id": record.archive_id,
|
||||
"progress": record.progress(),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# pending_actions 派生（job_status 工具返回给 QA agent 的下一步动作）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def derive_pending_actions(record: Any, runner: Any) -> list[dict[str, Any]]:
|
||||
"""由任务/阶段状态派生待办动作（QA agent 据此引导用户，人工边界不可代确认）。
|
||||
|
||||
按阶段链顺序找第一个「卡住」的环节，返回动作清单（通常 0-1 条）。
|
||||
"""
|
||||
if record.status == "paused":
|
||||
return [{"action": "resume", "stage": None, "requires_human": False,
|
||||
"message": "任务已暂停，可恢复"}]
|
||||
if record.status == "done":
|
||||
return [{"action": "archive", "stage": None, "requires_human": False,
|
||||
"message": "任务已完成，可存档复用"}]
|
||||
if record.status == "archived":
|
||||
return [{"action": "none", "stage": None, "requires_human": False,
|
||||
"message": "任务已存档"}]
|
||||
|
||||
for stage in record.stage_chain:
|
||||
status = record._stage_states.get(stage, {}).get("status", "waiting")
|
||||
if status == "await_files":
|
||||
return [{"action": "provide_files", "stage": stage, "requires_human": False,
|
||||
"message": f"阶段 {stage} 等待输入文件"}]
|
||||
if status == "waiting_confirm":
|
||||
human = stage in HUMAN_CONFIRM_STAGES
|
||||
return [{"action": "confirm", "stage": stage, "requires_human": human,
|
||||
"form_defaults": runner._form_defaults(stage, record),
|
||||
"message": (f"阶段 {stage} 需人工确认（不可代确认）" if human
|
||||
else f"阶段 {stage} 等待参数确认")}]
|
||||
if status == "failed":
|
||||
return [{"action": "rerun_or_skip", "stage": stage, "requires_human": False,
|
||||
"message": f"阶段 {stage} 失败，可重跑或跳过"}]
|
||||
# waiting/running/queued/done/skipped → 继续
|
||||
return []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 工具实现（命名 async 函数，带类型提示，供两个 MCPServer 注册）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _plan(message: str, allowed_stages: set[str]) -> dict[str, Any]:
|
||||
from .services.bioflow.planner import PlanCandidate, PlannerError, PlanResult
|
||||
|
||||
ctx = _ctx()
|
||||
if ctx.planner is None:
|
||||
return {"ok": False, "code": "NOT_READY", "message": "规划器未就绪"}
|
||||
try:
|
||||
plan = await ctx.planner.plan(message)
|
||||
except PlannerError as exc:
|
||||
fallback = [s for s in ("identify", "rnaseq_qc") if s in allowed_stages and ctx.catalog.has(s)]
|
||||
plan = PlanResult(candidates=[PlanCandidate(purpose="内置默认流程", stages=fallback)],
|
||||
source="llm_only", message=exc.message)
|
||||
candidates = []
|
||||
for c in plan.candidates:
|
||||
stages = [s for s in c.stages if s in allowed_stages]
|
||||
if stages:
|
||||
candidates.append({"purpose": c.purpose, "stages": stages})
|
||||
return {
|
||||
"ok": True,
|
||||
"candidates": candidates,
|
||||
"source": plan.source,
|
||||
"reuse_archive": plan.reuse_archive_id,
|
||||
"reuse_name": plan.reuse_name,
|
||||
"fallback": plan.message or None,
|
||||
}
|
||||
|
||||
|
||||
async def _create_job(session_id: str, title: str, stages: list[str],
|
||||
params: dict[str, Any] | None, auto_continue: bool,
|
||||
archive_id: str | None, allowed_stages: set[str]) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
unknown = [s for s in stages if s not in allowed_stages]
|
||||
if unknown:
|
||||
return {"ok": False, "code": "CROSS_CHAIN",
|
||||
"message": f"阶段不属于该能力单元: {', '.join(unknown)}"}
|
||||
try:
|
||||
record = await ctx.runner.create_job(
|
||||
session_id, title or "生信任务", stages, params or {},
|
||||
auto_continue=auto_continue, archive_id=archive_id,
|
||||
)
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
ctx.records[record.id] = record
|
||||
return {"ok": True, "data": _job_detail(record)}
|
||||
|
||||
|
||||
async def _job_status(job_id: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
detail = _job_detail(record)
|
||||
detail["pending_actions"] = derive_pending_actions(record, ctx.runner)
|
||||
return {"ok": True, "data": detail}
|
||||
|
||||
|
||||
async def _confirm_stage(job_id: str, stage: str, params: dict[str, Any] | None) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
if stage not in record.stage_chain:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
|
||||
try:
|
||||
await ctx.runner.confirm_stage(record, stage, params or {})
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
|
||||
"status": ctx.runner.stage_status(record, stage)}}
|
||||
|
||||
|
||||
async def _rerun_stage(job_id: str, stage: str, params: dict[str, Any] | None) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
if stage not in record.stage_chain:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
|
||||
try:
|
||||
await ctx.runner.rerun_stage(record, stage, params)
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
|
||||
"status": ctx.runner.stage_status(record, stage)}}
|
||||
|
||||
|
||||
async def _skip_stage(job_id: str, stage: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
if stage not in record.stage_chain:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": f"阶段不存在: {stage}"}
|
||||
try:
|
||||
await ctx.runner.skip_stage(record, stage)
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
return {"ok": True, "data": {"job_id": job_id, "stage": stage,
|
||||
"status": ctx.runner.stage_status(record, stage)}}
|
||||
|
||||
|
||||
async def _pause_job(job_id: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
try:
|
||||
await ctx.runner.pause_job(record)
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
return {"ok": True, "data": {"job_id": job_id, "status": record.status}}
|
||||
|
||||
|
||||
async def _resume_job(job_id: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
try:
|
||||
await ctx.runner.resume_job(record)
|
||||
except RunnerError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
return {"ok": True, "data": {"job_id": job_id, "status": record.status}}
|
||||
|
||||
|
||||
async def _archive_job(job_id: str, name: str, description: str,
|
||||
include_intermediate: bool) -> dict[str, Any]:
|
||||
"""任务存档：打包任务目录 + 落 bio_archives 行（复用 archive_from_job + store.insert_archive）。"""
|
||||
ctx = _ctx()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
archive_name = normalize_archive_name(name)
|
||||
existing = await store.find_archive_by_name(ctx.db, archive_name)
|
||||
if existing is not None:
|
||||
return {"ok": False, "code": "ARCHIVE_EXISTS", "message": f"归档名已存在: {archive_name}"}
|
||||
try:
|
||||
report_path = locate_report(record.workspace_dir)
|
||||
result = await archive_from_job(
|
||||
{
|
||||
"id": record.id, "session_id": record.session_id,
|
||||
"title": record.title, "stage_chain": record.stage_chain,
|
||||
"params": record.params,
|
||||
},
|
||||
record.job_dir,
|
||||
get_settings().archives_dir_abs,
|
||||
name=archive_name,
|
||||
description=description or "",
|
||||
include_intermediate=bool(include_intermediate),
|
||||
report_artifact=str(report_path) if report_path else None,
|
||||
)
|
||||
await store.insert_archive(ctx.db, {
|
||||
"id": result.id,
|
||||
"job_id": record.id,
|
||||
"name": archive_name,
|
||||
"description": description or "",
|
||||
"stage_chain": record.stage_chain,
|
||||
"params": record.params,
|
||||
"report_artifact": result.report_artifact,
|
||||
"disk_path": result.disk_path,
|
||||
"size_bytes": result.size_bytes,
|
||||
})
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"ok": False, "code": "ARCHIVE_FAILED",
|
||||
"message": f"存档失败: {type(exc).__name__}: {exc}"}
|
||||
ctx.events.publish("job_archived", {"job_id": record.id, "archive_id": result.id})
|
||||
return {"ok": True, "data": {
|
||||
"id": result.id,
|
||||
"job_id": record.id,
|
||||
"name": archive_name,
|
||||
"description": description or "",
|
||||
"disk_path": result.disk_path,
|
||||
"size_bytes": result.size_bytes,
|
||||
"report_artifact": result.report_artifact,
|
||||
}}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 扩展能力单元（第三个能力单元 bioflow-extension）：复用 extensions.py 状态机
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_EXT_VIEW_KEYS = (
|
||||
"ext_id", "display_name", "status", "command_template", "original_command",
|
||||
"runtime_image", "requirements", "parameters", "input_roles", "nl_triggers",
|
||||
"analysis_type", "golden", "repair_count", "trial", "validation", "last_build",
|
||||
"created_at", "updated_at",
|
||||
)
|
||||
|
||||
|
||||
def _ext_view(ext: dict[str, Any]) -> dict[str, Any]:
|
||||
"""扩展精简视图（剔除 source_files 等超长字段，供 MCP 工具结果返回）。"""
|
||||
return {k: ext.get(k) for k in _EXT_VIEW_KEYS if k in ext}
|
||||
|
||||
|
||||
def _file_items(files: Any) -> list[tuple[str, bytes]]:
|
||||
"""把 MCP 入参 [{path, content|data}] 转为 [(rel_path, bytes)]。"""
|
||||
out: list[tuple[str, bytes]] = []
|
||||
if not isinstance(files, list):
|
||||
return out
|
||||
for f in files:
|
||||
if not isinstance(f, dict):
|
||||
continue
|
||||
path = str(f.get("path") or f.get("name") or "").strip()
|
||||
if not path:
|
||||
continue
|
||||
content = f.get("content")
|
||||
if content is None:
|
||||
content = f.get("data") or ""
|
||||
if isinstance(content, (bytes, bytearray)):
|
||||
blob = bytes(content)
|
||||
else:
|
||||
blob = str(content).encode("utf-8")
|
||||
out.append((path, blob))
|
||||
return out
|
||||
|
||||
|
||||
def _ext_err(exc: Exception) -> str:
|
||||
if isinstance(exc, ext_mod.ExtensionNotFoundError):
|
||||
return "NOT_FOUND"
|
||||
if isinstance(exc, ext_mod.ExtensionStatusError):
|
||||
return "EXT_STATUS"
|
||||
if isinstance(exc, ext_mod.ExtensionBuildError):
|
||||
return "EXT_BUILD"
|
||||
if isinstance(exc, ext_mod.ExtensionRunError):
|
||||
return "EXT_RUN"
|
||||
return "EXT_ERROR"
|
||||
|
||||
|
||||
async def _create_extension(display_name: str, command: str, runtime_image: str,
|
||||
code_files: Any, data_files: Any) -> dict[str, Any]:
|
||||
code = _file_items(code_files)
|
||||
if not code:
|
||||
return {"ok": False, "code": "VALIDATION", "message": "扩展代码包不能为空"}
|
||||
try:
|
||||
ext = ext_mod.get_store().create_extension(
|
||||
display_name=display_name, command=command, runtime_image=runtime_image or "",
|
||||
code_files=code, data_files=_file_items(data_files),
|
||||
)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _extension_status(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = ext_mod.get_store().load_extension(ext_id)
|
||||
except ext_mod.ExtensionNotFoundError:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "扩展不存在"}
|
||||
return {"ok": True, "data": _ext_view(ext) if ext else None}
|
||||
|
||||
|
||||
async def _build_extension(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = await ext_mod.get_store().build_extension(ext_id)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _trial_run_extension(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = await ext_mod.get_store().trial_run(ext_id)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _validate_extension(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = ext_mod.get_store().validate(ext_id)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _confirm_extension(ext_id: str, nl_triggers: Any, parameters: Any,
|
||||
runtime_image: str) -> dict[str, Any]:
|
||||
triggers = [str(x).strip() for x in nl_triggers if str(x).strip()] if isinstance(nl_triggers, list) else None
|
||||
params = parameters if isinstance(parameters, list) else None
|
||||
try:
|
||||
ext = ext_mod.get_store().confirm(
|
||||
ext_id, nl_triggers=triggers, parameters=params,
|
||||
runtime_image=runtime_image or None,
|
||||
)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _publish_extension(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = ext_mod.get_store().publish(ext_id)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
async def _run_extension(ext_id: str, params: Any, files: Any) -> dict[str, Any]:
|
||||
try:
|
||||
result = await ext_mod.get_store().run(
|
||||
ext_id,
|
||||
params=params if isinstance(params, dict) else None,
|
||||
files=[str(f) for f in files] if isinstance(files, list) else None,
|
||||
)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": result}
|
||||
|
||||
|
||||
async def _repair_extension(ext_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
ext = await ext_mod.get_store().repair(ext_id)
|
||||
except ext_mod.ExtensionError as exc:
|
||||
return {"ok": False, "code": _ext_err(exc), "message": str(exc)}
|
||||
return {"ok": True, "data": _ext_view(ext)}
|
||||
|
||||
|
||||
# ---- 文件登记（复刻 biojobs._register_path 核心，无 FastAPI 依赖）----
|
||||
|
||||
def _blocked_reason(path: Path, settings: Any) -> str | None:
|
||||
s = str(path)
|
||||
for prefix in settings.sensitive_path_prefixes:
|
||||
if s == prefix or s.startswith(prefix + "/"):
|
||||
return f"敏感路径前缀 {prefix}"
|
||||
for pat in settings.sensitive_filename_patterns:
|
||||
if re.match(pat, path.name):
|
||||
return f"敏感文件名 {path.name}"
|
||||
return None
|
||||
|
||||
|
||||
async def _register_job_files(job_id: str, paths: list[str]) -> dict[str, Any]:
|
||||
"""登记服务器/局域网路径（单文件或目录展开）到任务 input。"""
|
||||
ctx = _ctx()
|
||||
settings = get_settings()
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
|
||||
input_dir = record.input_dir
|
||||
input_dir.mkdir(parents=True, exist_ok=True)
|
||||
items: list[dict[str, Any]] = []
|
||||
registered: list[str] = []
|
||||
|
||||
for raw in paths:
|
||||
p = Path(raw).expanduser().resolve()
|
||||
if not p.exists():
|
||||
return {"ok": False, "code": "PATH_NOT_EXIST", "message": f"路径不存在: {raw}"}
|
||||
blocked = _blocked_reason(p, settings)
|
||||
if blocked:
|
||||
return {"ok": False, "code": "PATH_NOT_ALLOWED",
|
||||
"message": f"路径不允许: {raw}（{blocked}）"}
|
||||
files: list[Path] = []
|
||||
if p.is_file():
|
||||
files = [p]
|
||||
else:
|
||||
ext_set = settings.upload_ext_set
|
||||
for f in sorted(p.rglob("*")):
|
||||
if f.is_file() and (not ext_set or f.suffix.lower() in ext_set) and not _blocked_reason(f, settings):
|
||||
files.append(f)
|
||||
if not files:
|
||||
return {"ok": False, "code": "VALIDATION", "message": f"目录内无可登记文件: {raw}"}
|
||||
for f in files:
|
||||
link = input_dir / f.name
|
||||
try:
|
||||
if not link.exists():
|
||||
link.symlink_to(f)
|
||||
except OSError as exc: # noqa: BLE001
|
||||
logger.warning("mcp register symlink 失败: %s -> %s (%s)", f, link, exc)
|
||||
items.append({"file_id": f"{record.id}:{f.name}", "name": f.name,
|
||||
"size": f.stat().st_size, "path": str(f), "source": "server_path"})
|
||||
registered.append(str(f))
|
||||
|
||||
# rnaseq 系：samples.yaml 生成（描述文件优先 + 实况分析；规则不可归纳走 LLM 兜底）
|
||||
try:
|
||||
source_roots = [str(Path(x).parent) for x in paths if Path(x).exists()]
|
||||
extra_mounts = ensure_samples_yaml(input_dir, source_roots=[Path(r) for r in source_roots])
|
||||
if not extra_mounts:
|
||||
try:
|
||||
await llm_sample_fallback(input_dir, [Path(r) for r in source_roots])
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("samples.yaml LLM 兜底失败: %s", exc)
|
||||
if extra_mounts:
|
||||
record.registered_server_paths = list(
|
||||
dict.fromkeys((getattr(record, "registered_server_paths", None) or [])
|
||||
+ [str(Path(m)) for m in extra_mounts]))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("samples.yaml 分析失败: %s", exc)
|
||||
|
||||
if registered:
|
||||
import json
|
||||
record.registered_server_paths = list(
|
||||
dict.fromkeys((getattr(record, "registered_server_paths", None) or []) + registered)
|
||||
)
|
||||
(record.job_dir / "server_registered.json").write_text(
|
||||
json.dumps({"registered_server_paths": record.registered_server_paths},
|
||||
ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
# 自动续跑（await_inputs 任务重推；非等待态 no-op）
|
||||
await ctx.runner.retry_after_files(record)
|
||||
return {"ok": True, "data": {"items": items, "total": len(items)}}
|
||||
|
||||
|
||||
async def _ask_question(question: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
hits: list[dict[str, Any]] = []
|
||||
if ctx.flow_store is not None:
|
||||
try:
|
||||
hits = ctx.flow_store.search(question, top_k=8).get("items") or []
|
||||
except Exception: # noqa: BLE001
|
||||
hits = []
|
||||
citations = []
|
||||
for h in hits:
|
||||
citations.append({
|
||||
"source": h.get("source") or h.get("doc_kind") or "",
|
||||
"title_path": h.get("title_path") or "",
|
||||
"path": h.get("path") or "",
|
||||
"text": str(h.get("text") or "")[:1200],
|
||||
})
|
||||
return {"ok": True, "citations": citations, "degraded": not citations}
|
||||
|
||||
|
||||
async def _list_archives(page: int, page_size: int) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
items, total = await store.list_archives(ctx.db, page=page, page_size=page_size)
|
||||
return {"ok": True, "items": items, "total": total, "page": page, "page_size": page_size}
|
||||
|
||||
|
||||
async def _reuse_archive(name: str) -> dict[str, Any]:
|
||||
ctx = _ctx()
|
||||
arch = await store.find_archive_by_name(ctx.db, (name or "").strip())
|
||||
if arch is None:
|
||||
return {"ok": False, "code": "ARCHIVE_NOT_FOUND", "message": f"未找到匹配存档: {name}"}
|
||||
return {"ok": True, "data": {
|
||||
"id": arch["id"], "name": arch["name"], "stage_chain": arch["stage_chain"],
|
||||
"params": arch["params"], "description": arch["description"],
|
||||
}}
|
||||
|
||||
|
||||
async def _fetch_source_file(job_id: str | None, source: str,
|
||||
credentials: dict[str, Any] | None) -> dict[str, Any]:
|
||||
settings = get_settings()
|
||||
dest_dir: Path
|
||||
if job_id:
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
dest_dir = record.input_dir
|
||||
else:
|
||||
dest_dir = Path(settings.jobs_dir_abs) / "_incoming" / uuid.uuid4().hex[:8]
|
||||
try:
|
||||
result = await fetch_source(source, dest_dir, credentials=credentials)
|
||||
except FetchError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message}
|
||||
result["job_id"] = job_id
|
||||
return result
|
||||
|
||||
|
||||
async def _inspect_file(path: str) -> dict[str, Any]:
|
||||
return _ft_inspect_file(path)
|
||||
|
||||
|
||||
async def _transform_file(src_path: str, target_format: str,
|
||||
column_map: dict[str, str] | None,
|
||||
dest_path: str | None) -> dict[str, Any]:
|
||||
from .services.bioflow.file_transform import TransformError
|
||||
from .services.bioflow.llm import complete_json
|
||||
|
||||
async def llm_adapter(user: str, system: str) -> Any:
|
||||
return await complete_json(system, user, temperature=0.2, max_tokens=2000)
|
||||
|
||||
try:
|
||||
return await _ft_transform_file(
|
||||
src_path, target_format, column_map=column_map, dest_path=dest_path, llm=llm_adapter,
|
||||
)
|
||||
except TransformError as exc:
|
||||
return {"ok": False, "code": exc.code, "message": exc.message, "details": exc.details}
|
||||
|
||||
|
||||
async def _generate_samples_yaml(job_id: str, sample_description: str) -> dict[str, Any]:
|
||||
import yaml
|
||||
|
||||
from .services.bioflow.llm import complete_json
|
||||
|
||||
record = await load_record(job_id)
|
||||
if record is None:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "任务不存在"}
|
||||
system = (
|
||||
"你是 RNA-seq 样本配置助手。根据用户的自然语言描述，输出 samples.yaml 结构 JSON："
|
||||
'{"samples": {"样本名": {"r1": "fastq路径", "r2": "fastq路径", "condition": "...", ...}}}。'
|
||||
"只输出 JSON。"
|
||||
)
|
||||
try:
|
||||
raw = await complete_json(system, sample_description, temperature=0.2, max_tokens=2000)
|
||||
samples = raw.get("samples") if isinstance(raw, dict) else None
|
||||
if not isinstance(samples, dict) or not samples:
|
||||
return {"ok": False, "code": "EMPTY_SAMPLES", "message": "未能从描述解析出样本"}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"ok": False, "code": "SAMPLES_GEN_FAILED", "message": f"样本配置生成失败: {exc}"}
|
||||
record.input_dir.mkdir(parents=True, exist_ok=True)
|
||||
out = record.input_dir / "samples.yaml"
|
||||
out.write_text(yaml.safe_dump({"samples": samples}, allow_unicode=True), encoding="utf-8")
|
||||
v = validate_format(out, "samples.yaml")
|
||||
if not v.get("ok"):
|
||||
out.unlink(missing_ok=True)
|
||||
return {"ok": False, "code": "VALIDATION_FAILED", "message": f"样本配置校验失败: {v.get('reason')}"}
|
||||
return {"ok": True, "data": {"path": str(out), "samples": samples, "validated": v}}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 能力单元工具（命名 async，带类型提示 + 中文描述）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def plan_gene_family(message: str) -> dict[str, Any]:
|
||||
"""根据用户自然语言需求规划基因家族分析候选路径（仅基因家族链阶段）。"""
|
||||
return await _plan(message, GENE_FAMILY_STAGES)
|
||||
|
||||
|
||||
async def create_gene_family_job(session_id: str, stages: list[str], title: str = "",
|
||||
params: dict[str, Any] | None = None,
|
||||
auto_continue: bool = False,
|
||||
archive_id: str | None = None) -> dict[str, Any]:
|
||||
"""按确认的阶段链创建基因家族任务（stages 只能是基因家族链阶段，跨链拒绝）。"""
|
||||
return await _create_job(session_id, title, stages, params, auto_continue, archive_id, GENE_FAMILY_STAGES)
|
||||
|
||||
|
||||
async def ask_gene_family_question(question: str) -> dict[str, Any]:
|
||||
"""回答基因家族流程/参数/生物学意义类问题（流程知识库检索，带引用）。"""
|
||||
return await _ask_question(question)
|
||||
|
||||
|
||||
async def plan_rnaseq(message: str) -> dict[str, Any]:
|
||||
"""根据用户自然语言需求规划转录组分析候选路径（仅转录组链阶段）。"""
|
||||
return await _plan(message, RNASEQ_STAGES)
|
||||
|
||||
|
||||
async def create_rnaseq_job(session_id: str, stages: list[str], title: str = "",
|
||||
params: dict[str, Any] | None = None,
|
||||
auto_continue: bool = False,
|
||||
archive_id: str | None = None) -> dict[str, Any]:
|
||||
"""按确认的阶段链创建转录组任务（stages 只能是转录组链阶段，跨链拒绝）。"""
|
||||
return await _create_job(session_id, title, stages, params, auto_continue, archive_id, RNASEQ_STAGES)
|
||||
|
||||
|
||||
async def generate_samples_yaml(job_id: str, sample_description: str) -> dict[str, Any]:
|
||||
"""把用户的自然语言样本描述（样本名/对照处理组/重复/单双端等）解析为 samples.yaml 写入任务 input。"""
|
||||
return await _generate_samples_yaml(job_id, sample_description)
|
||||
|
||||
|
||||
async def ask_rnaseq_question(question: str) -> dict[str, Any]:
|
||||
"""回答转录组流程/参数/实验设计类问题（流程知识库检索，带引用）。"""
|
||||
return await _ask_question(question)
|
||||
|
||||
|
||||
async def job_status(job_id: str) -> dict[str, Any]:
|
||||
"""查询生信任务状态：任务/各阶段状态 + 下一步待办动作(pending_actions)。卡住或需确认时用本工具查真相。"""
|
||||
return await _job_status(job_id)
|
||||
|
||||
|
||||
async def confirm_stage(job_id: str, stage: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""确认某阶段参数并执行。仅 waiting_confirm 状态可确认；参数名须在白名单内(未知参数被拒)。人工确认边界阶段(subfamily_confirm/rnaseq_de)不可代用户确认。"""
|
||||
return await _confirm_stage(job_id, stage, params)
|
||||
|
||||
|
||||
async def register_job_files(job_id: str, paths: list[str]) -> dict[str, Any]:
|
||||
"""把服务器/局域网路径(单文件或目录)登记为任务输入文件。目录按扩展名白名单展开。"""
|
||||
return await _register_job_files(job_id, paths)
|
||||
|
||||
|
||||
async def rerun_stage(job_id: str, stage: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""重跑失败阶段(带参数则先等确认)。仅 failed 状态可重跑。"""
|
||||
return await _rerun_stage(job_id, stage, params)
|
||||
|
||||
|
||||
async def skip_stage(job_id: str, stage: str) -> dict[str, Any]:
|
||||
"""跳过失败阶段并推进后续。仅 failed 状态可跳过。"""
|
||||
return await _skip_stage(job_id, stage)
|
||||
|
||||
|
||||
async def pause_job(job_id: str) -> dict[str, Any]:
|
||||
"""暂停运行中的任务(running 阶段会立即 kill 容器)。"""
|
||||
return await _pause_job(job_id)
|
||||
|
||||
|
||||
async def resume_job(job_id: str) -> dict[str, Any]:
|
||||
"""恢复已暂停的任务(从下一阶段继续,确认点不跳过)。"""
|
||||
return await _resume_job(job_id)
|
||||
|
||||
|
||||
async def list_archives(page: int = 1, page_size: int = 20) -> dict[str, Any]:
|
||||
"""列出历史存档任务(供复用参考)。"""
|
||||
return await _list_archives(page, page_size)
|
||||
|
||||
|
||||
async def reuse_archive(name: str) -> dict[str, Any]:
|
||||
"""按名称匹配存档,返回其阶段链+参数(供复用发起)。"""
|
||||
return await _reuse_archive(name)
|
||||
|
||||
|
||||
async def archive_job(job_id: str, name: str, description: str = "",
|
||||
include_intermediate: bool = False) -> dict[str, Any]:
|
||||
"""把已完成任务存档(打包结果/日志/事件 + 落档登记);重名拒绝(ARCHIVE_EXISTS)。"""
|
||||
return await _archive_job(job_id, name, description, include_intermediate)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 扩展能力单元工具（第三个能力单元 bioflow-extension）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def create_extension(display_name: str, command: str, runtime_image: str = "",
|
||||
code_files: list[dict[str, Any]] | None = None,
|
||||
data_files: list[dict[str, Any]] | None = None) -> dict[str, Any]:
|
||||
"""创建扩展草稿(draft)：上传代码文件清单(每项 {path, content 文本}) + 运行命令 + 显示名。"""
|
||||
return await _create_extension(display_name, command, runtime_image, code_files, data_files)
|
||||
|
||||
|
||||
async def extension_status(ext_id: str) -> dict[str, Any]:
|
||||
"""查询扩展详情与状态(draft/build/trial/validated/confirming/published)。"""
|
||||
return await _extension_status(ext_id)
|
||||
|
||||
|
||||
async def build_extension(ext_id: str) -> dict[str, Any]:
|
||||
"""AI 转换扩展(原始命令→command_template+参数+dependencies)；LLM 失败自动回退规则版。"""
|
||||
return await _build_extension(ext_id)
|
||||
|
||||
|
||||
async def trial_run_extension(ext_id: str) -> dict[str, Any]:
|
||||
"""试运行扩展(docker 执行；未接入执行器时返回 EXT_RUN 503 语义)。"""
|
||||
return await _trial_run_extension(ext_id)
|
||||
|
||||
|
||||
async def validate_extension(ext_id: str) -> dict[str, Any]:
|
||||
"""校验扩展(金标/试运行校验)。仅 built/trial_passed 可校验。"""
|
||||
return await _validate_extension(ext_id)
|
||||
|
||||
|
||||
async def confirm_extension(ext_id: str, nl_triggers: list[str] | None = None,
|
||||
parameters: list[dict[str, Any]] | None = None,
|
||||
runtime_image: str = "") -> dict[str, Any]:
|
||||
"""确认扩展参数/触发词(用户可编辑 nl_triggers)；仅 validated/trial_passed 可确认。"""
|
||||
return await _confirm_extension(ext_id, nl_triggers, parameters, runtime_image)
|
||||
|
||||
|
||||
async def publish_extension(ext_id: str) -> dict[str, Any]:
|
||||
"""发布扩展(仅 published 状态可被自然语言触发运行)。"""
|
||||
return await _publish_extension(ext_id)
|
||||
|
||||
|
||||
async def run_extension(ext_id: str, params: dict[str, Any] | None = None,
|
||||
files: list[str] | None = None) -> dict[str, Any]:
|
||||
"""正式运行已发布扩展(同步返回 run_result)。"""
|
||||
return await _run_extension(ext_id, params, files)
|
||||
|
||||
|
||||
async def repair_extension(ext_id: str) -> dict[str, Any]:
|
||||
"""AI 修复扩展(试运行失败上下文→LLM 修复 command_template/依赖；LLM 失败回退规则版)。"""
|
||||
return await _repair_extension(ext_id)
|
||||
|
||||
|
||||
async def fetch_source_file(source: str, job_id: str | None = None,
|
||||
credentials: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""从 web(http/https 流式)、OSS(oss:// 或预签名 URL)或局域网路径拉取源文件到任务 input;下载前检查磁盘余量并记录 provenance。"""
|
||||
return await _fetch_source_file(job_id, source, credentials)
|
||||
|
||||
|
||||
async def inspect_file(path: str) -> dict[str, Any]:
|
||||
"""探测文件格式(扩展名仅参考):返回预览/列结构/大小/格式猜测/乱码检测,供判断是否需要预处理。"""
|
||||
return await _inspect_file(path)
|
||||
|
||||
|
||||
async def transform_file(src_path: str, target_format: str,
|
||||
column_map: dict[str, str] | None = None,
|
||||
dest_path: str | None = None) -> dict[str, Any]:
|
||||
"""把非标准文件(xlsx/csv/tsv/txt)转换为 fasta/gff3/tsv/samples.yaml;LLM 定列映射规则+脚本执行,结果过格式验证器。"""
|
||||
return await _transform_file(src_path, target_format, column_map, dest_path)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 构建三个 MCPServer（基因家族 / 转录组 / 扩展）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def build_mcp_servers() -> tuple[Any, Any, Any]:
|
||||
from mcp.server.mcpserver import MCPServer
|
||||
|
||||
gf = MCPServer(
|
||||
"bioflow-gene-family",
|
||||
title="PlantAI 基因家族分析",
|
||||
description="基因家族分析能力单元：路径规划/建任务/确认参数/查状态/文件获取与预处理。",
|
||||
)
|
||||
rn = MCPServer(
|
||||
"bioflow-rnaseq",
|
||||
title="PlantAI 转录组分析",
|
||||
description="转录组分析能力单元：路径规划/样本配置生成/建任务/确认参数/查状态/文件获取与预处理。",
|
||||
)
|
||||
ext = MCPServer(
|
||||
"bioflow-extension",
|
||||
title="PlantAI 用户自定义扩展分析",
|
||||
description="用户自定义扩展分析能力单元：创建/转换/试运行/校验/确认/发布/运行。",
|
||||
)
|
||||
|
||||
gf.tool(name="plan_gene_family", description="根据用户自然语言需求规划基因家族分析候选路径（仅基因家族链阶段）。返回 2-3 条候选路径供用户确认。")(plan_gene_family)
|
||||
gf.tool(name="create_gene_family_job", description="按确认的阶段链创建基因家族任务。stages 只能是基因家族链阶段（跨链拒绝）。")(create_gene_family_job)
|
||||
gf.tool(name="ask_gene_family_question", description="回答基因家族流程/参数/生物学意义类问题（流程知识库检索，带引用）。")(ask_gene_family_question)
|
||||
|
||||
rn.tool(name="plan_rnaseq", description="根据用户自然语言需求规划转录组分析候选路径（仅转录组链阶段）。返回候选路径供用户确认。")(plan_rnaseq)
|
||||
rn.tool(name="create_rnaseq_job", description="按确认的阶段链创建转录组任务。stages 只能是转录组链阶段（跨链拒绝）。")(create_rnaseq_job)
|
||||
rn.tool(name="generate_samples_yaml", description="把用户的自然语言样本描述解析为 samples.yaml 写入任务 input。")(generate_samples_yaml)
|
||||
rn.tool(name="ask_rnaseq_question", description="回答转录组流程/参数/实验设计类问题（流程知识库检索，带引用）。")(ask_rnaseq_question)
|
||||
|
||||
ext.tool(name="create_extension", description="创建扩展草稿（draft）：上传代码文件清单(每项 {path, content 文本}) + 运行命令 + 显示名。")(create_extension)
|
||||
ext.tool(name="extension_status", description="查询扩展详情与状态（draft/build/trial/validated/confirming/published）。")(extension_status)
|
||||
ext.tool(name="build_extension", description="AI 转换扩展（原始命令→command_template+参数+依赖）；LLM 失败自动回退规则版。")(build_extension)
|
||||
ext.tool(name="trial_run_extension", description="试运行扩展（docker 执行；未接入执行器时返回 EXT_RUN）。")(trial_run_extension)
|
||||
ext.tool(name="validate_extension", description="校验扩展（金标/试运行校验）。仅 built/trial_passed 可校验。")(validate_extension)
|
||||
ext.tool(name="confirm_extension", description="确认扩展参数/触发词（用户可编辑 nl_triggers）；仅 validated/trial_passed 可确认。")(confirm_extension)
|
||||
ext.tool(name="publish_extension", description="发布扩展（仅 published 状态可被自然语言触发运行）。")(publish_extension)
|
||||
ext.tool(name="run_extension", description="正式运行已发布扩展（同步返回 run_result）。")(run_extension)
|
||||
ext.tool(name="repair_extension", description="AI 修复扩展（试运行失败上下文→LLM 修复；LLM 失败回退规则版）。")(repair_extension)
|
||||
|
||||
shared = [
|
||||
job_status, confirm_stage, register_job_files, rerun_stage, skip_stage,
|
||||
pause_job, resume_job, list_archives, reuse_archive, archive_job,
|
||||
fetch_source_file, inspect_file, transform_file,
|
||||
]
|
||||
for mcp in (gf, rn):
|
||||
for fn in shared:
|
||||
mcp.tool()(fn)
|
||||
return gf, rn, ext
|
||||
|
||||
|
||||
gene_family_mcp, rnaseq_mcp, extension_mcp = build_mcp_servers()
|
||||
@@ -0,0 +1,438 @@
|
||||
"""app/services/bioflow/file_transform.py —— 非标准文件预处理（探测 + 转换 + 校验）。
|
||||
|
||||
FR-BIO-014 文件医生闭环：
|
||||
- inspect_file：内容嗅探（扩展名仅参考），返回结构化摘要（预览行/列结构/大小/
|
||||
格式猜测/乱码检测/与目标阶段期望格式的差异）。
|
||||
- transform_file：LLM 抽取列映射/清洗规则 + 规则脚本执行，把 xlsx/csv/tsv/txt 转换
|
||||
为 fasta/gff3/tsv/samples.yaml；转换结果过格式验证器硬校验，失败返回原因供追问
|
||||
循环（≤3 次由上层 QA 控制）。
|
||||
|
||||
服务层纪律：纯 dataclass/dict/stdlib + yaml/openpyxl，无 pydantic。
|
||||
LLM 可注入（默认走 .llm.complete_json），测试注入 fake。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any, Awaitable, Callable
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PREVIEW_LINES = 10
|
||||
PREVIEW_BYTES = 4000
|
||||
|
||||
# 支持的目标格式
|
||||
TARGET_FORMATS = ("fasta", "gff3", "tsv", "samples.yaml")
|
||||
|
||||
# 乱码检测：非 ASCII 可打印且非 CJK 的异常字节比例阈值
|
||||
GARBLED_RATIO = 0.30
|
||||
|
||||
|
||||
class TransformError(Exception):
|
||||
def __init__(self, code: str, message: str, details: dict[str, Any] | None = None) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.details = details or {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 探测
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _guess_format(name: str, head: bytes) -> str:
|
||||
lower = name.lower()
|
||||
if lower.endswith((".fa", ".fasta", ".faa", ".fna", ".pep", ".cds")):
|
||||
return "fasta"
|
||||
if lower.endswith((".gff", ".gff3")):
|
||||
return "gff3"
|
||||
if lower.endswith((".gtf",)):
|
||||
return "gtf"
|
||||
if lower.endswith((".vcf", ".vcf.gz")):
|
||||
return "vcf"
|
||||
if lower.endswith((".tsv",)):
|
||||
return "tsv"
|
||||
if lower.endswith((".csv",)):
|
||||
return "csv"
|
||||
if lower.endswith((".xlsx", ".xls")):
|
||||
return "xlsx"
|
||||
if lower.endswith((".yaml", ".yml")):
|
||||
return "yaml"
|
||||
# 内容嗅探兜底
|
||||
text = head.decode("utf-8", errors="replace")
|
||||
if text.lstrip().startswith(">"):
|
||||
return "fasta"
|
||||
if "\t" in text and len(text.splitlines()[0].split("\t")) >= 3:
|
||||
return "tsv"
|
||||
return "txt"
|
||||
|
||||
|
||||
def _read_text(path: Path) -> str:
|
||||
raw = path.read_bytes()
|
||||
for enc in ("utf-8", "gb18030", "latin-1"):
|
||||
try:
|
||||
return raw.decode(enc)
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
return raw.decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
def _garbled_ratio(text: str) -> float:
|
||||
"""异常字符比例：控制字符 + 替换符 + 私有区（不含常规空白）。"""
|
||||
if not text:
|
||||
return 0.0
|
||||
bad = 0
|
||||
for ch in text:
|
||||
o = ord(ch)
|
||||
if o == 0xFFFD or (o < 32 and ch not in "\t\n\r") or (0xE000 <= o <= 0xF8FF):
|
||||
bad += 1
|
||||
return bad / len(text)
|
||||
|
||||
|
||||
def _preview(text: str) -> list[str]:
|
||||
return text.splitlines()[:PREVIEW_LINES]
|
||||
|
||||
|
||||
def _columns(path: Path, fmt: str) -> list[str]:
|
||||
"""表格列名提取（tsv/csv/xlsx）。"""
|
||||
try:
|
||||
if fmt == "xlsx":
|
||||
import openpyxl
|
||||
wb = openpyxl.load_workbook(path, read_only=True, data_only=True)
|
||||
try:
|
||||
ws = wb.active
|
||||
row = next(ws.iter_rows(min_row=1, max_row=1, values_only=True), None)
|
||||
return [str(c or "").strip() for c in (row or [])]
|
||||
finally:
|
||||
wb.close()
|
||||
if fmt in ("tsv", "csv"):
|
||||
text = _read_text(path)
|
||||
reader = csv.reader(io.StringIO(text), delimiter="\t" if fmt == "tsv" else ",")
|
||||
first = next(reader, None)
|
||||
return [str(c or "").strip() for c in (first or [])]
|
||||
except Exception as exc: # noqa: BLE001 - 列提取失败降级空
|
||||
logger.warning("file_transform: 列提取失败 %s: %s", path, exc)
|
||||
return []
|
||||
|
||||
|
||||
def inspect_file(path: str | Path, *, preview_lines: int = PREVIEW_LINES) -> dict[str, Any]:
|
||||
"""内容嗅探摘要（扩展名仅参考；失败返回 ok=False + 原因，不抛）。"""
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": f"文件不存在: {path}"}
|
||||
try:
|
||||
size = p.stat().st_size
|
||||
head = p.read_bytes()[:PREVIEW_BYTES]
|
||||
fmt = _guess_format(p.name, head)
|
||||
text = _read_text(p) if size <= 4 * 1024 * 1024 else _read_text_preview(p)
|
||||
return {
|
||||
"ok": True,
|
||||
"path": str(p),
|
||||
"name": p.name,
|
||||
"ext": p.suffix.lower() or "",
|
||||
"size": size,
|
||||
"format": fmt,
|
||||
"preview": _preview(text)[:preview_lines],
|
||||
"columns": _columns(p, fmt),
|
||||
"garbled": _garbled_ratio(text),
|
||||
"line_count": text.count("\n") + 1 if text else 0,
|
||||
}
|
||||
except OSError as exc:
|
||||
return {"ok": False, "code": "READ_ERROR", "message": f"文件读取失败: {exc}"}
|
||||
|
||||
|
||||
def _read_text_preview(path: Path) -> str:
|
||||
"""大文件只读头部做预览/乱码检测，避免整读内存。"""
|
||||
with path.open("rb") as fh:
|
||||
head = fh.read(PREVIEW_BYTES)
|
||||
for enc in ("utf-8", "gb18030", "latin-1"):
|
||||
try:
|
||||
return head.decode(enc)
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
return head.decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 校验器
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def validate_fasta(path: str | Path) -> dict[str, Any]:
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
return {"ok": False, "reason": "文件不存在"}
|
||||
text = _read_text(p)
|
||||
records = re.findall(r"^>.*$", text, flags=re.M)
|
||||
if not records:
|
||||
return {"ok": False, "reason": "无 FASTA 记录（缺 '>' 头行）"}
|
||||
# 序列非空：头行之间至少存在非空白序列行
|
||||
body = re.sub(r"^>.*$", "", text, flags=re.M)
|
||||
if not body.strip():
|
||||
return {"ok": False, "reason": "FASTA 记录缺少序列"}
|
||||
return {"ok": True, "records": len(records)}
|
||||
|
||||
|
||||
def validate_gff3(path: str | Path) -> dict[str, Any]:
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
return {"ok": False, "reason": "文件不存在"}
|
||||
bad = 0
|
||||
total = 0
|
||||
for line in _read_text(p).splitlines():
|
||||
if not line.strip() or line.startswith("#"):
|
||||
continue
|
||||
total += 1
|
||||
cols = line.split("\t")
|
||||
if len(cols) != 9:
|
||||
bad += 1
|
||||
continue
|
||||
try:
|
||||
int(cols[3])
|
||||
int(cols[4])
|
||||
except ValueError:
|
||||
bad += 1
|
||||
continue
|
||||
if cols[6] not in ("+", "-", "."):
|
||||
bad += 1
|
||||
if total == 0:
|
||||
return {"ok": False, "reason": "无 GFF3 数据行"}
|
||||
if bad:
|
||||
return {"ok": False, "reason": f"{bad}/{total} 行列数或字段非法"}
|
||||
return {"ok": True, "records": total}
|
||||
|
||||
|
||||
def validate_tsv(path: str | Path) -> dict[str, Any]:
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
return {"ok": False, "reason": "文件不存在"}
|
||||
widths: set[int] = set()
|
||||
for line in _read_text(p).splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
widths.add(len(line.split("\t")))
|
||||
if len(widths) > 1:
|
||||
return {"ok": False, "reason": f"列数不一致: {sorted(widths)}"}
|
||||
if not widths:
|
||||
return {"ok": False, "reason": "空文件"}
|
||||
return {"ok": True, "columns": widths.pop()}
|
||||
|
||||
|
||||
def validate_samples_yaml(path: str | Path) -> dict[str, Any]:
|
||||
import yaml
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
return {"ok": False, "reason": "文件不存在"}
|
||||
try:
|
||||
raw = yaml.safe_load(_read_text(p))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"ok": False, "reason": f"YAML 解析失败: {exc}"}
|
||||
samples = (raw or {}).get("samples") if isinstance(raw, dict) else None
|
||||
if not isinstance(samples, dict) or not samples:
|
||||
return {"ok": False, "reason": "缺 samples 映射或为空"}
|
||||
for name, entry in samples.items():
|
||||
if not isinstance(entry, dict) or not (entry.get("r1") or entry.get("r2")):
|
||||
return {"ok": False, "reason": f"样本 {name} 缺 r1/r2"}
|
||||
return {"ok": True, "samples": len(samples)}
|
||||
|
||||
|
||||
_VALIDATORS = {
|
||||
"fasta": validate_fasta,
|
||||
"gff3": validate_gff3,
|
||||
"tsv": validate_tsv,
|
||||
"samples.yaml": validate_samples_yaml,
|
||||
}
|
||||
|
||||
|
||||
def validate_format(path: str | Path, target_format: str) -> dict[str, Any]:
|
||||
fn = _VALIDATORS.get(target_format)
|
||||
if fn is None:
|
||||
return {"ok": False, "reason": f"不支持的目标格式: {target_format}"}
|
||||
return fn(path)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 转换
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
LLMFn = Callable[[str, str], Awaitable[Any]]
|
||||
|
||||
|
||||
def _read_table(path: Path) -> tuple[list[str], list[dict[str, str]]]:
|
||||
"""读表格（tsv/csv/xlsx）→ (表头, 行 dict 列表)。"""
|
||||
fmt = _guess_format(path.name, path.read_bytes()[:64])
|
||||
if fmt == "xlsx":
|
||||
import openpyxl
|
||||
wb = openpyxl.load_workbook(path, read_only=True, data_only=True)
|
||||
try:
|
||||
ws = wb.active
|
||||
rows = [list(r) for r in ws.iter_rows(values_only=True)]
|
||||
finally:
|
||||
wb.close()
|
||||
if not rows:
|
||||
return [], []
|
||||
headers = [str(c or "").strip() for c in rows[0]]
|
||||
return headers, [dict(zip(headers, ["" if v is None else str(v).strip() for v in r])) for r in rows[1:]]
|
||||
text = _read_text(path)
|
||||
delimiter = "\t" if fmt == "tsv" else ","
|
||||
reader = csv.reader(io.StringIO(text), delimiter=delimiter)
|
||||
rows = list(reader)
|
||||
if not rows:
|
||||
return [], []
|
||||
headers = [str(c or "").strip() for c in rows[0]]
|
||||
return headers, [dict(zip(headers, ["" if v is None else str(v).strip() for v in r])) for r in rows[1:]]
|
||||
|
||||
|
||||
def _default_column_map(headers: list[str], target: str) -> dict[str, str]:
|
||||
"""按表头名启发式推断列映射（名称/序列/基因名等）。"""
|
||||
m: dict[str, str] = {}
|
||||
for h in headers:
|
||||
lh = h.lower()
|
||||
if target == "fasta":
|
||||
if not m.get("seq") and any(k in lh for k in ("seq", "sequence", "序列", "蛋白", "pep", "cds")):
|
||||
m["seq"] = h
|
||||
elif not m.get("name") and any(k in lh for k in ("name", "id", "gene", "名称", "基因", "基因名", "accession")):
|
||||
m["name"] = h
|
||||
elif target == "samples.yaml":
|
||||
if "r1" in lh or "read1" in lh or "forward" in lh:
|
||||
m.setdefault("r1", h)
|
||||
elif "r2" in lh or "read2" in lh or "reverse" in lh:
|
||||
m.setdefault("r2", h)
|
||||
elif "run" in lh or "sample" in lh or "样本" in lh or "name" in lh:
|
||||
m.setdefault("name", h)
|
||||
return m
|
||||
|
||||
|
||||
async def _llm_column_map(headers: list[str], target: str, llm: LLMFn, sample_rows: list[dict[str, str]]) -> dict[str, str]:
|
||||
"""LLM 抽取列映射（启发式不充分时）。返回 {目标字段: 列名}。"""
|
||||
import json as _json
|
||||
prompt = (
|
||||
f"你是生信数据整理助手。下表头 {_json.dumps(headers, ensure_ascii=False)} "
|
||||
f"需转换为 {target} 格式。样本行：{_json.dumps(sample_rows[:3], ensure_ascii=False)[:2000]}。"
|
||||
"请输出列映射 JSON，键为目标字段名，值为表头列名。"
|
||||
)
|
||||
try:
|
||||
result = await llm(prompt, "只输出 JSON 对象，如 {\"name\":\"基因名\",\"seq\":\"序列\"}。")
|
||||
if isinstance(result, dict):
|
||||
return {str(k): str(v) for k, v in result.items() if str(v) in headers}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("file_transform: LLM 列映射失败 %s", exc)
|
||||
return {}
|
||||
|
||||
|
||||
def _write_fasta(rows: list[dict[str, str]], name_col: str, seq_col: str, dest: Path) -> int:
|
||||
n = 0
|
||||
with dest.open("w", encoding="utf-8") as fh:
|
||||
for r in rows:
|
||||
name = r.get(name_col) or f"seq{n + 1}"
|
||||
seq = (r.get(seq_col) or "").replace("\n", "").replace("\r", "")
|
||||
if not seq:
|
||||
continue
|
||||
fh.write(f">{name}\n{seq}\n")
|
||||
n += 1
|
||||
return n
|
||||
|
||||
|
||||
def _write_tsv(rows: list[dict[str, str]], cols: list[str], dest: Path) -> int:
|
||||
n = 0
|
||||
with dest.open("w", encoding="utf-8", newline="") as fh:
|
||||
w = csv.writer(fh, delimiter="\t")
|
||||
w.writerow(cols)
|
||||
for r in rows:
|
||||
w.writerow([r.get(c, "") for c in cols])
|
||||
n += 1
|
||||
return n
|
||||
|
||||
|
||||
def _write_samples_yaml(rows: list[dict[str, str]], name_col: str, r1_col: str, r2_col: str, dest: Path) -> int:
|
||||
import yaml
|
||||
samples: dict[str, dict[str, str]] = {}
|
||||
for r in rows:
|
||||
name = r.get(name_col) or ""
|
||||
r1 = r.get(r1_col) or ""
|
||||
r2 = r.get(r2_col) or ""
|
||||
if not name and not r1 and not r2:
|
||||
continue
|
||||
entry: dict[str, str] = {}
|
||||
if r1:
|
||||
entry["r1"] = r1
|
||||
if r2:
|
||||
entry["r2"] = r2
|
||||
# 额外列作为 meta（run/condition/group 等）
|
||||
for k, v in r.items():
|
||||
if v and k not in (name_col, r1_col, r2_col):
|
||||
entry[k] = v
|
||||
samples[name or Path(r1).stem] = entry
|
||||
dest.write_text(yaml.safe_dump({"samples": samples}, allow_unicode=True), encoding="utf-8")
|
||||
return len(samples)
|
||||
|
||||
|
||||
async def transform_file(
|
||||
src_path: str | Path,
|
||||
target_format: str,
|
||||
*,
|
||||
column_map: dict[str, str] | None = None,
|
||||
dest_path: str | Path | None = None,
|
||||
llm: LLMFn | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""非标准文件转换（LLM 列映射兜底 + 规则执行 + 格式校验）。
|
||||
|
||||
column_map：用户/QA 确认的 {目标字段: 源列名}（可部分），缺列经启发式/LLM 补齐。
|
||||
返回 ok=True + path/size/validated/records，或 ok=False + code/message（供追问）。
|
||||
"""
|
||||
src = Path(src_path)
|
||||
if not src.is_file():
|
||||
raise TransformError("NOT_FOUND", f"源文件不存在: {src_path}")
|
||||
if target_format not in TARGET_FORMATS:
|
||||
raise TransformError("BAD_FORMAT", f"不支持的目标格式: {target_format}")
|
||||
|
||||
dest = Path(dest_path) if dest_path else src.with_suffix("").with_name(
|
||||
f"{src.stem}.{target_format.replace('.', '_')}"
|
||||
)
|
||||
|
||||
headers, rows = _read_table(src)
|
||||
if not headers:
|
||||
raise TransformError("EMPTY_TABLE", "表格为空或无表头，无法转换")
|
||||
cm = dict(column_map or {})
|
||||
cm = {**cm, **_default_column_map(headers, target_format)}
|
||||
|
||||
if target_format == "fasta":
|
||||
if "seq" not in cm or "name" not in cm:
|
||||
if llm is not None:
|
||||
inferred = await _llm_column_map(headers, target_format, llm, rows)
|
||||
cm = {**cm, **inferred}
|
||||
if "seq" not in cm or "name" not in cm:
|
||||
raise TransformError("MISSING_COLUMNS", "无法确定序列列/名称列，请确认列映射")
|
||||
n = _write_fasta(rows, cm["name"], cm["seq"], dest)
|
||||
elif target_format == "tsv":
|
||||
cols = [cm.get("name")] if cm.get("name") else []
|
||||
cols = [c for c in (cols + [h for h in headers if h not in cols]) if c]
|
||||
n = _write_tsv(rows, cols, dest)
|
||||
elif target_format == "samples.yaml":
|
||||
if "r1" not in cm and "r2" not in cm:
|
||||
if llm is not None:
|
||||
inferred = await _llm_column_map(headers, target_format, llm, rows)
|
||||
cm = {**cm, **inferred}
|
||||
if "r1" not in cm and "r2" not in cm:
|
||||
raise TransformError("MISSING_COLUMNS", "无法确定 fastq 路径列（r1/r2），请确认列映射")
|
||||
n = _write_samples_yaml(rows, cm.get("name", ""), cm.get("r1", ""), cm.get("r2", ""), dest)
|
||||
else: # gff3：表格式 GFF 需严格 9 列映射，LLM 定规则后由上层脚本执行
|
||||
raise TransformError("UNSUPPORTED", f"{target_format} 需 9 列映射，请经 LLM 定规则后转换")
|
||||
|
||||
validated = validate_format(dest, target_format)
|
||||
if not validated.get("ok"):
|
||||
# 校验失败：删除产物，返回原因供追问
|
||||
dest.unlink(missing_ok=True)
|
||||
raise TransformError("VALIDATION_FAILED", f"转换结果校验失败: {validated.get('reason')}", validated)
|
||||
return {
|
||||
"ok": True,
|
||||
"path": str(dest),
|
||||
"name": dest.name,
|
||||
"size": dest.stat().st_size,
|
||||
"format": target_format,
|
||||
"records": n,
|
||||
"validated": validated,
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
"""app/services/bioflow/source_fetcher.py —— 源文件拉取（web/OSS/局域网）+ provenance。
|
||||
|
||||
三类来源（FR-BIO-013 网络源扩展）：
|
||||
- web：http/https。httpx 流式下载，不设大小上限，超时 300s，重定向跟随。
|
||||
- OSS：`oss://bucket/key` 或预签名 URL。无 OSS SDK 依赖：`oss://` 经可配 endpoint
|
||||
（BIOFLOW_OSS_ENDPOINT）转 http 下载；预签名 URL 直接走 http 通道。
|
||||
- 局域网：本机已挂载/可达路径直读（不复制，登记原路径）；或 `[user@]host:/abs/path`
|
||||
经 scp 拉取（密钥认证优先；密码需本机有 sshpass）。凭据仅单次内存使用，不落盘。
|
||||
|
||||
- 下载前检查磁盘剩余空间（Content-Length 已知时硬校验；否则保底阈值 512MB）。
|
||||
- 每次拉取记录 provenance（来源/哈希/时间/kind）到返回值并落 sidecar .provenance.json，
|
||||
不记明文密码。
|
||||
服务层纪律：纯 dataclass/dict/stdlib + httpx，无 pydantic。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import urllib.parse
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ALLOWED_SCHEMES = ("http", "https")
|
||||
DEFAULT_TIMEOUT = 300.0
|
||||
MIN_FREE_FALLBACK = 512 * 1024 * 1024 # 未知大小时保底磁盘余量检查（512MB）
|
||||
CHUNK = 1024 * 1024
|
||||
|
||||
|
||||
class FetchError(Exception):
|
||||
"""源文件拉取错误（带错误码；MCP 工具层转自然语言提示）。"""
|
||||
|
||||
def __init__(self, code: str, message: str) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.message = message
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
h = hashlib.sha256()
|
||||
with path.open("rb") as fh:
|
||||
for block in iter(lambda: fh.read(CHUNK), b""):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def _safe_name(name: str) -> str:
|
||||
"""URL 解码后取文件名（去路径分隔符），空则占位。"""
|
||||
name = urllib.parse.unquote(name or "")
|
||||
base = Path(name).name or "download"
|
||||
return base
|
||||
|
||||
|
||||
def _name_from_url(url: str) -> str:
|
||||
return _safe_name(urllib.parse.urlparse(url).path.rsplit("/", 1)[-1])
|
||||
|
||||
|
||||
def check_disk_space(dest_dir: str | Path, needed: int) -> tuple[bool, int]:
|
||||
"""检查目标目录所在盘剩余空间是否 ≥ needed 字节。返回 (够, 剩余字节)。"""
|
||||
dest_dir = Path(dest_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
free = shutil.disk_usage(dest_dir).free
|
||||
return (free >= needed, free)
|
||||
|
||||
|
||||
def _write_provenance(dest: Path, source: str, kind: str, sha256: str, extra: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""落 provenance sidecar（{dest}.provenance.json），返回 provenance dict。"""
|
||||
prov: dict[str, Any] = {
|
||||
"source": source,
|
||||
"kind": kind,
|
||||
"sha256": sha256,
|
||||
"fetched_at": _now_iso(),
|
||||
}
|
||||
if extra:
|
||||
prov.update(extra)
|
||||
sidecar = dest.parent / f"{dest.name}.provenance.json"
|
||||
try:
|
||||
sidecar.write_text(json.dumps(prov, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
except OSError: # noqa: BLE001 - provenance 落盘失败不阻断拉取
|
||||
logger.warning("source_fetcher: provenance 落盘失败 %s", sidecar)
|
||||
return prov
|
||||
|
||||
|
||||
# ---- web ----
|
||||
|
||||
async def fetch_web(url: str, dest_dir: str | Path, *, timeout: float = DEFAULT_TIMEOUT) -> dict[str, Any]:
|
||||
"""http/https 流式下载（不设大小上限；Content-Length 已知时下载前查磁盘余量）。"""
|
||||
scheme = urllib.parse.urlparse(url).scheme.lower()
|
||||
if scheme not in ALLOWED_SCHEMES:
|
||||
raise FetchError("BAD_SCHEME", f"仅支持 http/https 协议: {url}")
|
||||
dest_dir = Path(dest_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
name = _name_from_url(url)
|
||||
dest = dest_dir / name
|
||||
|
||||
async with httpx.AsyncClient(follow_redirects=True) as client:
|
||||
try:
|
||||
async with client.stream("GET", url, timeout=httpx.Timeout(connect=30.0, read=timeout, write=60.0, pool=10.0)) as resp:
|
||||
if resp.status_code >= 400:
|
||||
raise FetchError("HTTP_ERROR", f"下载失败 HTTP {resp.status_code}: {url}")
|
||||
total = int(resp.headers.get("content-length") or 0)
|
||||
if total:
|
||||
ok, free = check_disk_space(dest_dir, total)
|
||||
if not ok:
|
||||
raise FetchError("DISK_FULL", f"磁盘空间不足：需 {total} 字节，剩余 {free} 字节")
|
||||
written = 0
|
||||
with dest.open("wb") as fh:
|
||||
async for chunk in resp.aiter_bytes(CHUNK):
|
||||
fh.write(chunk)
|
||||
written += len(chunk)
|
||||
if total and written != total:
|
||||
dest.unlink(missing_ok=True)
|
||||
raise FetchError("TRUNCATED", f"下载不完整：期望 {total} 字节，实收 {written} 字节")
|
||||
except FetchError:
|
||||
raise
|
||||
except httpx.HTTPError as exc:
|
||||
dest.unlink(missing_ok=True)
|
||||
raise FetchError("NETWORK_ERROR", f"网络错误: {exc}") from exc
|
||||
|
||||
size = dest.stat().st_size
|
||||
sha = _sha256(dest)
|
||||
prov = _write_provenance(dest, url, "web", sha)
|
||||
return {"ok": True, "path": str(dest), "name": name, "size": size, "kind": "web", "provenance": prov}
|
||||
|
||||
|
||||
# ---- OSS ----
|
||||
|
||||
def _resolve_oss_url(spec: str) -> str:
|
||||
"""oss://bucket/key → http 下载 URL（public-read 或预签名；无 SDK，需 endpoint 配置）。
|
||||
|
||||
环境键：BIOFLOW_OSS_ENDPOINT（如 oss-cn-hangzhou.aliyuncs.com，或内网 endpoint）。
|
||||
预签名 URL（含 aliyuncs.com 的 http/https）原样返回，走 web 通道。
|
||||
"""
|
||||
if not spec.startswith("oss://"):
|
||||
return spec # 已是 http/https 预签名 URL
|
||||
rest = spec[len("oss://"):]
|
||||
bucket, _, key = rest.partition("/")
|
||||
endpoint = (os.environ.get("BIOFLOW_OSS_ENDPOINT") or "").strip().rstrip("/")
|
||||
if not bucket or not key:
|
||||
raise FetchError("BAD_OSS", f"OSS 地址格式非法（应为 oss://bucket/key）: {spec}")
|
||||
if not endpoint:
|
||||
raise FetchError("OSS_UNCONFIGURED", "未配置 BIOFLOW_OSS_ENDPOINT，无法解析 oss:// 地址；请提供预签名 URL")
|
||||
# 内网/公网 endpoint 统一拼 http（https 由 endpoint 决定；公网 OSS 建议 https）
|
||||
scheme = "https" if "aliyuncs.com" in endpoint else "http"
|
||||
return f"{scheme}://{bucket}.{endpoint}/{key}"
|
||||
|
||||
|
||||
async def fetch_oss(spec: str, dest_dir: str | Path, *, timeout: float = DEFAULT_TIMEOUT) -> dict[str, Any]:
|
||||
url = _resolve_oss_url(spec)
|
||||
result = await fetch_web(url, dest_dir, timeout=timeout)
|
||||
result["kind"] = "oss"
|
||||
# 重写 provenance kind（sidecar 已由 fetch_web 写为 web；此处修正）
|
||||
if result.get("provenance"):
|
||||
result["provenance"]["kind"] = "oss"
|
||||
result["provenance"]["oss_spec"] = spec
|
||||
try:
|
||||
sidecar = Path(result["path"]).parent / f"{Path(result['path']).name}.provenance.json"
|
||||
sidecar.write_text(json.dumps(result["provenance"], ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
except OSError:
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
# ---- 局域网 ----
|
||||
|
||||
def _parse_lan_remote(spec: str) -> tuple[str, str] | None:
|
||||
"""解析 `[user@]host:/abs/path` 远端形态；非远端返回 None。"""
|
||||
if ":" not in spec:
|
||||
return None
|
||||
# 排除协议（http:// / oss:// 含 ://，已在上层分流）
|
||||
if "://" in spec:
|
||||
return None
|
||||
host_part, _, path_part = spec.partition(":")
|
||||
if not path_part.startswith("/"):
|
||||
return None
|
||||
return host_part, path_part
|
||||
|
||||
|
||||
async def _scp_fetch(host_part: str, remote_path: str, dest_dir: Path, credentials: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""scp 拉取远端文件（密钥认证；密码需本机 sshpass）。"""
|
||||
creds = credentials or {}
|
||||
user = str(creds.get("user") or "").strip()
|
||||
password = str(creds.get("password") or "").strip()
|
||||
port = str(creds.get("port") or "22").strip()
|
||||
target = f"{user}@{host_part}" if user else host_part
|
||||
name = _safe_name(remote_path.rsplit("/", 1)[-1])
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cmd: list[str] = []
|
||||
if password:
|
||||
if shutil.which("sshpass") is None:
|
||||
raise FetchError("SSHPASS_MISSING", "本机未安装 sshpass，无法使用密码拉取；请改用密钥认证或提供已挂载路径")
|
||||
cmd += ["sshpass", "-p", password]
|
||||
cmd += ["scp", "-P", port, "-o", "StrictHostKeyChecking=accept-new", "-o", "ConnectTimeout=30",
|
||||
f"{target}:{remote_path}", str(dest_dir / name)]
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
|
||||
_out, err = await asyncio.wait_for(proc.communicate(), timeout=DEFAULT_TIMEOUT)
|
||||
except asyncio.TimeoutError as exc:
|
||||
raise FetchError("LAN_TIMEOUT", f"局域网拉取超时: {target}:{remote_path}") from exc
|
||||
except OSError as exc:
|
||||
raise FetchError("LAN_ERROR", f"局域网拉取失败: {exc}") from exc
|
||||
if proc.returncode != 0:
|
||||
raise FetchError("LAN_ERROR", f"scp 失败: {(err or b'').decode(errors='replace')[:500]}")
|
||||
dest = dest_dir / name
|
||||
if not dest.is_file():
|
||||
raise FetchError("LAN_ERROR", f"scp 未产出文件: {dest}")
|
||||
size = dest.stat().st_size
|
||||
sha = _sha256(dest)
|
||||
prov = _write_provenance(dest, f"{target}:{remote_path}", "lan", sha)
|
||||
return {"ok": True, "path": str(dest), "name": name, "size": size, "kind": "lan", "provenance": prov}
|
||||
|
||||
|
||||
async def fetch_lan(spec: str, dest_dir: str | Path, *, credentials: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""局域网路径：本机已挂载/可达直读（返回原路径，不复制）；远端走 scp。"""
|
||||
dest_dir = Path(dest_dir)
|
||||
remote = _parse_lan_remote(spec)
|
||||
if remote is not None:
|
||||
return await _scp_fetch(remote[0], remote[1], dest_dir, credentials)
|
||||
# 本机路径：直读（存在即成功；不复制，登记语义由 register_job_files 处理）
|
||||
p = Path(spec).expanduser()
|
||||
if not p.exists():
|
||||
raise FetchError("PATH_NOT_EXIST", f"局域网路径不存在或不可达: {spec}")
|
||||
if p.is_dir():
|
||||
raise FetchError("IS_DIR", f"路径是目录（仅支持单文件拉取）: {spec}")
|
||||
sha = _sha256(p)
|
||||
prov = _write_provenance(p, str(p), "lan", sha)
|
||||
return {"ok": True, "path": str(p), "name": p.name, "size": p.stat().st_size, "kind": "lan", "provenance": prov}
|
||||
|
||||
|
||||
# ---- 总入口 ----
|
||||
|
||||
async def fetch_source(
|
||||
source: str,
|
||||
dest_dir: str | Path,
|
||||
*,
|
||||
credentials: dict[str, Any] | None = None,
|
||||
timeout: float = DEFAULT_TIMEOUT,
|
||||
) -> dict[str, Any]:
|
||||
"""按来源形态分流拉取（web/OSS/局域网/本机路径）。
|
||||
|
||||
credentials（可选，仅局域网 scp 用）：{"user": ..., "password": ..., "port": ...}。
|
||||
凭据不落盘、不进入 provenance。
|
||||
"""
|
||||
source = (source or "").strip()
|
||||
if not source:
|
||||
raise FetchError("EMPTY_SOURCE", "源地址为空")
|
||||
lower = source.lower()
|
||||
# 协议白名单：含 "://" 的 URL 只允许 http/https/oss（其余如 ftp/file/sftp 拒绝）
|
||||
if "://" in source and not lower.startswith(("http://", "https://", "oss://")):
|
||||
raise FetchError("BAD_SCHEME", f"不支持的协议（仅 http/https/oss）: {source}")
|
||||
if lower.startswith(("http://", "https://")):
|
||||
if lower.startswith("oss://") or "aliyuncs.com" in lower:
|
||||
return await fetch_oss(source, dest_dir, timeout=timeout)
|
||||
return await fetch_web(source, dest_dir, timeout=timeout)
|
||||
if lower.startswith("oss://"):
|
||||
return await fetch_oss(source, dest_dir, timeout=timeout)
|
||||
return await fetch_lan(source, dest_dir, credentials=credentials)
|
||||
@@ -0,0 +1,403 @@
|
||||
"""tests/test_file_transform.py —— 非标准文件预处理（探测 + 转换 + 校验）。
|
||||
|
||||
分层标记 -m bioflow_unit。覆盖（S0 验收）：
|
||||
- inspect_file：fasta/tsv 探测、乱码检测、列结构提取。
|
||||
- transform_file：xlsx→fasta（显式列映射）、csv→fasta（启发式列映射）、
|
||||
samples.yaml 转换 + 校验。
|
||||
- 校验失败 → TransformError VALIDATION_FAILED（追问循环依据）。
|
||||
- 缺列映射且无 LLM → MISSING_COLUMNS。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.bioflow.file_transform import (
|
||||
TransformError,
|
||||
_garbled_ratio,
|
||||
inspect_file,
|
||||
transform_file,
|
||||
validate_fasta,
|
||||
validate_format,
|
||||
validate_gff3,
|
||||
validate_samples_yaml,
|
||||
validate_tsv,
|
||||
)
|
||||
|
||||
|
||||
def _run(coro):
|
||||
return asyncio.new_event_loop().run_until_complete(coro)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# inspect_file 探测
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_inspect_file_fasta(tmp_path):
|
||||
f = tmp_path / "proteins.fa"
|
||||
f.write_text(">SPL1\nMKTA\n>SPL2\nMKTB\n")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "fasta"
|
||||
assert r["size"] > 0
|
||||
|
||||
|
||||
def test_inspect_file_tsv_columns(tmp_path):
|
||||
f = tmp_path / "samples.tsv"
|
||||
f.write_text("run\tfastq_1\tfastq_2\nSRR1\ta_1.fq\ta_2.fq\n")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "tsv"
|
||||
assert r["columns"] == ["run", "fastq_1", "fastq_2"]
|
||||
|
||||
|
||||
def test_inspect_file_garbled(tmp_path):
|
||||
f = tmp_path / "messy.txt"
|
||||
f.write_bytes(b"\x00\x01\x02\x03\xff\xfe garbage \x00\x00")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["garbled"] > 0.3
|
||||
|
||||
|
||||
def test_garbled_ratio_clean():
|
||||
assert _garbled_ratio("ATCGATCG\n>seq\n") < 0.01
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# transform_file
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_transform_xlsx_to_fasta(tmp_path):
|
||||
import openpyxl
|
||||
src = tmp_path / "spl_list.xlsx"
|
||||
wb = openpyxl.Workbook()
|
||||
ws = wb.active
|
||||
ws.append(["基因名", "序列", "备注"])
|
||||
ws.append(["SPL1", "ATGGCTA", "水稻"])
|
||||
ws.append(["SPL2", "ATGGCTB", "水稻"])
|
||||
wb.save(src)
|
||||
|
||||
r = _run(transform_file(src, "fasta", column_map={"name": "基因名", "seq": "序列"}))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
assert r["validated"]["ok"] is True
|
||||
text = Path(r["path"]).read_text()
|
||||
assert ">SPL1" in text and "ATGGCTA" in text
|
||||
|
||||
|
||||
def test_transform_csv_to_fasta_heuristic(tmp_path):
|
||||
src = tmp_path / "genes.csv"
|
||||
src.write_text("gene_id,protein_sequence\nG1,MKTA\nG2,MKTB\n")
|
||||
r = _run(transform_file(src, "fasta"))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
|
||||
|
||||
def test_transform_csv_to_samples_yaml(tmp_path):
|
||||
src = tmp_path / "design.csv"
|
||||
src.write_text("sample,fastq_1,fastq_2,condition\nA,a_1.fq,a_2.fq,ctrl\nB,b_1.fq,b_2.fq,treated\n")
|
||||
r = _run(transform_file(src, "samples.yaml", column_map={"name": "sample", "r1": "fastq_1", "r2": "fastq_2"}))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
assert r["validated"]["ok"] is True
|
||||
import yaml
|
||||
samples = yaml.safe_load(Path(r["path"]).read_text())["samples"]
|
||||
assert "A" in samples and samples["A"]["r1"] == "a_1.fq"
|
||||
|
||||
|
||||
def test_transform_validation_failed(tmp_path):
|
||||
# seq 列全空 → 产物无序列 → 校验失败
|
||||
src = tmp_path / "bad.csv"
|
||||
src.write_text("name,seq\nG1,\nG2,\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta", column_map={"name": "name", "seq": "seq"}))
|
||||
assert ei.value.code == "VALIDATION_FAILED"
|
||||
|
||||
|
||||
def test_transform_missing_columns(tmp_path):
|
||||
# 无列映射、无 LLM、表头无启发式命中 → MISSING_COLUMNS
|
||||
src = tmp_path / "unknown.csv"
|
||||
src.write_text("col_a,col_b\n1,2\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta"))
|
||||
assert ei.value.code == "MISSING_COLUMNS"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 校验器
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_validate_fasta(tmp_path):
|
||||
f = tmp_path / "ok.fa"
|
||||
f.write_text(">a\nACGT\n")
|
||||
assert validate_fasta(f)["ok"] is True
|
||||
|
||||
|
||||
def test_validate_fasta_empty(tmp_path):
|
||||
f = tmp_path / "bad.fa"
|
||||
f.write_text("no records here\n")
|
||||
assert validate_fasta(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_tsv_inconsistent(tmp_path):
|
||||
f = tmp_path / "bad.tsv"
|
||||
f.write_text("a\tb\tc\n1\t2\n")
|
||||
assert validate_tsv(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_samples_yaml(tmp_path):
|
||||
f = tmp_path / "s.yaml"
|
||||
f.write_text("samples:\n A:\n r1: a.fq\n r2: a2.fq\n")
|
||||
assert validate_samples_yaml(f)["ok"] is True
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强（1）inspect_file：NOT_FOUND / xlsx 列 / csv 格式 / 大文件 / 内容嗅探
|
||||
# ===========================================================================
|
||||
|
||||
def test_inspect_file_not_found():
|
||||
r = inspect_file("/no/such/file.fa")
|
||||
assert r["ok"] is False
|
||||
assert r["code"] == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_inspect_file_xlsx_columns(tmp_path):
|
||||
import openpyxl
|
||||
f = tmp_path / "spl.xlsx"
|
||||
wb = openpyxl.Workbook()
|
||||
ws = wb.active
|
||||
ws.append(["基因名", "序列"])
|
||||
ws.append(["SPL1", "MKTA"])
|
||||
wb.save(f)
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "xlsx"
|
||||
assert r["columns"] == ["基因名", "序列"]
|
||||
|
||||
|
||||
def test_inspect_file_csv_format(tmp_path):
|
||||
f = tmp_path / "genes.csv"
|
||||
f.write_text("id,seq\nG1,MKTA\n")
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "csv"
|
||||
assert r["columns"] == ["id", "seq"]
|
||||
|
||||
|
||||
def test_inspect_file_large_uses_preview(tmp_path):
|
||||
f = tmp_path / "big.fa"
|
||||
f.write_bytes(b">seq\nACGT\n" + b"A" * (4 * 1024 * 1024))
|
||||
r = inspect_file(f)
|
||||
assert r["ok"] is True
|
||||
assert r["format"] == "fasta"
|
||||
|
||||
|
||||
def test_guess_format_content_sniffing_fasta(tmp_path):
|
||||
from app.services.bioflow.file_transform import _guess_format
|
||||
assert _guess_format("unknown.dat", b">seq1\nACGT\n") == "fasta"
|
||||
|
||||
|
||||
def test_guess_format_content_sniffing_tsv(tmp_path):
|
||||
from app.services.bioflow.file_transform import _guess_format
|
||||
assert _guess_format("unknown.dat", b"a\tb\tc\n1\t2\t3\n") == "tsv"
|
||||
|
||||
|
||||
def test_garbled_ratio_empty():
|
||||
assert _garbled_ratio("") == 0.0
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强（2）transform_file：NOT_FOUND / BAD_FORMAT / EMPTY_TABLE / tsv / gff3 /
|
||||
# samples.yaml 启发式 / LLM 兜底 / LLM 异常兜底
|
||||
# ===========================================================================
|
||||
|
||||
def test_transform_not_found(tmp_path):
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(tmp_path / "nope.csv", "fasta"))
|
||||
assert ei.value.code == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_transform_bad_format(tmp_path):
|
||||
src = tmp_path / "x.csv"
|
||||
src.write_text("a,b\n1,2\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "bogus"))
|
||||
assert ei.value.code == "BAD_FORMAT"
|
||||
|
||||
|
||||
def test_transform_empty_table(tmp_path):
|
||||
src = tmp_path / "empty.csv"
|
||||
src.write_text("")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta"))
|
||||
assert ei.value.code == "EMPTY_TABLE"
|
||||
|
||||
|
||||
def test_transform_to_tsv(tmp_path):
|
||||
src = tmp_path / "genes.csv"
|
||||
src.write_text("gene_id,protein_sequence\nG1,MKTA\nG2,MKTB\n")
|
||||
r = _run(transform_file(src, "tsv"))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
assert r["validated"]["ok"] is True
|
||||
|
||||
|
||||
def test_transform_to_gff3_unsupported(tmp_path):
|
||||
src = tmp_path / "x.csv"
|
||||
src.write_text("a,b\n1,2\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "gff3"))
|
||||
assert ei.value.code == "UNSUPPORTED"
|
||||
|
||||
|
||||
def test_transform_samples_yaml_heuristic(tmp_path):
|
||||
src = tmp_path / "design.csv"
|
||||
src.write_text("name,r1,r2,condition\nA,a_1.fq,a_2.fq,ctrl\nB,b_1.fq,b_2.fq,trt\n")
|
||||
r = _run(transform_file(src, "samples.yaml"))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
|
||||
|
||||
def test_transform_samples_yaml_missing_columns(tmp_path):
|
||||
src = tmp_path / "unknown.csv"
|
||||
src.write_text("col_a,col_b\n1,2\n")
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "samples.yaml"))
|
||||
assert ei.value.code == "MISSING_COLUMNS"
|
||||
|
||||
|
||||
def test_transform_fasta_llm_column_map(tmp_path):
|
||||
src = tmp_path / "candidates.csv"
|
||||
src.write_text("col_a,col_b\nSPL1,ATGGCTA\nSPL2,ATGGCTB\n")
|
||||
|
||||
async def fake_llm(user, system):
|
||||
return {"name": "col_a", "seq": "col_b"}
|
||||
|
||||
r = _run(transform_file(src, "fasta", llm=fake_llm))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 2
|
||||
|
||||
|
||||
def test_transform_fasta_llm_exception_falls_back_missing(tmp_path):
|
||||
src = tmp_path / "candidates.csv"
|
||||
src.write_text("col_a,col_b\nSPL1,ATGGCTA\n")
|
||||
|
||||
async def fake_llm(user, system):
|
||||
raise RuntimeError("llm down")
|
||||
|
||||
with pytest.raises(TransformError) as ei:
|
||||
_run(transform_file(src, "fasta", llm=fake_llm))
|
||||
assert ei.value.code == "MISSING_COLUMNS"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强（3）校验器：gff3 各失败态 / samples.yaml 缺 r1r2 / YAML 解析失败 /
|
||||
# tsv 空文件 / validate_format 未知格式
|
||||
# ===========================================================================
|
||||
|
||||
def test_validate_gff3_ok(tmp_path):
|
||||
f = tmp_path / "ok.gff3"
|
||||
f.write_text("chr1\tsrc\tgene\t1\t100\t.\t+\t.\tID=x\n")
|
||||
assert validate_gff3(f)["ok"] is True
|
||||
|
||||
|
||||
def test_validate_gff3_no_data(tmp_path):
|
||||
f = tmp_path / "empty.gff3"
|
||||
f.write_text("# just a comment\n")
|
||||
assert validate_gff3(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_gff3_bad_columns(tmp_path):
|
||||
f = tmp_path / "bad.gff3"
|
||||
f.write_text("chr1\tsrc\tgene\t1\t100\n") # 5 列 ≠ 9
|
||||
assert validate_gff3(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_gff3_bad_strand(tmp_path):
|
||||
f = tmp_path / "bad.gff3"
|
||||
f.write_text("chr1\tsrc\tgene\t1\t100\t.\tx\t.\tID=x\n") # 第 7 列非法
|
||||
assert validate_gff3(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_gff3_bad_int(tmp_path):
|
||||
f = tmp_path / "bad.gff3"
|
||||
f.write_text("chr1\tsrc\tgene\tabc\t100\t.\t+\t.\tID=x\n") # 起止非整数
|
||||
assert validate_gff3(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_fasta_no_sequence(tmp_path):
|
||||
f = tmp_path / "bad.fa"
|
||||
f.write_text(">a\n")
|
||||
assert validate_fasta(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_samples_yaml_missing_r1r2(tmp_path):
|
||||
f = tmp_path / "s.yaml"
|
||||
f.write_text("samples:\n A:\n condition: ctrl\n")
|
||||
assert validate_samples_yaml(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_samples_yaml_parse_fail(tmp_path):
|
||||
f = tmp_path / "s.yaml"
|
||||
f.write_text("samples: [unclosed\n")
|
||||
assert validate_samples_yaml(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_tsv_empty(tmp_path):
|
||||
f = tmp_path / "empty.tsv"
|
||||
f.write_text("")
|
||||
assert validate_tsv(f)["ok"] is False
|
||||
|
||||
|
||||
def test_validate_format_unknown(tmp_path):
|
||||
f = tmp_path / "x.txt"
|
||||
f.write_text("x")
|
||||
r = validate_format(f, "bogus")
|
||||
assert r["ok"] is False
|
||||
assert "不支持" in r["reason"]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强（4）校验器缺失文件 / 格式猜测扩展名 / samples 非 dict / 空行跳过
|
||||
# ===========================================================================
|
||||
|
||||
def test_validators_missing_file(tmp_path):
|
||||
missing = tmp_path / "nope"
|
||||
assert validate_fasta(missing)["ok"] is False
|
||||
assert validate_gff3(missing)["ok"] is False
|
||||
assert validate_tsv(missing)["ok"] is False
|
||||
assert validate_samples_yaml(missing)["ok"] is False
|
||||
|
||||
|
||||
def test_guess_format_all_extensions():
|
||||
from app.services.bioflow.file_transform import _guess_format
|
||||
assert _guess_format("a.gff3", b"") == "gff3"
|
||||
assert _guess_format("a.gff", b"") == "gff3"
|
||||
assert _guess_format("a.gtf", b"") == "gtf"
|
||||
assert _guess_format("a.vcf", b"") == "vcf"
|
||||
assert _guess_format("a.vcf.gz", b"") == "vcf"
|
||||
assert _guess_format("a.yaml", b"") == "yaml"
|
||||
assert _guess_format("a.yml", b"") == "yaml"
|
||||
assert _guess_format("a.xls", b"") == "xlsx"
|
||||
assert _guess_format("a.faa", b"") == "fasta"
|
||||
assert _guess_format("a.fna", b"") == "fasta"
|
||||
assert _guess_format("a.pep", b"") == "fasta"
|
||||
assert _guess_format("a.cds", b"") == "fasta"
|
||||
assert _guess_format("a.dat", b"no marker") == "txt"
|
||||
|
||||
|
||||
def test_validate_samples_yaml_not_dict(tmp_path):
|
||||
f = tmp_path / "s.yaml"
|
||||
f.write_text("samples: just_a_string\n")
|
||||
assert validate_samples_yaml(f)["ok"] is False
|
||||
|
||||
|
||||
def test_transform_samples_yaml_skips_empty_rows(tmp_path):
|
||||
src = tmp_path / "d.csv"
|
||||
src.write_text("name,r1,r2\n,,\nA,a.fq,a2.fq\n")
|
||||
r = _run(transform_file(src, "samples.yaml"))
|
||||
assert r["ok"] is True
|
||||
assert r["records"] == 1
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,455 @@
|
||||
"""tests/test_source_fetcher.py —— 源文件拉取（web/OSS/局域网）+ 磁盘余量 + provenance。
|
||||
|
||||
分层标记 -m bioflow_unit。覆盖（S0 验收）：
|
||||
- web http 流式下载（本地 http.server 实测 httpx 通道）+ provenance 落盘。
|
||||
- 协议白名单（ftp:// 拒绝 BAD_SCHEME）。
|
||||
- oss:// 无 endpoint 拒绝 / 预签名 URL 直通 web / endpoint 解析。
|
||||
- 局域网本机路径直读；远端 host:/path 解析。
|
||||
- 磁盘余量不足 DISK_FULL（check_disk_space 纯函数）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import http.server
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.bioflow.source_fetcher import (
|
||||
FetchError,
|
||||
_parse_lan_remote,
|
||||
_resolve_oss_url,
|
||||
check_disk_space,
|
||||
fetch_lan,
|
||||
fetch_source,
|
||||
fetch_web,
|
||||
)
|
||||
|
||||
|
||||
def _run(coro):
|
||||
return asyncio.new_event_loop().run_until_complete(coro)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 本地 http 服务器（测 httpx 通道，不依赖外网）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.fixture()
|
||||
def http_url(tmp_path):
|
||||
content = b">seq1\nACGTACGT\n>seq2\nTGCATGCA\n"
|
||||
(tmp_path / "test.fa").write_bytes(content)
|
||||
handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=str(tmp_path))
|
||||
server = http.server.HTTPServer(("127.0.0.1", 0), handler)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
yield f"http://127.0.0.1:{server.server_address[1]}/test.fa"
|
||||
server.shutdown()
|
||||
|
||||
|
||||
def test_fetch_web_download_and_provenance(http_url, tmp_path):
|
||||
out = tmp_path / "out"
|
||||
r = _run(fetch_web(http_url, out))
|
||||
assert r["ok"] is True
|
||||
assert r["name"] == "test.fa"
|
||||
assert r["size"] > 0
|
||||
assert r["kind"] == "web"
|
||||
assert r["provenance"]["source"] == http_url
|
||||
assert len(r["provenance"]["sha256"]) == 64
|
||||
assert (out / "test.fa").read_text().startswith(">seq1")
|
||||
# sidecar provenance
|
||||
assert (out / "test.fa.provenance.json").is_file()
|
||||
|
||||
|
||||
def test_fetch_web_http_error(http_url, tmp_path):
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_web(http_url + "/missing", tmp_path / "out"))
|
||||
assert ei.value.code == "HTTP_ERROR"
|
||||
|
||||
|
||||
def test_fetch_source_bad_scheme(tmp_path):
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_source("ftp://host/x.fa", tmp_path))
|
||||
assert ei.value.code == "BAD_SCHEME"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# OSS
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_oss_url_no_endpoint_rejected():
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_resolve_oss_url("oss://bucket/key.fa")
|
||||
assert ei.value.code == "OSS_UNCONFIGURED"
|
||||
|
||||
|
||||
def test_oss_url_endpoint_resolve(monkeypatch):
|
||||
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-cn-hangzhou.aliyuncs.com")
|
||||
url = _resolve_oss_url("oss://seq-bucket/data/x.fa")
|
||||
assert url == "https://seq-bucket.oss-cn-hangzhou.aliyuncs.com/data/x.fa"
|
||||
|
||||
|
||||
def test_oss_presigned_passthrough():
|
||||
url = _resolve_oss_url("https://bucket.oss-cn-hangzhou.aliyuncs.com/x.fa?sig=abc")
|
||||
assert url.startswith("https://")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 局域网 / 本机路径
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_parse_lan_remote():
|
||||
assert _parse_lan_remote("192.168.1.5:/data/x.fa") == ("192.168.1.5", "/data/x.fa")
|
||||
assert _parse_lan_remote("xzq@192.168.1.5:/data/x.fa") == ("xzq@192.168.1.5", "/data/x.fa")
|
||||
assert _parse_lan_remote("/local/path/x.fa") is None
|
||||
assert _parse_lan_remote("http://x/y") is None
|
||||
|
||||
|
||||
def test_fetch_lan_local_path(tmp_path):
|
||||
f = tmp_path / "genome.fa"
|
||||
f.write_text(">g1\nACGT\n")
|
||||
r = _run(fetch_lan(str(f), tmp_path / "out"))
|
||||
assert r["ok"] is True
|
||||
assert r["kind"] == "lan"
|
||||
assert r["path"] == str(f)
|
||||
assert r["provenance"]["sha256"]
|
||||
|
||||
|
||||
def test_fetch_lan_missing_path(tmp_path):
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("/no/such/path.fa", tmp_path))
|
||||
assert ei.value.code == "PATH_NOT_EXIST"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 磁盘余量
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_check_disk_space_insufficient(tmp_path):
|
||||
ok, free = check_disk_space(tmp_path, 10 ** 18) # 1EB，必不足
|
||||
assert ok is False
|
||||
assert free < 10 ** 18
|
||||
|
||||
|
||||
def test_check_disk_space_sufficient(tmp_path):
|
||||
ok, _free = check_disk_space(tmp_path, 1) # 1 字节
|
||||
assert ok is True
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强（1）web：BAD_SCHEME / 磁盘满 DISK_FULL / 断点 TRUNCATED / NETWORK_ERROR
|
||||
# ===========================================================================
|
||||
|
||||
def test_fetch_web_bad_scheme(tmp_path):
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_web("ftp://host/x.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "BAD_SCHEME"
|
||||
|
||||
|
||||
def test_fetch_web_disk_full(http_url, tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
monkeypatch.setattr(sf, "check_disk_space", lambda dest_dir, needed: (False, 0))
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_web(http_url, tmp_path / "out"))
|
||||
assert ei.value.code == "DISK_FULL"
|
||||
|
||||
|
||||
def test_fetch_web_truncated(tmp_path, monkeypatch):
|
||||
"""断点/不完整下载：aiter_bytes 实收字节 < Content-Length → TRUNCATED。
|
||||
|
||||
用 fake httpx 确定性触发（真实服务器下 httpx 会因短读抛 HTTPError）。
|
||||
"""
|
||||
import types as _types
|
||||
|
||||
import httpx as real_httpx
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
|
||||
class FakeResp:
|
||||
status_code = 200
|
||||
headers = {"content-length": "100"}
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
async def aiter_bytes(self, chunk_size):
|
||||
yield b"short"
|
||||
|
||||
class FakeStream:
|
||||
async def __aenter__(self):
|
||||
return FakeResp()
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, follow_redirects=True):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
def stream(self, method, url, timeout=None):
|
||||
return FakeStream()
|
||||
|
||||
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
|
||||
AsyncClient=FakeClient, Timeout=real_httpx.Timeout, HTTPError=real_httpx.HTTPError,
|
||||
))
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_web("http://x/y.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "TRUNCATED"
|
||||
|
||||
|
||||
def test_fetch_web_network_error(tmp_path, monkeypatch):
|
||||
"""网络错误：httpx.HTTPError → NETWORK_ERROR（fake 确定性触发）。"""
|
||||
import types as _types
|
||||
|
||||
import httpx as real_httpx
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
|
||||
class FailingStream:
|
||||
async def __aenter__(self):
|
||||
raise real_httpx.HTTPError("connection reset")
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
class FailingClient:
|
||||
def __init__(self, follow_redirects=True):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
def stream(self, method, url, timeout=None):
|
||||
return FailingStream()
|
||||
|
||||
monkeypatch.setattr(sf, "httpx", _types.SimpleNamespace(
|
||||
AsyncClient=FailingClient, Timeout=real_httpx.Timeout, HTTPError=real_httpx.HTTPError,
|
||||
))
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_web("http://x/y.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "NETWORK_ERROR"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强（2）OSS：BAD_OSS / endpoint 解析（内网 http）/ fetch_oss 走 web + kind 修正
|
||||
# ===========================================================================
|
||||
|
||||
def test_oss_url_bad_format():
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_resolve_oss_url("oss://onlybucket")
|
||||
assert ei.value.code == "BAD_OSS"
|
||||
|
||||
|
||||
def test_oss_url_intranet_endpoint_http(monkeypatch):
|
||||
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-internal.example.com")
|
||||
url = _resolve_oss_url("oss://seq-bucket/data/x.fa")
|
||||
assert url == "http://seq-bucket.oss-internal.example.com/data/x.fa"
|
||||
|
||||
|
||||
def test_fetch_oss_marks_kind_and_provenance(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
from pathlib import Path as _Path
|
||||
|
||||
async def fake_web(url, dest_dir, timeout=None):
|
||||
dest = _Path(dest_dir) / "x.fa"
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(">x\nACGT\n")
|
||||
return {"ok": True, "path": str(dest), "name": "x.fa", "size": 4,
|
||||
"kind": "web", "provenance": {"source": url, "kind": "web",
|
||||
"sha256": "0" * 64, "fetched_at": "t"}}
|
||||
|
||||
monkeypatch.setattr(sf, "fetch_web", fake_web)
|
||||
monkeypatch.setenv("BIOFLOW_OSS_ENDPOINT", "oss-cn-hangzhou.aliyuncs.com")
|
||||
r = _run(sf.fetch_oss("oss://b/k.fa", tmp_path / "out"))
|
||||
assert r["ok"] is True
|
||||
assert r["kind"] == "oss"
|
||||
assert r["provenance"]["kind"] == "oss"
|
||||
assert r["provenance"]["oss_spec"] == "oss://b/k.fa"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强（3）fetch_source：EMPTY_SOURCE / oss 路由 / aliyuncs 预签名路由
|
||||
# ===========================================================================
|
||||
|
||||
def test_fetch_source_empty(tmp_path):
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_source("", tmp_path))
|
||||
assert ei.value.code == "EMPTY_SOURCE"
|
||||
|
||||
|
||||
def test_fetch_source_oss_routing(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
calls = []
|
||||
|
||||
async def fake_oss(spec, dest_dir, timeout=None):
|
||||
calls.append(spec)
|
||||
return {"ok": True, "kind": "oss"}
|
||||
|
||||
monkeypatch.setattr(sf, "fetch_oss", fake_oss)
|
||||
r = _run(sf.fetch_source("oss://bucket/k.fa", tmp_path))
|
||||
assert r["kind"] == "oss"
|
||||
assert calls == ["oss://bucket/k.fa"]
|
||||
|
||||
|
||||
def test_fetch_source_aliyuncs_presigned_routes_to_oss(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
calls = []
|
||||
|
||||
async def fake_oss(spec, dest_dir, timeout=None):
|
||||
calls.append(spec)
|
||||
return {"ok": True, "kind": "oss"}
|
||||
|
||||
monkeypatch.setattr(sf, "fetch_oss", fake_oss)
|
||||
presigned = "https://bucket.aliyuncs.com/x.fa?sig=abc"
|
||||
r = _run(sf.fetch_source(presigned, tmp_path))
|
||||
assert r["kind"] == "oss"
|
||||
assert calls == [presigned]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# S0 补强（4）局域网：目录 IS_DIR / scp 拉取 / sshpass 缺失 / scp 失败 / 解析边界
|
||||
# ===========================================================================
|
||||
|
||||
def test_fetch_lan_dir_rejected(tmp_path):
|
||||
d = tmp_path / "adir"
|
||||
d.mkdir()
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan(str(d), tmp_path / "out"))
|
||||
assert ei.value.code == "IS_DIR"
|
||||
|
||||
|
||||
def test_parse_lan_remote_relative_rejected():
|
||||
assert _parse_lan_remote("host:relative/path") is None
|
||||
|
||||
|
||||
def test_fetch_lan_scp_success(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
from pathlib import Path as _Path
|
||||
|
||||
async def fake_scp(*args, **kwargs):
|
||||
dest = _Path(args[-1])
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(">g\nACGT\n")
|
||||
|
||||
class P:
|
||||
returncode = 0
|
||||
|
||||
async def communicate(self):
|
||||
return (b"", b"")
|
||||
|
||||
return P()
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
|
||||
r = _run(fetch_lan("xzq@192.168.1.5:/data/genome.fa", tmp_path / "out",
|
||||
credentials={"user": "xzq"}))
|
||||
assert r["ok"] is True
|
||||
assert r["kind"] == "lan"
|
||||
assert r["name"] == "genome.fa"
|
||||
|
||||
|
||||
def test_fetch_lan_scp_sshpass_missing(tmp_path, monkeypatch):
|
||||
import shutil
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
monkeypatch.setattr(shutil, "which", lambda name: None)
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("host:/data/x.fa", tmp_path / "out", credentials={"password": "p"}))
|
||||
assert ei.value.code == "SSHPASS_MISSING"
|
||||
|
||||
|
||||
def test_fetch_lan_scp_failed(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
|
||||
async def fake_scp(*args, **kwargs):
|
||||
class P:
|
||||
returncode = 1
|
||||
|
||||
async def communicate(self):
|
||||
return (b"", b"scp: no such file")
|
||||
|
||||
return P()
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "LAN_ERROR"
|
||||
|
||||
|
||||
def test_scp_fetch_remote_uses_port_and_user(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
from pathlib import Path as _Path
|
||||
captured = {}
|
||||
|
||||
async def fake_scp(*args, **kwargs):
|
||||
captured["args"] = list(args)
|
||||
dest = _Path(args[-1])
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(">g\nACGT\n")
|
||||
|
||||
class P:
|
||||
returncode = 0
|
||||
|
||||
async def communicate(self):
|
||||
return (b"", b"")
|
||||
|
||||
return P()
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
|
||||
_run(fetch_lan("host:/data/x.fa", tmp_path / "out",
|
||||
credentials={"user": "u", "port": "2222"}))
|
||||
# scp -P 2222 u@host:/data/x.fa <dest>
|
||||
assert captured["args"][1] == "-P"
|
||||
assert captured["args"][2] == "2222"
|
||||
assert captured["args"][-2] == "u@host:/data/x.fa"
|
||||
|
||||
|
||||
def test_fetch_lan_scp_timeout(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
|
||||
async def fake_scp(*args, **kwargs):
|
||||
class P:
|
||||
returncode = 0
|
||||
|
||||
async def communicate(self):
|
||||
return (b"", b"")
|
||||
|
||||
return P()
|
||||
|
||||
async def fake_wait_for(coro, timeout=None):
|
||||
coro.close() # 未 await 的 communicate 协程关闭，避免 never-awaited 警告
|
||||
raise asyncio.TimeoutError()
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
|
||||
monkeypatch.setattr(asyncio, "wait_for", fake_wait_for)
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "LAN_TIMEOUT"
|
||||
|
||||
|
||||
def test_fetch_lan_scp_no_file(tmp_path, monkeypatch):
|
||||
from app.services.bioflow import source_fetcher as sf
|
||||
|
||||
async def fake_scp(*args, **kwargs):
|
||||
# returncode 0 但未产出目标文件
|
||||
class P:
|
||||
returncode = 0
|
||||
|
||||
async def communicate(self):
|
||||
return (b"", b"")
|
||||
|
||||
return P()
|
||||
|
||||
monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_scp)
|
||||
with pytest.raises(FetchError) as ei:
|
||||
_run(fetch_lan("host:/data/x.fa", tmp_path / "out"))
|
||||
assert ei.value.code == "LAN_ERROR"
|
||||
@@ -1,10 +0,0 @@
|
||||
#!/bin/bash
|
||||
# deploy/paper_restart.sh —— 论文撰写服务重启（8097，dev 树明文；不触碰 8090/8091/8098）。
|
||||
# 用法：bash deploy/paper_restart.sh
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
|
||||
bash "$REPO_ROOT/paper_writer_service/restart.sh"
|
||||
echo "paper_writer_service 重启已触发（端口 8097，日志 paper_writer_service/paper.log）"
|
||||
Executable
+141
@@ -0,0 +1,141 @@
|
||||
#!/usr/bin/env bash
|
||||
# deploy/toggle_bioflow.sh —— 生信能力按需装卸（热，不重启 QA）+ bioflow_service 启停 + 注册状态自检。
|
||||
#
|
||||
# 用法：
|
||||
# bash deploy/toggle_bioflow.sh <gene-family|rnaseq|extension|all> <on|off>
|
||||
#
|
||||
# 原理：
|
||||
# - 能力装卸经 BFF POST /api/capabilities/install（透传 QA install_manager，热加载生效不重启 QA）
|
||||
# - on = register（manifest 校验 + 原子写 capabilities.yaml + registry 差分刷新）
|
||||
# - off = unregister（按 name 移除能力段，其余能力不受影响）
|
||||
# - 自检 = GET /api/capabilities 的 installed 数组是否含对应能力
|
||||
# - bioflow_service 启停：全部能力关闭时停 8098；任一开启时启动 8098
|
||||
# - 卸载活跃任务：三表已持久化，服务恢复后 runner.recover 续跑；QA 侧按「生信服务未启动」降级话术
|
||||
set -uo pipefail
|
||||
|
||||
BFF="http://127.0.0.1:8090"
|
||||
BIOFLOW_SERVICE_DIR="/home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new/bioflow_service"
|
||||
PY="/home/xzq/5090/teamWorkspace/zh/plantai-rag-new-venv/bin/python"
|
||||
BIOFLOW_PORT="${BIOFLOW_PORT:-8098}"
|
||||
|
||||
CAP="${1:-}"
|
||||
ACTION="${2:-}"
|
||||
|
||||
CAPS=(gene-family rnaseq extension)
|
||||
|
||||
usage() {
|
||||
echo "用法: $0 <gene-family|rnaseq|extension|all> <on|off>" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
# capability key -> registry 能力名
|
||||
cap_name() {
|
||||
case "$1" in
|
||||
gene-family) echo "bioflow-gene-family" ;;
|
||||
rnaseq) echo "bioflow-rnaseq" ;;
|
||||
extension) echo "bioflow-extension" ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# capability key -> register manifest YAML（与 agent/qa/configs/capabilities.yaml 一致）
|
||||
manifest() {
|
||||
case "$1" in
|
||||
gene-family) cat <<'YAML'
|
||||
capabilities:
|
||||
- name: bioflow-gene-family
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 基因家族分析（路径规划/建任务/参数确认/文件获取与预处理/状态查询），自然语言触发
|
||||
session_default: true
|
||||
nl_triggers: [基因家族, 家族成员, 亚家族, 系统发育, 建树, 进化树, 保守基序, motif, 结构域, 基因结构, 共线性, Ka/Ks, 选择压力, 组合证据]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/gene-family/
|
||||
prompt_pack:
|
||||
file: bioflow_gene_family_agent_rules.md
|
||||
YAML
|
||||
;;
|
||||
rnaseq) cat <<'YAML'
|
||||
capabilities:
|
||||
- name: bioflow-rnaseq
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 转录组分析（实验设计确认/样本配置/路径规划/建任务/差异表达/富集），自然语言触发
|
||||
session_default: true
|
||||
nl_triggers: [转录组, RNA-seq, RNAseq, rnaseq, 差异表达, 表达定量, 富集, WGCNA, GSEA, 实验设计, fastq]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/rnaseq/
|
||||
prompt_pack:
|
||||
file: bioflow_rnaseq_agent_rules.md
|
||||
YAML
|
||||
;;
|
||||
extension) cat <<'YAML'
|
||||
capabilities:
|
||||
- name: bioflow-extension
|
||||
provider: mcp
|
||||
permission: read_only
|
||||
description: 用户自定义扩展分析（创建/转换/试运行/校验/发布/运行），自然语言触发
|
||||
session_default: true
|
||||
nl_triggers: [扩展, 拓展分析, 扩展分析, 自定义分析, 分析模板, 分析脚本, 分析工具]
|
||||
mcp:
|
||||
url: http://127.0.0.1:8098/mcp/extension/
|
||||
prompt_pack:
|
||||
file: bioflow_extension_agent_rules.md
|
||||
YAML
|
||||
;;
|
||||
*) usage ;;
|
||||
esac
|
||||
}
|
||||
|
||||
[ -n "$CAP" ] && [ -n "$ACTION" ] || usage
|
||||
[ "$ACTION" = "on" ] || [ "$ACTION" = "off" ] || usage
|
||||
if [ "$CAP" != "all" ]; then
|
||||
cap_name "$CAP" >/dev/null 2>&1 || usage
|
||||
fi
|
||||
|
||||
# 1. 逐能力 register/unregister（热加载不重启 QA）
|
||||
targets=("$CAP")
|
||||
[ "$CAP" = "all" ] && targets=("${CAPS[@]}")
|
||||
for key in "${targets[@]}"; do
|
||||
name="$(cap_name "$key")"
|
||||
if [ "$ACTION" = "on" ]; then
|
||||
body="$(manifest "$key")"
|
||||
resp=$(curl -s -m 30 -X POST "$BFF/api/capabilities/install" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "$(python3 -c 'import json,sys; print(json.dumps({"action":"register","manifest":sys.stdin.read()}))' <<<"$body")")
|
||||
else
|
||||
resp=$(curl -s -m 30 -X POST "$BFF/api/capabilities/install" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"action\":\"unregister\",\"name\":\"$name\"}")
|
||||
fi
|
||||
echo "[$ACTION] $name -> $(echo "$resp" | head -c 300)"
|
||||
done
|
||||
|
||||
# 2. 自检：GET /api/capabilities 的 installed 数组
|
||||
installed=$(curl -s -m 10 "$BFF/api/capabilities")
|
||||
echo "注册状态: $(echo "$installed" | python3 -c 'import json,sys; d=json.load(sys.stdin); print([i.get("name") for i in d.get("installed") or []])' 2>/dev/null || echo 'parse-fail')"
|
||||
|
||||
# 3. bioflow_service 按需启停：任一能力开启则启动，全部关闭则停止
|
||||
any_on=false
|
||||
for key in "${CAPS[@]}"; do
|
||||
name="$(cap_name "$key")"
|
||||
if echo "$installed" | grep -q "\"$name\""; then
|
||||
any_on=true
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$any_on" = true ]; then
|
||||
if ! lsof -ti tcp:$BIOFLOW_PORT -sTCP:LISTEN >/dev/null 2>&1; then
|
||||
echo "启动 bioflow_service :$BIOFLOW_PORT"
|
||||
( cd "$BIOFLOW_SERVICE_DIR" && setsid "$PY" -m uvicorn app.main:app --host 127.0.0.1 --port "$BIOFLOW_PORT" \
|
||||
> /tmp/plantai-bioflow.log 2>&1 < /dev/null & ) && disown -a
|
||||
sleep 2
|
||||
fi
|
||||
curl -s -m 8 "http://127.0.0.1:$BIOFLOW_PORT/api/bioflow/health" | head -c 150
|
||||
echo
|
||||
else
|
||||
echo "全部能力已卸载：停止 bioflow_service :$BIOFLOW_PORT"
|
||||
kill $(lsof -ti tcp:$BIOFLOW_PORT -sTCP:LISTEN) 2>/dev/null || true
|
||||
fi
|
||||
|
||||
echo "DONE"
|
||||
@@ -1,51 +0,0 @@
|
||||
#!/bin/bash
|
||||
# deploy/toggle_paper.sh —— 论文撰写快速开关（dev 树 175；不触碰生产 176 /srv/plantai）。
|
||||
# 用法：bash deploy/toggle_paper.sh off|on
|
||||
# 原理：BFF main.py 按 PAPER_WRITER_ENABLED 决定是否注册 /api/paper 路由；
|
||||
# 不注册时 /api/paper/health 返回 404，前端探活落 offline，触发词落 docgen 单发降级。
|
||||
set -uo pipefail
|
||||
|
||||
ACTION="${1:-}"
|
||||
SDK_DIR="/home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new/sdk"
|
||||
ENV_FILE="$SDK_DIR/.env"
|
||||
PY="/home/xzq/5090/teamWorkspace/zh/plantai-rag-new-venv/bin/python"
|
||||
BIND="${PAPER_BFF_BIND:-0.0.0.0}"
|
||||
|
||||
if [ "$ACTION" != "off" ] && [ "$ACTION" != "on" ]; then
|
||||
echo "用法: $0 <off|on>" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
VAL="true"
|
||||
[ "$ACTION" = "off" ] && VAL="false"
|
||||
|
||||
# 1. 改 env（追加或覆盖）
|
||||
if grep -q "^PAPER_WRITER_ENABLED=" "$ENV_FILE"; then
|
||||
sed -i "s/^PAPER_WRITER_ENABLED=.*/PAPER_WRITER_ENABLED=$VAL/" "$ENV_FILE"
|
||||
else
|
||||
echo "PAPER_WRITER_ENABLED=$VAL" >> "$ENV_FILE"
|
||||
fi
|
||||
echo "PAPER_WRITER_ENABLED -> $VAL"
|
||||
|
||||
# 2. 重启 BFF 8090（dev 树；仅取 LISTEN 者；venv 必须绝对路径，cd sdk 后才能 import app.main）
|
||||
kill $(lsof -ti tcp:8090 -sTCP:LISTEN) 2>/dev/null || true
|
||||
sleep 2
|
||||
cd "$SDK_DIR" && set -a && . .env && set +a
|
||||
( cd "$SDK_DIR" && setsid "$PY" -m uvicorn app.main:app --host "$BIND" --port 8090 \
|
||||
> /tmp/plantai-bff.log 2>&1 < /dev/null & ) && disown -a
|
||||
|
||||
# 3. 自检
|
||||
for i in $(seq 1 20); do
|
||||
sleep 2
|
||||
curl -s -m 3 http://127.0.0.1:8090/api/health >/dev/null 2>&1 && break
|
||||
done
|
||||
curl -s -m 10 http://127.0.0.1:8090/api/health | head -c 120
|
||||
echo
|
||||
if [ "$ACTION" = "off" ]; then
|
||||
CODE=$(curl -s -o /dev/null -w "%{http_code}" -m 8 http://127.0.0.1:8090/api/paper/health)
|
||||
echo "paper/health HTTP $CODE (off 期望 404)"
|
||||
[ "$CODE" = "404" ] && echo "OFF_OK" || echo "OFF_CHECK_FAIL"
|
||||
else
|
||||
curl -s -m 8 http://127.0.0.1:8090/api/paper/health | head -c 150
|
||||
echo
|
||||
fi
|
||||
@@ -0,0 +1,177 @@
|
||||
# 生信流程 · 对话式手工测试手册（S6 更新版）
|
||||
|
||||
> 供用户逐项实测。生信模块已从「卡片式交互」改造为「自然语言对话式交互 + MCP 外挂」，
|
||||
> 本手册以**对话剧本**形式给出每个场景的：对话示例 → 系统应如何回应 → 通过判据。
|
||||
> 环境：内网 http://192.168.0.175:8090（BFF）；生信服务 8098、QA 8091 已部署；
|
||||
> 三能力（基因家族/转录组/扩展）按 §0 装卸。
|
||||
|
||||
---
|
||||
|
||||
## 0. 前置条件
|
||||
|
||||
1. 浏览器打开 http://192.168.0.175:8090。
|
||||
2. **能力安装检查**：设置面板（SettingsPanel）应能看到三个生信开关（基因家族/转录组/扩展）。
|
||||
打开任意一个后，工作区出现「生信任务」「生信档案」TAB；全关则 TAB 消失（**TAB 显隐 = 能力注册状态**，非服务健康）。
|
||||
3. 服务器 docker 已有 8 个 bioflow 镜像；金标数据在 `bioflow_kernel/assets/examples/test_input/`（Ft 四件套 pep/cds/genome/gff3）。
|
||||
4. 熔断开关：`QA_V1_ENABLE_BIOFLOW_AGENT=1`（默认 0，关时生信句走普通 QA）。
|
||||
|
||||
---
|
||||
|
||||
## 1. 多配置开关回归（S6）
|
||||
|
||||
| 配置 | 操作 | 期望 | 判据 |
|
||||
|---|---|---|---|
|
||||
| 全关（熔断=0 + 三能力卸载） | 发「帮我做基因家族分析」 | 普通 QA 礼貌回答（说明未安装生信能力），**无**路径卡 | 无生信分流 |
|
||||
| 只开基因家族 | 发「帮我做基因家族分析」 | 进入生信对话（列路径+挂路径卡） | 命中 |
|
||||
| 只开基因家族 | 发「帮我做转录组差异表达」 | 走普通 QA（转录组能力未装） | 不响应 |
|
||||
| 只开转录组 | 发「帮我做转录组」 | 进入生信对话 | 命中 |
|
||||
| 只开转录组 | 发「帮我做基因家族分析」 | 走普通 QA | 不响应 |
|
||||
| 全开但非生信提问 | 发「光合作用的电子传递链是什么」 | 走普通 QA（非生信意图不调工具） | 不调工具 |
|
||||
|
||||
---
|
||||
|
||||
## 2. 对话剧本 A：基因家族分析（卡片辅助通道）
|
||||
|
||||
**第 1 轮 —— 需求与路径**
|
||||
> 用户：帮我做水稻 SPL 基因家族分析
|
||||
> 系统（自然语言）：列「核心流程（必选）+ 可选分析」，说明每个可选项生物学意义；
|
||||
> **同帧挂出路径选择卡**。
|
||||
> 判据：回复含路径列表 + 路径卡出现（卡片不替代自然语言）。
|
||||
|
||||
**第 2 轮 —— 卡片操作 → 系统复述**
|
||||
> 用户（路径卡勾选「核心流程 + Ka/Ks + 启动子」，点确认；打字补充文件位置）
|
||||
> 系统：先自然语言**复述**用户选择（"您选择了核心流程 + Ka/Ks + 启动子分析…"），再请确认。
|
||||
> 判据：系统复述内容与卡片勾选一致；建任务前必有一轮复述确认（不直接建任务）。
|
||||
|
||||
**第 3 轮 —— 网络文件 + 非标准文件预处理**
|
||||
> 用户：文件在 `https://…/rice.fa` 与 `oss://seq-bucket/rice.gff3`，另传 `spl_list.xlsx`
|
||||
> 系统：拉取 web/OSS 文件（**下载前检查磁盘余量、记录 provenance**）；对 `spl_list.xlsx`
|
||||
> 做内容嗅探后**自然语言说明列结构并提问**（"推测第二列是蛋白序列，备注列忽略？"）。
|
||||
> 判据：文件获取成功；Excel 触发预处理提问（列表方式，可卡片点选也可打字回答）。
|
||||
|
||||
**第 4 轮 —— 建任务执行**
|
||||
> 用户：都对，备注列不要；确认，开始吧
|
||||
> 系统：复述整理结果（"已整理为 FASTA 3 条序列"）→ 创建任务 → 消息内出现**只读精简进度块**，
|
||||
> 右侧「生信任务」TAB 完整滚动日志实时跟随。
|
||||
> 判据：进度块只读（无按钮）；TAB 日志自动滚底。
|
||||
|
||||
**执行中 —— 进度转述 + 随时提问**
|
||||
> 用户：什么是组合证据？
|
||||
> 系统：经流程知识库自然语言回答（带引用），**不打断任务**。
|
||||
> 判据：提问得到引用回答，任务继续推进。
|
||||
|
||||
**人工确认边界（不可跳过）**
|
||||
> 系统：第 8 步亚家族预测完成（6 个亚家族）。⚠ 需您人工确认，我不能代您确认。
|
||||
> 用户：把 SPL3 从亚家族 A 移到 B → 系统复述修改 → 用户「确认」。
|
||||
> 判据：subfamily_confirm 必停、必复述、绝不代确认。
|
||||
|
||||
**收尾 —— 存档**
|
||||
> 系统：全部完成（19 成员 6 亚家族），同帧挂出存档卡。
|
||||
> 用户（存档卡填名称点「存档」或打字「存档，叫 SPL 水稻鉴定」）
|
||||
> 系统：实际执行 `archive_job` 存档（复用 REST 语义），复述存档三要素（名称/位置/大小）。
|
||||
> 判据：存档成功（重名提示 ARCHIVE_EXISTS）。
|
||||
|
||||
---
|
||||
|
||||
## 3. 对话剧本 B：转录组分析（纯自然语言通道，全程不点卡片）
|
||||
|
||||
**第 1 轮 —— 必确认信息**
|
||||
> 用户：帮我做转录组，比较干旱处理和对照
|
||||
> 系统：列主流程 + 可选；**必须确认**四项（物种/参考基因组版本、FASTQ 单双端、
|
||||
> 对照处理组重复数、链特异性），列表展示。
|
||||
> 判据：实验设计确认边界被明确列出，未确认前不得建差异表达任务。
|
||||
|
||||
**第 2 轮 —— OSS 文件 + 样本信息（纯打字，不点卡）**
|
||||
> 用户：数据在 oss://seq-bucket/drought/，水稻日本晴，干旱 3 重复对照 3 重复，双端非链特异
|
||||
> 系统：从 OSS 拉取 12 个 FASTQ 并核对成对；复述样本设计（"干旱 3 vs 对照 3，双端，非链特异"），
|
||||
> 请确认基线（对照为基线、比较方向「干旱 vs 对照」）。
|
||||
> 判据：纯自然语言路径走通（**卡片与打字两条通道服务端语义一致**）。
|
||||
|
||||
**第 3 轮 —— 执行 + 结果转述**
|
||||
> 用户：对 → 系统执行 QC→比对→定量→样本质控→差异表达→富集，逐段转述（接头率/Q30/唯一比对率/差异基因数/PCA 分离）。
|
||||
> 判据：每步完成后自然语言转述关键指标；样本质控 PCA 结论有转述。
|
||||
|
||||
---
|
||||
|
||||
## 4. 三种网络文件来源用例
|
||||
|
||||
| 来源 | 示例 | 期望 |
|
||||
|---|---|---|
|
||||
| web | 「文件在 https://…/x.fa」 | httpx 流式下载，不设大小上限；下载前查磁盘余量；记录 provenance |
|
||||
| OSS | 「数据在 oss://bucket/key 或预签名 URL」 | 经 BIOFLOW_OSS_ENDPOINT 解析或预签名直通；无 endpoint 时提示改用预签名 |
|
||||
| 局域网 | 「文件在 192.168.x.x:/data/x.fa，账号 xx 密码 xx」 | ssh/scp 拉取（凭据单次会话内存用，不落盘、审计不记明文） |
|
||||
|
||||
判据：文件落 `bioflow_data/jobs/{job}/input/`，先内容嗅探后登记，provenance 记录来源/哈希/时间。
|
||||
|
||||
---
|
||||
|
||||
## 5. 非标准文件预处理（Excel / 乱码文本）闭环
|
||||
|
||||
1. 上传 `xxx.xlsx` / 乱码 txt → 系统 `inspect_file` 探测（列结构/格式猜测/乱码检测）。
|
||||
2. 系统自然语言说明 + 列表提问（哪列是序列、是否去重、是否去表头）。
|
||||
3. 用户回答 → `transform_file` 转换（LLM 定规则 + 脚本执行，大文件抽样定规则）。
|
||||
4. 结果过格式验证器（fasta 可解析/gff3 9 列/tsv 列一致/samples.yaml schema）；
|
||||
失败说明问题并追问，循环 ≤3 次后建议人工整理。
|
||||
5. 转换后 `register_job_files` 登记，provenance 记「原始→规则→产物」链。
|
||||
|
||||
判据：全程自然语言引导；转换产物格式校验通过；原文件永不覆盖。
|
||||
|
||||
---
|
||||
|
||||
## 6. 卡片辅助 vs 纯自然语言双通道等价
|
||||
|
||||
| 场景 | 卡片通道 | 纯 NL 通道 | 等价判据 |
|
||||
|---|---|---|---|
|
||||
| 路径选择 | 路径卡勾选 | 打字「核心流程 + Ka/Ks」 | 两者复述的确认内容一致，建任务入参一致 |
|
||||
| 参数确认 | 参数卡填值 | 打字「用 8 核、padj 用 0.01」 | confirm_stage 入参一致 |
|
||||
| 文件提供 | 文件卡拖拽/填路径 | 打字「文件在 …」 | register_job_files / fetch_source_file 入参一致 |
|
||||
| 存档 | 存档卡点存档 | 打字「存档，叫 X」 | archive_job 入参一致 |
|
||||
|
||||
判据：卡片操作产生的是 `card_result` 结构化消息进 QA 流（**不直连 bioflow 控制面**），
|
||||
与打字在服务端语义完全一致（QA 对话状态机是唯一决策点）。
|
||||
|
||||
---
|
||||
|
||||
## 7. 旧会话兼容（Legacy 块只读）
|
||||
|
||||
1. 打开一个改造前创建、含旧生信块（bioflow_qa / 旧 bioflow_* 块）的会话。
|
||||
2. 期望：旧块以只读形式降级渲染（标题 + 只读引用/提示），**不报错、无可交互控件**。
|
||||
|
||||
---
|
||||
|
||||
## 8. 扩展子系统对话化（第三能力单元）
|
||||
|
||||
1. 打开「扩展」能力开关。
|
||||
2. 对话创建扩展：描述分析脚本（代码文件 + 运行命令）→ `create_extension`（draft）。
|
||||
3. `build` → `trial_run` → `validate` → `confirm` → `publish` → `run` 逐步走（卡片或打字等价）。
|
||||
4. 试运行失败 → 「AI 修复」触发 `repair_extension`（修复上限 20 轮）；校验失败/状态不足时系统给出确定性提示。
|
||||
|
||||
判据：扩展生命周期全程对话化；非法状态返回清晰提示（EXT_STATUS），不报 500。
|
||||
|
||||
---
|
||||
|
||||
## 9. 后端金标验证（服务器命令行，S6 回归）
|
||||
|
||||
```bash
|
||||
cd /home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new
|
||||
BIOFLOW_DATA=/home/xzq/5090/teamWorkspace/zh/bioflow-data \
|
||||
bash bioflow_kernel/tests/smoke_golden.sh bioflow-gf:dev
|
||||
```
|
||||
|
||||
期望：4 步全绿——python 3.11.14 / ldd 0 缺库 / 无 conda / 金标 identify 24 ID diff 空（`GOLDEN_DIFF_OK 24 IDs`）。
|
||||
|
||||
---
|
||||
|
||||
## 10. 端到端主线（一次性走全流程，双通道各一遍）
|
||||
|
||||
1. 卡片通道：§2 全程（路径卡→文件卡→参数卡→进度块→存档卡）。
|
||||
2. 纯 NL 通道：§3 全程（不点任何卡片）。
|
||||
3. 含一次 Excel 预处理、一次局域网/OSS 文件拉取。
|
||||
4. 任务完成后存档 → 生信档案 TAB 回看 → 复用发起。
|
||||
|
||||
期望：两条通道均走通，7 项引导（路径/文件/参数/报告/存档/复用/过程展示）逐项达成，
|
||||
进度块实时流转、右侧日志滚底。
|
||||
|
||||
---
|
||||
|
||||
*本手册为 S6 对话式改造更新版，替代旧卡片式 `MANUAL_TEST_BIOFLOW.md` 的生信流程部分。*
|
||||
@@ -1,42 +0,0 @@
|
||||
# feat(review): journal_directory 启动自动 seed（部署即运行）
|
||||
|
||||
## 背景
|
||||
|
||||
source_v4.xlsx（1056 条目录）此前需手动跑 import_journals.py 入库，部署后忘跑即出现"目录数据未导入"。本次实现启动时自动检测 + 自动导入。线上当前环境已由人工完成导入（review.db 内 1056 行），本功能消除的是新环境部署/重建库时的手动步骤风险。
|
||||
|
||||
## 修改内容
|
||||
|
||||
| 文件 | 变更 |
|
||||
|---|---|
|
||||
| `app/services/journal_seed.py`（新增） | 共享核心：schema/清洗/解析/upsert 从 CLI 原样迁入；`extract_rows_from_xlsx()`（口径 779/277 与 id 冲突闸门，不符不写库）、`upsert_to_db()`、`ensure_journal_directory_seeded()`（空表才导，四键 dict 契约，绝不抛异常） |
|
||||
| `scripts/import_journals.py` | 瘦身为薄 CLI（-431/+92 行）：对账报告/退出码 0/1/2/3 全保留，核心复用 journal_seed |
|
||||
| `app/config.py` | +`seed_on_startup=True`（REVIEW_SEED_ON_STARTUP 可关）、+`journal_directory_seed_path` |
|
||||
| `app/main.py` | lifespan：开关门控 → `asyncio.to_thread` 导入（不堵 event loop）→ try/except 不阻塞启动；路径走 `db.path` 注入链 |
|
||||
| `tests/conftest.py`（新增） | 全局关 seed，防止单测触碰真库/拖慢套件 |
|
||||
| `tests/test_seed.py`（新增） | 6 用例（见下） |
|
||||
|
||||
不动：source_v4.xlsx、journal_directory 路由/表 schema、db.py、前端。
|
||||
|
||||
## 测试用例与结果（pytest 78 passed，新增 6 条）
|
||||
|
||||
1. 空库 → seed 成功，rows=1056（en=779/zh=277）✓
|
||||
2. 非空库（1 行）→ skip，count 不变 ✓
|
||||
3. xlsx 不存在 → 返回 error，不抛异常，不写库 ✓
|
||||
4. 幂等：连调两次，第二次 skip ✓
|
||||
5. 开关 off（lifespan）→ 表不创建 ✓
|
||||
6. 开关 on（lifespan）→ tmp 库 1056 行 ✓
|
||||
|
||||
## 部署验证
|
||||
|
||||
```bash
|
||||
bash review_service/restart.sh # 非空库：日志 "already has 1056 rows, skip"
|
||||
curl 'http://127.0.0.1:8100/api/review/journals/directory?lang=en' # total=779
|
||||
# 空库冷启动（本机已验证等价路径）：日志 "seeded 1056 rows (en=779, zh=277, N.Ns)"
|
||||
# 特殊部署：REVIEW_SEED_ON_STARTUP=false → 日志 "seed skipped"
|
||||
```
|
||||
|
||||
本机已实测：真机重启走 skip 路径（`INFO:app.main:already has 1056 rows, skip`），CLI 薄壳回归 exit=0 / `en=779 zh=277 → ✓ 相符`。
|
||||
|
||||
## 是否需要立即部署
|
||||
|
||||
待管理员确认。线上已手动导入 1056 行，本改动属部署便利性补强（消除新环境忘跑导入脚本的风险），可随下次正常发版评估带上。
|
||||
@@ -1,56 +0,0 @@
|
||||
# fix(review): 冲突降级真影响 recommendation（major → minor 真降级）
|
||||
|
||||
- 分支：`feature/academic-review-assistant`
|
||||
- 日期：2026-10-12
|
||||
- 服务：`review_service`（:8100）/ 前端 `PresubmitPanel.vue`
|
||||
- 状态：**未 commit**，待验收
|
||||
|
||||
## 背景
|
||||
|
||||
s41588（已发表 Nature Genetics 论文）真实链路里，关键词扫描器报告"3 处冲突已降级为非硬性问题"，但 `recommendation` 仍为大修、`major_weaknesses` 仍 5 条。诊断确认：原降级只做了"剥 hard_rule 标签 + 记 conflicts"，**条目仍在 major 列表内**，`major_count` / 引擎输入未变，故判定不变。本次把"降级"从标签维度升级为**列表维度的真降级**（major → minor）。
|
||||
|
||||
## 变更
|
||||
|
||||
**后端**
|
||||
- `paper_scanner.reconcile_hard_rules`：命中冲突的 major 从列表**移出**，`hard_rule=None` + `scanner_downgraded=true`，落入 demoted 列表；`fatal=true` 条目例外（只记 conflict 不降级，须人工过目）。
|
||||
- `routers/presubmit.py`：demoted 追加进 `report.minor_weaknesses`；`major_weaknesses` 为过滤后列表；explanation 文案改"已降级为小修级问题"。
|
||||
- `self_review.WeaknessItem`：新增 `scanner_downgraded: bool = False`（coerce 不读、LLM 无法伪造）。
|
||||
- 引擎 `recommend_engine.decide_recommendation` **零改动**——计数因列表变短而自动真实。
|
||||
|
||||
**前端**
|
||||
- `PresubmitPanel.vue`：minor 卡片对 `scanner_downgraded` 显示黄徽标"冲突降级"（与 major 卡片"硬性"红徽标对仗）；冲突明细块保留。
|
||||
|
||||
## 测试
|
||||
|
||||
- pytest **104 passed**（新增：`test_reconcile_fatal_not_demoted`、`test_http_partial_conflict_leaves_residual_soft`；改写 `test_reconcile_*` 与 `test_http_scanner_conflict_downgrades` 为真降级断言）。
|
||||
- vitest **382 passed / 5 failed**（5F 为既有他人 `benchpanel`+`Composer .mode-pill` labels.js 引发，与本改动无关）；`npm run build` ✓。
|
||||
|
||||
## 验收（真 LLM，:8100）
|
||||
|
||||
| 样本 | rec | rule | M/S/minor | conflicts | downgraded→minor |
|
||||
|---|---|---|---|---|---|
|
||||
| diag_strong | minor_revision | R5 | 0/0/4 | 0 | 无 |
|
||||
| diag_weak | major_revision | R1 | 6/2/7 | 1(hic) | 有（真降级生效） |
|
||||
|
||||
弱稿仍大修：4 条硬性命中（busco/accession/voucher/abs）为真缺，扫描器无冲突、不降级——真缺陷不放行。
|
||||
|
||||
## 验收：s41588（真 LLM，:8100）
|
||||
|
||||
输入 `/tmp/opencode/s41588-025-02113-5.pdf`，target_journal=Genome Biology（tier 1 override）。
|
||||
|
||||
| 项 | 修复前 | 修复后 |
|
||||
|---|---|---|
|
||||
| recommendation | major_revision | **minor_revision**（R5） |
|
||||
| major_count / soft / minor | 5 / — / — | **0 / 0 / 6** |
|
||||
| hard_hits | 含 busco/accession… | **[]** |
|
||||
| scanner signals | — | accession_found=True, busco=partial, hic_found=True |
|
||||
| major_weaknesses 列表 | 5 条 | **0 条** |
|
||||
| scanner_conflicts | — | **2 条保留**（busco_missing 格式不完整、accession_missing 部分未给号）→ 均真降级进 minor（`scanner_downgraded=true`） |
|
||||
|
||||
判定链：扫描器证伪 2 条硬性 → 真降级 → 剩余为 minor → R5（tier1 accept 门槛 minor≤1，6 条 → minor_revision）。conflicts 透明保留、explanation 明示"已降级为小修级问题，请人工核对"。
|
||||
|
||||
注：本次 LLM 未再把"树图注未标数值 / GWAS 校正描述"判成 major（负向清单 Prompt 生效），故 major 从 5 归 0，优于预期的"降到 2"。
|
||||
|
||||
## 部署
|
||||
|
||||
待管理员确认。属判定正确性修复（真论文误判大修），建议随下次正常发版带上。
|
||||
@@ -1,81 +0,0 @@
|
||||
# fix(review): presubmit 四 Bug 修复——截断保真 / 分档 / 词表 / 扫描器兜底
|
||||
|
||||
## 背景
|
||||
|
||||
真实案例：已发表 Nature Genetics 论文（s41588-026-02682-z，黄瓜泛基因组）被判
|
||||
"大修"，但论文实有 BUSCO 98.57%（P2）、94.18% sequences anchored（P2）、
|
||||
PRJCA038097 等 accession（P17 Data Availability）。诊断出四个独立缺陷：
|
||||
|
||||
1. **PDF 截断**：`max_text_chars=60000` 从头切，30 页论文 P17 的 Data Availability 被截掉；
|
||||
2. **期刊分档**：Nature Genetics 不在 5 本精选 → 默认 tier 2（应 tier 1）；
|
||||
3. **硬性规则词表过窄**：缺 PRJCA/CRA（中国 NGDC/GSA）、Hi-C 同义词只认 anchoring rate、
|
||||
BUSCO 仅百分比也算缺失；
|
||||
4. **无程序层兜底**：hard_rule 全靠 LLM 输出，无关键词证伪机制。
|
||||
|
||||
## 修改内容
|
||||
|
||||
| 文件 | 变更 |
|
||||
|---|---|
|
||||
| `app/routers/presubmit.py` | **Bug 1**：段落感知装箱 `pack_feed_text`（超预算按
|
||||
Abstract→Data Availability 全文→Methods 前 60%→Results→Discussion 优先保留，
|
||||
dropped_sections 入 meta）；无节标题回退头+尾窗（捞页尾 availability 块）。
|
||||
**双层保护**：估算 token（len/1.5）>48K → 自动降 100K 重装；LLM 400/413 → 降 80K
|
||||
重试一次。**Bug 4 接线**：coerce 后扫描器证伪 → rationale.scanner_conflicts |
|
||||
**Bug 2**：tier_source ∈ override/featured/jif/默认 |
|
||||
| `app/services/paper_scanner.py`（新增） | 关键词扫描器（纯 stdlib）：accession
|
||||
词表 PRJNA/PRJEB/PRJCA/SRP/ERP/DRP/CRA/SRR/ERR/DRR/GCA_/GCF_ **只在 Data Availability
|
||||
段扫**（正文引用他人数据不误判）；BUSCO 三级 none/partial/full；Hi-C 同义词
|
||||
（anchored to chromosome(s)/anchoring rate/pseudo-chromosome(s)/chromosome-level）。
|
||||
`reconcile_hard_rules`：LLM 报缺 X + 扫描器见 X → 剥硬性标签降 soft + 记冲突
|
||||
（只证伪不证实，不引入新误判） |
|
||||
| `app/services/recommend_engine.py` | **Bug 2**：tier 查找优先级 override
|
||||
（13 本顶刊硬编码，"The " 前缀归一化）→ featured（journals.json）→ jif 启发式
|
||||
（journal_directory 只读 jif 列；jif≥15→tier1；综述刊名含 Review/Trends/Annual
|
||||
Review 除外）→ 默认 tier 2 |
|
||||
| `app/services/self_review.py` | **Bug 3**：Prompt 词表扩充（PRJCA/CRA/SRR…）；
|
||||
BUSCO 三级判定（仅百分比→minor 不打 hard_rule）；Hi-C 同义词清单。
|
||||
**2026-10-12 追加 soft major 收紧**（s41588 复盘）：维度三系统发育树分诊重写
|
||||
（完全未做→major / 已声明仅图未标数值→presentation / 附带树→minor）；新增
|
||||
【负向清单】节（图未标统计量、标准方法描述不全、阈值缺数值、引用排版 →
|
||||
minor/presentation，严禁列 major；总纲"只有补实验/补数据/重跑分析/改核心论证
|
||||
才是 major"；与各维度冲突时以本节为准） |
|
||||
| `app/config.py` | `max_text_chars` 60000 → **150000**（REVIEW_MAX_TEXT_CHARS 可覆盖） |
|
||||
| `ui/.../PresubmitPanel.vue` | 冲突黄标块（scanner_conflicts + 降级提示）；截断
|
||||
dropped_sections 美化提示；截断文案改为"保摘要/数据可用性/方法/结果/讨论" |
|
||||
| `tests/test_presubmit.py` | 新增 15 用例（装箱/回退/token 守卫/400 重试/扫描器
|
||||
三类信号/DA 段限定/冲突降级/端到端/tier 优先级矩阵/Prompt 新词表） |
|
||||
|
||||
## 测试用例与结果
|
||||
|
||||
- **pytest：102 passed**（87 基线 + 15 新增）。
|
||||
- **vitest：379 passed / 5 failed**（5F 为既有他人 i18n labels 改动引发，与本改动
|
||||
无关；新增冲突黄标/截断提示 2 条全绿）。`npm run build` ✓。
|
||||
- **真实链路验收**（真 LLM，target_journal=Genome Biology → tier 1 override）：
|
||||
|
||||
| 样本 | 修复前 | 修复后 | 扫描器 |
|
||||
|---|---|---|---|
|
||||
| 强稿 diag_strong.pdf | minor_revision | **minor_revision**（R5，0/0/5） | accession/busco=full/hic 全命中，0 冲突 ✓ |
|
||||
| 弱稿 diag_weak.pdf | major_revision | **major_revision**（R1，硬性 4 + soft 4 + minor 5） | accession/busco 未命中（真缺不降）；Hi-C 冲突降级并记录 ✓ |
|
||||
| **真实案例 s41588**（NG 论文） | major_revision | 见下"验收"节 | — |
|
||||
|
||||
## 部署验证
|
||||
|
||||
```bash
|
||||
bash review_service/restart.sh
|
||||
curl -X POST http://127.0.0.1:8100/api/review/presubmit \
|
||||
-F "file=@paper.pdf" -F "target_journal=Nature Genetics"
|
||||
# 检查：meta.tier=1/tier_source=override；meta.dropped_sections；
|
||||
# report.recommendation_rationale.scanner_conflicts
|
||||
```
|
||||
|
||||
## 是否需要立即部署
|
||||
|
||||
待管理员确认。四修复均为判定正确性问题（真实发表论文误判大修），建议随下次正常发版带上。
|
||||
|
||||
## 遗留
|
||||
|
||||
- tier jif 启发式仅覆盖 journal_directory 已录入的 1056 刊；未收录刊名走默认 tier 2。
|
||||
- 扫描器 accession 仅扫 DA 段——若论文把 accession 放 Methods 数据声明处且无 DA 节，
|
||||
扫描器不证伪（保守方向，LLM 硬性判定仍生效，不会误放行）。
|
||||
- BUSCO full 判定要求 S: 与 D: 标记在 200 字符窗内相邻——极端排版可能判 partial
|
||||
（只会更严，不误放）。
|
||||
@@ -1,61 +0,0 @@
|
||||
# fix(review): presubmit recommendation 判定过严修复（配额制 → 规则引擎）
|
||||
|
||||
## 背景
|
||||
|
||||
用户反馈 AI 预审稿多次测试均返回 major_revision。2026-10-12 诊断确认根因是
|
||||
**配额制**而非"尺子紧"：
|
||||
|
||||
- `self_review.py` 硬性规则要求 `major_weaknesses 3-8 条`（强制配额）→ accept/minor_revision 几乎不可达；
|
||||
- LLM 直接输出 recommendation，无程序层规则；JSON 示例锚定 `major_revision`、coerce 缺省回落 `major_revision`；
|
||||
- 对照实验：近 GB 完备强稿（BUSCO 96.8% / Hi-C 98.7% / PRJNA+GCA+figshare 齐 / UFBoot 1000）仍被凑出 6 条 major，与弱稿（7 条）几乎无区分度；
|
||||
- `target_journal` 传入但 Prompt 零分档标准。
|
||||
|
||||
## 修改内容
|
||||
|
||||
| 文件 | 变更 |
|
||||
|---|---|
|
||||
| `app/services/recommend_engine.py`（新增） | 纯函数规则引擎：R0 fatal→reject / R1 硬性 major→大修（不可协商，与 tier 无关）/ R2' tier3 仅 1-2 条 soft 可降小修 / R2 soft≥1→大修 / R4 0 major 且 minor≤tier 阈值→接收 / R5→小修；rationale 模板拼装（rule/hard_hits/计数/tier/explanation，含 tier 降级提示句）；`resolve_tier` 按 journals.json `review_tier` 匹配，失败默认 tier 2 |
|
||||
| `app/services/self_review.py` | Prompt 重写：严重分级标准（major/minor/presentation），**三类均不设下限、可 0 条、严禁凑数**；硬性规则 5 项打标签（busco/hic/accession/voucher/abs_missing）；删除 recommendation 输出字段与示例锚定；coerce 不再读 recommendation（防注入），hard_rule 白名单过滤 |
|
||||
| `app/routers/presubmit.py` | coerce 后接引擎；meta 增加 tier/tier_label/tier_source |
|
||||
| `data/journals/journals.json` | 精选 5 本加 `review_tier`（GB/MP=1；BMC PB/FPS/Rice=2）+ notes 一行 |
|
||||
| `ui/.../PresubmitPanel.vue` | 判定依据块（explanation + 硬性 chips）；判定档位徽标；reject 红标"建议人工复核"；顶部 AI 免责行；major 卡片"硬性/fatal"徽标 |
|
||||
| `tests/test_presubmit.py` | 用例 1-10（引擎分支全覆盖 + tier 对照 + 注入忽略 + rationale 字段） |
|
||||
|
||||
判定链与旧链路唯一对外契约变化：`report.recommendation` 语义不变（四档枚举），
|
||||
新增 `report.recommendation_rationale` 与 `meta.tier*`。
|
||||
|
||||
## 测试用例与结果
|
||||
|
||||
- **pytest：87 passed**（78 基线 + 9 新增/重写），含用例 1-10：
|
||||
强稿形态不判大修、弱稿形态判大修且 hard_hits 正确、0 major 0/多 minor 分支、
|
||||
soft major 分支、fatal→reject、tier3 降级与硬性豁免、tier1 accept 门槛、
|
||||
LLM recommendation 注入被忽略、hard_rule 白名单过滤、rationale 含 tier/explanation。
|
||||
- **vitest：377 passed / 5 failed**（5F 为既有他人 i18n labels 改动引发的 benchpanel
|
||||
失败，与本改动无关；新增 4 条 rationale/档位/fatal 展示用例全绿）。
|
||||
- **真实链路验收**（诊断同款样本、同 target_journal=Genome Biology，真 LLM 调用）：
|
||||
|
||||
| 样本 | 修复前 | 修复后 | 依据 |
|
||||
|---|---|---|---|
|
||||
| 弱稿 diag_weak.pdf | major_revision（7 major） | **major_revision**（R1；硬性 5 命中：BUSCO/Hi-C/Accession/voucher/ABS + soft 2 + minor 6） | 真缺陷不漏判 ✓ |
|
||||
| 强稿 diag_strong.pdf | major_revision（凑出 6 major） | **minor_revision**（R5；0 major / 5 minor） | 区分度恢复 ✓ |
|
||||
|
||||
## 部署验证
|
||||
|
||||
```bash
|
||||
bash review_service/restart.sh
|
||||
# 上传强/弱样本对比（或任意 PDF）：
|
||||
curl -X POST http://127.0.0.1:8100/api/review/presubmit \
|
||||
-F "file=@paper.pdf" -F "target_journal=Genome Biology"
|
||||
# 检查 report.recommendation_rationale.rule / meta.tier；前端看"判定依据"块
|
||||
```
|
||||
|
||||
## 是否需要立即部署
|
||||
|
||||
待管理员确认。属判定质量修复（误报大修影响用户体验与信任），建议随下次正常发版带上。
|
||||
|
||||
## 遗留（Phase 2 候选）
|
||||
|
||||
- 程序层兜底 `_sanity_check_missing_hard_rules`：正文无 PRJNA/GCA_/SRP 关键词但 LLM
|
||||
未报 accession_missing → 追加 warning（不进 recommendation 计算）；先观察 LLM 是否
|
||||
"躺平"再决定。
|
||||
- tier 数据仅覆盖精选 5 本；扩展到 journal_directory 1056 条需人工核验后另行入库。
|
||||
@@ -1,161 +0,0 @@
|
||||
# PR 报告：Rebuttal Phase 1（审稿回复助手）
|
||||
|
||||
- 日期：2026-10-12
|
||||
- 分支：`feature/academic-review-assistant`（未 commit，按用户约定统一处理）
|
||||
- 范围：`review_service/` 新增 5 文件 + main 注册；`ui/` 新增面板与 store/api 扩展；无 sdk/、agent/、vstore/ 改动；无新增依赖。
|
||||
|
||||
## 功能概述
|
||||
|
||||
粘贴审稿意见文本（主入口）/ 上传批注版 PDF（次要入口）→ LLM+正则拆条（Reviewer→Comment）
|
||||
→ 逐条填回复要点（5 类下拉，无 auto）→ 生成草稿（反幻觉校验前置）
|
||||
→ 逐条勾确认 → 拼装 response letter → 下载 `.md`（human_ack 门槛）。docx 属 Phase 2。
|
||||
|
||||
## 10 个 API 端点（review_service:8100；BFF 8095 通用透传已验证可用）
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
|---|---|---|
|
||||
| POST | /api/review/rebuttal/jobs | 文本粘贴主入口 |
|
||||
| POST | /api/review/rebuttal/jobs/pdf | PDF 次要入口（vstore 解析；非 PDF/超限→VALIDATION） |
|
||||
| GET | /api/review/rebuttal/jobs/{id} | 全量还原 |
|
||||
| PATCH | .../comments/{cid}/response | 保存 parts（改后旧 draft/validation 作废） |
|
||||
| POST | .../comments/{cid}/draft | 生成草稿（校验C前置；MISSING_RESPONSE 时不调 LLM） |
|
||||
| PATCH | .../comments/{cid}/structure | 手工 merge（≥2）/ split（≥2 非空子项） |
|
||||
| PATCH | .../comments/{cid}/confirm | 逐条勾；后端重算 human_ack（全勾才 true，空 job 不置位） |
|
||||
| POST | .../letter | 拼装 letter_md |
|
||||
| GET | .../letter.md | 下载（human_ack=false → 400 HUMAN_ACK_REQUIRED） |
|
||||
| DELETE | .../jobs/{id} | 手动删（替代 TTL） |
|
||||
|
||||
信封错误复用约定：HTTP 200 + `{ok:false,error:{code,message,degraded}}`。
|
||||
|
||||
## 数据模型
|
||||
|
||||
- Job：`{id,status,source:{kind,filename,raw_text},template:'generic',reviewers:[...],letter_md,validation,human_ack,created_at,updated_at}`
|
||||
- Comment：`{id,group_id,severity,title,body,sub_items[],cross_refs[],response:{parts[]},draft,validation,human_confirmed}`
|
||||
- part.type 枚举：`new_data | reanalysis | location | clarification | rebuttal`（决策 2：无 auto）
|
||||
- SQLite 表 `rebuttal_jobs` 落在既有 `review_service/review.db`（自含短连接，不动 app/db.py）；
|
||||
启动幂等建表（main lifespan）。
|
||||
|
||||
## 关键实现决策（方案未写清处，按"保守、可测、不猜测"）
|
||||
|
||||
1. **LLM 注入形态**：parse/draft 用 `async (messages, temperature) → dict` 回调，路由内适配
|
||||
`llm.complete_json(system, user)`；单测直接 monkeypatch `llm.complete_json`。
|
||||
2. **拆条双轨**：正则先切骨架（Reviewer 段头 + 编号行，兼容行内 `. 2.` 编号）；
|
||||
LLM 逐段出 `{title,severity,sub_items[]}`。LLM 失败/结构非法 → 该段回退正则
|
||||
（title=首 60 字，severity=major）。`sub_items ≥2 且 max confidence ≥0.7` 才拆
|
||||
（决策 3），拆出条目共享 group_id；未达阈值多子问题并回一条且 **body 保留段原文**
|
||||
（不采信 LLM 改写）。
|
||||
3. **交叉引用**：`see comment N / 参见第 N 条 / 同上` → 解析目标 id；解析不到
|
||||
`target_id=null` 保留 raw（不猜）。
|
||||
4. **反幻觉校验（负向词表收窄版）**：
|
||||
- location：`shows that/as shown` 且无 `added/新增` → `LOCATION_EVIDENCE_WEAK` FAIL；
|
||||
- rebuttal：命中 `we agree/accept/concede/you are right/we were wrong` →
|
||||
`REBUTTAL_NEGATIVE` FAIL；无立场词（maintain/disagree/respectfully…）→
|
||||
`REBUTTAL_NO_STANCE` FAIL；
|
||||
- 其余三类 Phase 1 仅查非空。校验在 draft 端点前置，FAIL 不调 LLM。
|
||||
5. **human_ack**：confirm 端点改完后由 store 重算（所有 comment 全 true 才置位；
|
||||
空 job 不置位防平凡真）。前端无法直接置位。
|
||||
6. **letter.md 下载**：纯文本端点（非信封），门槛失败返回 400 + 明文
|
||||
`HUMAN_ACK_REQUIRED:…`；前端 `rebuttalDownloadLetterMd` 适配成 ApiError。
|
||||
7. **MISSING_RESPONSE**：draft 端点 parts 为空或全空内容 → 信封错误，LLM 调用次数=0
|
||||
（单测断言）。
|
||||
8. **PDF 入口**：复用 vstore `parse_pdf_bytes`；校验顺序 = 非 .pdf → 空文件 → 超
|
||||
`max_pdf_bytes`（50MB）→ 解析。与任务清单"PDF 超限/非 PDF"信封一致。
|
||||
9. **前端状态**：`store.rebuttal` 纯内存（切面板不丢、刷新丢——与任务"状态持久化"
|
||||
不同诉求，Phase 1 未要求 localStorage）；draft 失败信封里的 job/validation 一并镜像。
|
||||
10. **不改禁区**：sdk/agent/vstore/未动；PresubmitPanel 等既有面板未动；无新增依赖。
|
||||
|
||||
## 测试
|
||||
|
||||
### 后端 pytest（9 例新增，全量 113 passed）
|
||||
|
||||
```
|
||||
tests/test_rebuttal.py::test_single_reviewer_single_comment PASSED
|
||||
tests/test_rebuttal.py::test_multi_reviewer_attribution PASSED
|
||||
tests/test_rebuttal.py::test_subitem_split_shares_group_id PASSED
|
||||
tests/test_rebuttal.py::test_location_validation_fail_and_pass PASSED
|
||||
tests/test_rebuttal.py::test_rebuttal_validation PASSED
|
||||
tests/test_rebuttal.py::test_missing_response_no_llm_call PASSED
|
||||
tests/test_rebuttal.py::test_cross_ref_unresolved_keeps_raw PASSED
|
||||
tests/test_rebuttal.py::test_envelope_errors PASSED
|
||||
tests/test_rebuttal.py::test_human_ack_gate_on_letter_download PASSED
|
||||
|
||||
全量: 113 passed in 10.05s（104 基线 + 9 新增）
|
||||
```
|
||||
|
||||
### 前端 vitest（5 例新增）
|
||||
|
||||
```
|
||||
rebuttal.panel.test.js 5 passed（5 类下拉无 auto / location hint 条件显示 /
|
||||
未填禁用生成 / 未全确认禁用下载+横幅 / merge-split 仅 active 表单）
|
||||
全量: 387 passed | 5 failed（失败全部为已知 labels.js 引发的 benchpanel×4 +
|
||||
Composer .mode-pill×1 预存失败，与本次改动无关；基线 382P/5F → 387P/5F）
|
||||
```
|
||||
|
||||
### build
|
||||
|
||||
`npm run build` ✓ built in 3.04s
|
||||
|
||||
## 手工冒烟（curl，8100 真 LLM + 真 SQLite）
|
||||
|
||||
```
|
||||
POST /jobs（真实双 Reviewer 意见）→ ok=True job=rbt_ed245cd63b3a reviewers=2 comments=4
|
||||
titles: 缺少FDR校正与q值报告 / Figure 3 blurry… / Missing coordinates… / Overstated novelty…
|
||||
POST draft（未填） → ok=False code=MISSING_RESPONSE（LLM 0 调用）
|
||||
PATCH response(location 弱) + POST draft
|
||||
→ ok=False VALIDATION_FAILED findings=[LOCATION_EVIDENCE_WEAK]
|
||||
PATCH response(location 合法) + POST draft → ok=True（真 LLM 草稿）
|
||||
PATCH response(rebuttal 让步词) + draft → VALIDATION_FAILED findings=[REBUTTAL_NEGATIVE]
|
||||
3 条合理回复（new_data/clarification/rebuttal）→ draft 全 ok=True
|
||||
GET letter.md（未全勾） → HTTP 400 HUMAN_ACK_REQUIRED
|
||||
PATCH confirm ×3 → human_ack=False（c0001 漏勾，恰好验证逐条门槛）
|
||||
PATCH confirm c0001 → human_ack=True
|
||||
POST letter → ok=True letter_md len=2731
|
||||
GET letter.md → HTTP 200（/tmp/opencode/letter.md 落盘，含
|
||||
Response to Reviewers + 4 条原文/回复）
|
||||
DELETE job → ok=True meta={deleted:…}；GET → NOT_FOUND
|
||||
BFF 8095 探测 /rebuttal/jobs/nonexistent → HTTP 200（透传信封，REVIEW_ENABLED=1）
|
||||
```
|
||||
|
||||
## 交付物清单
|
||||
|
||||
| 文件 | 动作 |
|
||||
|---|---|
|
||||
| review_service/app/services/rebuttal_store.py | 新增（SQLite CRUD + human_ack 重算） |
|
||||
| review_service/app/services/rebuttal_parse.py | 新增（正则骨架 + LLM 拆分 + 交叉引用） |
|
||||
| review_service/app/services/rebuttal_draft.py | 新增（5 类 Prompt + 反幻觉校验 + letter） |
|
||||
| review_service/app/routers/rebuttal.py | 新增（10 端点） |
|
||||
| review_service/app/main.py | +import +include_router +lifespan 建表（约 8 行） |
|
||||
| review_service/tests/test_rebuttal.py | 新增（9 例） |
|
||||
| ui/src/api/review.js | +10 API 封装 + rebuttal 错误码未进 CODE_HINTS（后端原话直达） |
|
||||
| ui/src/stores/edu.js | +SECTIONS 'rebuttal' + rebuttal 状态块 + 10 actions |
|
||||
| ui/src/components/edu/panels/RebuttalPanel.vue | 新增（拆条树/逐条表单/校验红条/draft 预览/手工拆合/letter） |
|
||||
| ui/src/components/edu/EduSideNav.vue | +1 行菜单（审稿回复助手 💬） |
|
||||
| ui/src/components/edu/EduWorkspace.vue | +import +panelMap 挂载 |
|
||||
| tests/frontend/unit/edu/rebuttal.panel.test.js | 新增（5 例） |
|
||||
| docs/pr-reports/2026-10-12-rebuttal-phase1.md | 本报告 |
|
||||
|
||||
## git status --short（新增部分）
|
||||
|
||||
```
|
||||
?? docs/pr-reports/2026-10-12-rebuttal-phase1.md
|
||||
?? review_service/app/routers/rebuttal.py
|
||||
?? review_service/app/services/rebuttal_draft.py
|
||||
?? review_service/app/services/rebuttal_parse.py
|
||||
?? review_service/app/services/rebuttal_store.py
|
||||
?? review_service/tests/test_rebuttal.py
|
||||
?? tests/frontend/unit/edu/rebuttal.panel.test.js
|
||||
?? ui/src/components/edu/panels/RebuttalPanel.vue
|
||||
M review_service/app/main.py
|
||||
M ui/src/api/review.js
|
||||
M ui/src/components/edu/EduSideNav.vue
|
||||
M ui/src/components/edu/EduWorkspace.vue
|
||||
M ui/src/stores/edu.js
|
||||
（此前 AI 预审稿修复的未 commit 改动保持原样，未混入本报告范围）
|
||||
```
|
||||
|
||||
## 已知边界（Phase 1 明确不做）
|
||||
|
||||
- docx 导出（Phase 2，python-docx 已在 lock 备用）；
|
||||
- job TTL 自动清理（决策 5：手动 DELETE 替代）；
|
||||
- 跨 Reviewer 合并（merge 限同 Reviewer 内）；
|
||||
- 刷新页面后 job 恢复（纯内存态；GET /jobs/{id} 已备，Phase 2 可接列表入口）。
|
||||
@@ -1,13 +0,0 @@
|
||||
Reviewer #1:
|
||||
|
||||
1. The GWAS analysis lacks multiple-testing correction. Please re-run with FDR control and report q-values for all significant loci. In addition, the sample size justification is missing — please add a power analysis or at least discuss the detectable effect size.
|
||||
|
||||
2. Figure 3 is blurry and the axis labels are unreadable. As noted in Comment 3, the same issue applies to Figure S2. Please replace all figures with high-resolution versions.
|
||||
|
||||
3. Minor: please add the standard error estimates to Supplementary Table 2.
|
||||
|
||||
Reviewer #2:
|
||||
|
||||
1. The claim of novelty overstates the advance relative to Zhang et al. 2023. The introduction should more clearly delineate what is new.
|
||||
|
||||
2. How were the batch effects between the two sequencing centers corrected? This is essential to support the conclusions.
|
||||
@@ -1 +0,0 @@
|
||||
"""paper_writer_service —— 论文撰写独立 MCP 服务（8097）。"""
|
||||
@@ -1,53 +0,0 @@
|
||||
"""config.py —— paper 服务（8097）环境变量驱动配置（pydantic-settings）。
|
||||
|
||||
全部键可用环境变量覆盖（前缀 PAPER_；LLM 沿用 docgen 的 DOCGEN_LLM_* 直连模式，
|
||||
由 services/paper/llm.py 直接读 os.environ）。零硬编码宿主路径：相对路径按 __file__ 推导。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
_SERVICE_ROOT = Path(__file__).resolve().parent.parent # paper_writer_service/
|
||||
|
||||
|
||||
def _resolve(v: str) -> str:
|
||||
"""仓库内相对路径按 __file__ 推导服务根解析为绝对路径。"""
|
||||
p = Path(v).expanduser()
|
||||
if not p.is_absolute():
|
||||
p = _SERVICE_ROOT / p
|
||||
return str(p)
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(
|
||||
env_file=".env",
|
||||
env_file_encoding="utf-8",
|
||||
extra="ignore",
|
||||
env_prefix="PAPER_",
|
||||
)
|
||||
|
||||
app_name: str = "plantai-paper-writer-service"
|
||||
version: str = "0.1.0"
|
||||
|
||||
db_path: str = "paper.db" # 独立 SQLite（与 BFF 完全隔离）
|
||||
vstore_base_url: str = "http://127.0.0.1:8094"
|
||||
read_timeout: float = 360.0 # write_paper_section 同步长调用读超时
|
||||
|
||||
@property
|
||||
def db_path_abs(self) -> str:
|
||||
return _resolve(self.db_path)
|
||||
|
||||
@property
|
||||
def vstore_base(self) -> str:
|
||||
return self.vstore_base_url.rstrip("/")
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_settings() -> Settings:
|
||||
return Settings()
|
||||
|
||||
|
||||
settings = get_settings()
|
||||
@@ -1,168 +0,0 @@
|
||||
"""db.py —— paper 服务独立 SQLite（aiosqlite + WAL，无 ORM），三张表。
|
||||
|
||||
- papers：论文主表（brief/outline/final_markdown/references/disclaimer）
|
||||
- paper_sections：章节表（逐节 markdown + 状态）
|
||||
- citations：结构化引用（temp_n 生成期全局递增，final_n complete 按出现序重排写回）
|
||||
|
||||
模式沿用 bioflow/docgen 同款：CREATE TABLE IF NOT EXISTS + aiosqlite 单连接 + WAL + 写锁。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import AsyncIterator, Iterable
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import aiosqlite
|
||||
|
||||
from .config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS papers (
|
||||
id TEXT PRIMARY KEY,
|
||||
session_id TEXT NOT NULL DEFAULT '',
|
||||
title TEXT NOT NULL DEFAULT '',
|
||||
paper_type TEXT NOT NULL DEFAULT '',
|
||||
language TEXT NOT NULL DEFAULT '',
|
||||
format_req TEXT NOT NULL DEFAULT '',
|
||||
style TEXT NOT NULL DEFAULT '',
|
||||
status TEXT NOT NULL DEFAULT 'outline',
|
||||
brief_json TEXT NOT NULL DEFAULT '{}',
|
||||
outline_json TEXT NOT NULL DEFAULT '{}',
|
||||
final_markdown TEXT NOT NULL DEFAULT '',
|
||||
references_json TEXT NOT NULL DEFAULT '[]',
|
||||
disclaimer TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_papers_session ON papers(session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_papers_status ON papers(status);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS paper_sections (
|
||||
id TEXT PRIMARY KEY,
|
||||
paper_id TEXT NOT NULL REFERENCES papers(id) ON DELETE CASCADE,
|
||||
section_id TEXT NOT NULL,
|
||||
title TEXT NOT NULL DEFAULT '',
|
||||
key_points TEXT NOT NULL DEFAULT '[]',
|
||||
status TEXT NOT NULL DEFAULT 'pending',
|
||||
markdown TEXT NOT NULL DEFAULT '',
|
||||
word_count INTEGER NOT NULL DEFAULT 0,
|
||||
updated_at TEXT NOT NULL,
|
||||
UNIQUE(paper_id, section_id)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_paper_sections_paper ON paper_sections(paper_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS citations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
paper_id TEXT NOT NULL REFERENCES papers(id) ON DELETE CASCADE,
|
||||
section_id TEXT NOT NULL DEFAULT '',
|
||||
temp_n INTEGER,
|
||||
final_n INTEGER,
|
||||
kind TEXT NOT NULL DEFAULT 'title',
|
||||
ref_text TEXT NOT NULL DEFAULT '',
|
||||
source_doc_id TEXT NOT NULL DEFAULT '',
|
||||
snippet TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_citations_paper ON citations(paper_id);
|
||||
"""
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def dumps_json(value: Any, default: Any = None) -> str:
|
||||
try:
|
||||
return json.dumps(value, ensure_ascii=False)
|
||||
except (TypeError, ValueError):
|
||||
return json.dumps(default, ensure_ascii=False)
|
||||
|
||||
|
||||
def loads_json(text: str | None, default: Any = None) -> Any:
|
||||
if text is None:
|
||||
return default
|
||||
try:
|
||||
return json.loads(text)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
class Database:
|
||||
"""aiosqlite 单连接封装（WAL；写串行化，同 bioflow 模式）。"""
|
||||
|
||||
def __init__(self, path: str | Path | None = None) -> None:
|
||||
self._path: Path | None = Path(path) if path is not None else None
|
||||
self._conn: aiosqlite.Connection | None = None
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
@property
|
||||
def path(self) -> Path:
|
||||
if self._path is None:
|
||||
self._path = Path(get_settings().db_path_abs)
|
||||
return self._path
|
||||
|
||||
async def connect(self) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._conn = await aiosqlite.connect(self.path)
|
||||
self._conn.row_factory = aiosqlite.Row
|
||||
await self._conn.execute("PRAGMA journal_mode=WAL")
|
||||
await self._conn.execute("PRAGMA foreign_keys=ON")
|
||||
await self._conn.execute("PRAGMA busy_timeout=5000")
|
||||
await self._conn.executescript(_SCHEMA)
|
||||
await self._conn.commit()
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._conn is not None:
|
||||
await self._conn.close()
|
||||
self._conn = None
|
||||
|
||||
@property
|
||||
def conn(self) -> aiosqlite.Connection:
|
||||
if self._conn is None:
|
||||
raise RuntimeError("Database 未初始化：先 await init_db()")
|
||||
return self._conn
|
||||
|
||||
async def fetchall(self, sql: str, params: Iterable[Any] = ()) -> list[aiosqlite.Row]:
|
||||
async with self._conn.execute(sql, tuple(params)) as cursor:
|
||||
return list(await cursor.fetchall())
|
||||
|
||||
async def fetchone(self, sql: str, params: Iterable[Any] = ()) -> aiosqlite.Row | None:
|
||||
async with self._conn.execute(sql, tuple(params)) as cursor:
|
||||
return await cursor.fetchone()
|
||||
|
||||
async def execute(self, sql: str, params: Iterable[Any] = ()) -> int:
|
||||
async with self._lock:
|
||||
cur = await self._conn.execute(sql, tuple(params))
|
||||
await self._conn.commit()
|
||||
return cur.lastrowid
|
||||
|
||||
|
||||
db = Database()
|
||||
|
||||
|
||||
async def init_db(path: str | Path | None = None) -> Database:
|
||||
# 原地改单例路径后连接（不做 db = Database(path) 重赋值）：
|
||||
# 服务模块以 `from ...db import db` 持有了该单例引用，重赋值会让它们拿到旧对象。
|
||||
if path is not None:
|
||||
db._path = Path(path)
|
||||
await db.connect()
|
||||
return db
|
||||
|
||||
|
||||
async def get_db() -> AsyncIterator[Database]:
|
||||
yield db
|
||||
|
||||
|
||||
async def close_db() -> None:
|
||||
await db.close()
|
||||
|
||||
|
||||
def monotonic() -> float:
|
||||
return time.monotonic()
|
||||
@@ -1,84 +0,0 @@
|
||||
"""main.py —— paper 服务应用入口（uvicorn 入口，明文白名单文件）。
|
||||
|
||||
- 独立部署树 paper_writer_service/（FR：独立进程 8097 + 独立 SQLite paper.db）。
|
||||
- lifespan：init_db（幂等建表）→ 显式进入 MCP session_manager.run()（Starlette 不自动
|
||||
运行子 Mount 的 lifespan，streamable-http 处理请求时要求 task group 已初始化）→ 清理。
|
||||
- 端点：/health（GET，BFF 探活反代）+ /mcp（MCP streamable-http）。
|
||||
- 不依赖 BFF/生产服务（8090 等绝不探测/连接）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from .config import get_settings
|
||||
from .db import close_db, db, init_db
|
||||
from .mcp import paper_mcp
|
||||
from .services.paper import search
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# MCP streamable-http app（调用即创建 session_manager，供 lifespan 进入 run()）。
|
||||
# 端点地址：/mcp（streamable_http_path="/" + 挂载前缀 /mcp）。
|
||||
_paper_mcp_app = paper_mcp.streamable_http_app(streamable_http_path="/")
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
settings = get_settings()
|
||||
app = FastAPI(title=settings.app_name, version=settings.version)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=False,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
@asynccontextmanager
|
||||
async def _lifespan(_app: FastAPI):
|
||||
# 测试注入：db 已由测试预置（init_db(tmp_path)），跳过启动初始化。
|
||||
try:
|
||||
await init_db()
|
||||
except Exception:
|
||||
logger.exception("main: 数据库初始化失败（health 将报 db 降级）")
|
||||
|
||||
# MCP session manager 的 task group 必须在服务运行期内保持活跃。
|
||||
async with paper_mcp.session_manager.run():
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
try:
|
||||
await close_db()
|
||||
except Exception:
|
||||
pass
|
||||
await search.close_client()
|
||||
|
||||
app.router.lifespan_context = _lifespan
|
||||
|
||||
@app.get("/health")
|
||||
async def health() -> dict[str, object]:
|
||||
db_ok = True
|
||||
db_error: str | None = None
|
||||
try:
|
||||
await db.fetchone("SELECT 1 AS ok")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
db_ok = False
|
||||
db_error = str(exc)
|
||||
return {
|
||||
"ok": True,
|
||||
"status": "ok" if db_ok else "degraded",
|
||||
"db": {"connected": db_ok, "error": db_error},
|
||||
"version": settings.version,
|
||||
}
|
||||
|
||||
# paper MCP 能力外挂（mcp 2.x：MCPServer.streamable_http_app）。
|
||||
app.mount("/mcp", _paper_mcp_app)
|
||||
return app
|
||||
|
||||
|
||||
app = create_app()
|
||||
@@ -1,48 +0,0 @@
|
||||
"""app/mcp.py —— paper_writer_service MCP server（单 streamable-http 端点 /mcp）。
|
||||
|
||||
9 个工具：propose_paper_outline / write_paper_section / get_paper_progress / search_refs /
|
||||
revise_paper_section / revise_paper / get_paper / complete_paper / cancel_paper。
|
||||
|
||||
mcp 2.x 注意：MCPServer（非 FastMCP，来自 mcp.server.mcpserver）；工具返回 dict，由 MCP
|
||||
序列化为 CallToolResult（structuredContent + content:[{type:"text",text:"<json>"}]）。
|
||||
本文件位于 app/ 层（非 services/），可用 pydantic（mcp 自身依赖 pydantic 构造 schema）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from .services.paper import complete, outline, search, write
|
||||
|
||||
|
||||
def build_paper_mcp() -> Any:
|
||||
from mcp.server.mcpserver import MCPServer
|
||||
|
||||
mcp = MCPServer(
|
||||
"paper-writer",
|
||||
title="PlantAI 论文撰写",
|
||||
description="论文撰写能力单元：大纲生成/逐节撰写/文献引用/全文合并重排/修订。",
|
||||
)
|
||||
|
||||
mcp.tool(name="propose_paper_outline",
|
||||
description="根据论文主题与写作要求生成大纲并建论文（内部检索知识库注入选题线索）。")(outline.propose_paper_outline)
|
||||
mcp.tool(name="write_paper_section",
|
||||
description="撰写论文某一章节（同步长调用，90-180s）。按全局引用编号表生成正文与引用。")(write.write_paper_section)
|
||||
mcp.tool(name="get_paper_progress",
|
||||
description="查询论文与各章节撰写进度/字数/引用数。")(complete.get_paper_progress)
|
||||
mcp.tool(name="search_refs",
|
||||
description="在知识库(vstore)中检索与查询相关的文献片段。")(search.search_refs)
|
||||
mcp.tool(name="revise_paper_section",
|
||||
description="按用户意见重写论文某一章节，新引用追加到引用表末尾。")(write.revise_paper_section)
|
||||
mcp.tool(name="revise_paper",
|
||||
description="按用户意见判定需要修改哪些章节，返回受影响节清单与修订计划。")(write.revise_paper)
|
||||
mcp.tool(name="get_paper",
|
||||
description="拉取论文全文 markdown、参考文献列表与免责声明。")(complete.get_paper)
|
||||
mcp.tool(name="complete_paper",
|
||||
description="合并全文、按首次出现顺序重排引用编号、渲染 GB/T 7714 文献列表。")(complete.complete_paper)
|
||||
mcp.tool(name="cancel_paper",
|
||||
description="取消论文（置 cancelled）。")(complete.cancel_paper)
|
||||
|
||||
return mcp
|
||||
|
||||
|
||||
paper_mcp = build_paper_mcp()
|
||||
@@ -1,10 +0,0 @@
|
||||
"""schemas.py —— paper 服务共享常量（明文白名单文件）。
|
||||
|
||||
MCP 工具入参/出参均为 dict（由 mcp 层序列化为 content:[{type:text,text:<json>}]），
|
||||
本文件只放状态/引用类型常量，避免服务层散落魔法字面量。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
PAPER_STATUSES = ("outline", "writing", "complete", "cancelled", "error")
|
||||
SECTION_STATUSES = ("pending", "writing", "done", "error")
|
||||
CITATION_KINDS = ("doi", "pmid", "title")
|
||||
@@ -1,65 +0,0 @@
|
||||
"""cite.py —— 引用协议：GB/T 7714 渲染、[n] 校验、complete 按出现序重排。"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
_CITE_RE = re.compile(r"\[(\d+)\]")
|
||||
|
||||
|
||||
def find_cite_numbers(markdown: str | None) -> list[int]:
|
||||
"""正文中出现的所有 [n] 编号（含重复，按出现顺序）。"""
|
||||
return [int(m) for m in _CITE_RE.findall(markdown or "")]
|
||||
|
||||
|
||||
def render_gbt7714(meta: dict[str, Any], *, fallback_year: str = "2026") -> str:
|
||||
"""按可用元数据渲染 GB/T 7714 近似条目。
|
||||
|
||||
- 有 DOI → [J/OL] + DOI
|
||||
- 有 PMID → [J] + PMID
|
||||
- 无 DOI/PMID（知识库资料）→ 降级格式 + [EB/OL]
|
||||
"""
|
||||
title = (meta.get("title") or "").strip()
|
||||
journal = (meta.get("journal") or "").strip()
|
||||
authors = (meta.get("authors") or "").strip()
|
||||
year = (meta.get("year") or fallback_year).strip()
|
||||
doi = (meta.get("doi") or "").strip()
|
||||
pmid = (meta.get("pmid") or "").strip()
|
||||
head = f"{authors} " if authors else ""
|
||||
if doi:
|
||||
jpart = f"{journal}, {year}." if journal else f"{year}."
|
||||
return f"{head}{title}[J/OL]. {jpart} DOI:{doi}."
|
||||
if pmid:
|
||||
jpart = f"{journal}, {year}." if journal else f"{year}."
|
||||
return f"{head}{title}[J]. {jpart} PMID:{pmid}."
|
||||
return f"{head}{title}. 知识库资料, {year}. [EB/OL]."
|
||||
|
||||
|
||||
def render_disclaimer() -> str:
|
||||
return "参考文献依据知识库资料生成，格式为 GB/T 7714 近似，建议人工核对。"
|
||||
|
||||
|
||||
def first_appearance_order(full_markdown: str, temp_numbers: list[int]) -> list[int]:
|
||||
"""按全文首次出现顺序返回 temp_n；未出现的排最后（保持原序）。"""
|
||||
seen: dict[int, int] = {}
|
||||
for m in _CITE_RE.finditer(full_markdown or ""):
|
||||
n = int(m.group(1))
|
||||
if n not in seen and n in temp_numbers:
|
||||
seen[n] = m.start()
|
||||
ordered = sorted((n for n in temp_numbers if n in seen), key=lambda n: seen[n])
|
||||
ordered += [n for n in temp_numbers if n not in seen]
|
||||
return ordered
|
||||
|
||||
|
||||
def renumber_map(temp_numbers: list[int], full_markdown: str) -> dict[int, int]:
|
||||
"""temp_n → final_n（按全文首次出现顺序 1..N）。"""
|
||||
ordered = first_appearance_order(full_markdown, temp_numbers)
|
||||
return {temp: final for final, temp in enumerate(ordered, start=1)}
|
||||
|
||||
|
||||
def replace_numbers(markdown: str | None, mapping: dict[int, int]) -> str:
|
||||
def _sub(m: re.Match[str]) -> str:
|
||||
n = int(m.group(1))
|
||||
return f"[{mapping.get(n, n)}]"
|
||||
|
||||
return _CITE_RE.sub(_sub, markdown or "")
|
||||
@@ -1,109 +0,0 @@
|
||||
"""complete.py —— get_paper / get_paper_progress / complete_paper / cancel_paper。"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from ...db import db, dumps_json, loads_json, now_iso
|
||||
from .cite import render_disclaimer, renumber_map, replace_numbers
|
||||
|
||||
|
||||
async def _load(paper_id: str) -> dict[str, Any]:
|
||||
paper = await db.fetchone("SELECT * FROM papers WHERE id = ?", (paper_id,))
|
||||
if not paper:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
|
||||
sections = [dict(r) for r in await db.fetchall(
|
||||
"SELECT * FROM paper_sections WHERE paper_id = ? ORDER BY rowid", (paper_id,))]
|
||||
citations = [dict(r) for r in await db.fetchall(
|
||||
"SELECT * FROM citations WHERE paper_id = ? ORDER BY temp_n", (paper_id,))]
|
||||
return {"ok": True, "paper": dict(paper), "sections": sections, "citations": citations}
|
||||
|
||||
|
||||
def _merge_sections(paper: dict[str, Any], sections: list[dict[str, Any]]) -> str:
|
||||
parts = [f"# {paper.get('title', '')}"]
|
||||
for s in sections:
|
||||
parts.append(f"\n## {s['title']}\n\n{s.get('markdown', '')}")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
async def get_paper(paper_id: str, format: str = "md") -> dict[str, Any]:
|
||||
p = await _load(paper_id)
|
||||
if not p.get("ok"):
|
||||
return p
|
||||
paper, sections, citations = p["paper"], p["sections"], p["citations"]
|
||||
full_markdown = paper.get("final_markdown") or _merge_sections(paper, sections)
|
||||
references = loads_json(paper.get("references_json"), []) or [
|
||||
{"n": c.get("final_n") or c.get("temp_n"), "text": c["ref_text"], "kind": c["kind"],
|
||||
"source_doc_id": c["source_doc_id"]} for c in citations
|
||||
]
|
||||
brief = loads_json(paper.get("brief_json"), {})
|
||||
return {
|
||||
"ok": True, "paper_id": paper_id, "title": paper["title"], "status": paper["status"],
|
||||
"template_key": brief.get("template_key") or "document",
|
||||
"brief": brief,
|
||||
"sections": [
|
||||
{"section_id": s["section_id"], "title": s["title"], "status": s["status"],
|
||||
"word_count": s["word_count"]} for s in sections
|
||||
],
|
||||
"full_markdown": full_markdown, "references": references,
|
||||
"disclaimer": paper.get("disclaimer", ""),
|
||||
}
|
||||
|
||||
|
||||
async def get_paper_progress(paper_id: str) -> dict[str, Any]:
|
||||
paper = await db.fetchone("SELECT * FROM papers WHERE id = ?", (paper_id,))
|
||||
if not paper:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
|
||||
paper = dict(paper)
|
||||
sections = [dict(r) for r in await db.fetchall(
|
||||
"SELECT * FROM paper_sections WHERE paper_id = ? ORDER BY rowid", (paper_id,))]
|
||||
cit_rows = [dict(r) for r in await db.fetchall(
|
||||
"SELECT section_id, COUNT(*) AS c FROM citations WHERE paper_id = ? GROUP BY section_id", (paper_id,))]
|
||||
cit_by_sec = {r["section_id"]: r["c"] for r in cit_rows}
|
||||
current = next((s["section_id"] for s in sections if s["status"] != "done"), None)
|
||||
return {
|
||||
"ok": True, "paper_id": paper_id, "status": paper["status"], "current_section_id": current,
|
||||
"sections": [
|
||||
{"section_id": s["section_id"], "title": s["title"], "status": s["status"],
|
||||
"word_count": s["word_count"], "citation_count": cit_by_sec.get(s["section_id"], 0)}
|
||||
for s in sections
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def complete_paper(paper_id: str) -> dict[str, Any]:
|
||||
p = await _load(paper_id)
|
||||
if not p.get("ok"):
|
||||
return p
|
||||
paper, sections, citations = p["paper"], p["sections"], p["citations"]
|
||||
full = _merge_sections(paper, sections)
|
||||
temp_numbers = [c.get("temp_n") for c in citations if c.get("temp_n")]
|
||||
mapping = renumber_map(temp_numbers, full)
|
||||
|
||||
final_full = replace_numbers(full, mapping)
|
||||
for c in citations:
|
||||
await db.execute("UPDATE citations SET final_n = ? WHERE id = ?", (mapping.get(c.get("temp_n")), c["id"]))
|
||||
for s in sections:
|
||||
await db.execute("UPDATE paper_sections SET markdown = ? WHERE id = ?",
|
||||
(replace_numbers(s.get("markdown"), mapping), s["id"]))
|
||||
|
||||
ordered = sorted(citations, key=lambda c: mapping.get(c.get("temp_n"), 0))
|
||||
references = [
|
||||
{"n": mapping.get(c.get("temp_n"), c.get("temp_n")), "text": c["ref_text"],
|
||||
"kind": c["kind"], "source_doc_id": c["source_doc_id"]} for c in ordered
|
||||
]
|
||||
has_degraded = any(c["kind"] == "title" for c in citations)
|
||||
disclaimer = render_disclaimer() if has_degraded else ""
|
||||
await db.execute(
|
||||
"UPDATE papers SET status = 'complete', final_markdown = ?, references_json = ?, disclaimer = ?, updated_at = ? WHERE id = ?",
|
||||
(final_full, dumps_json(references), disclaimer, now_iso(), paper_id),
|
||||
)
|
||||
return {"ok": True, "full_markdown": final_full, "references": references,
|
||||
"renumber_map": mapping, "disclaimer": disclaimer}
|
||||
|
||||
|
||||
async def cancel_paper(paper_id: str) -> dict[str, Any]:
|
||||
paper = await db.fetchone("SELECT id FROM papers WHERE id = ?", (paper_id,))
|
||||
if not paper:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
|
||||
await db.execute("UPDATE papers SET status = 'cancelled', updated_at = ? WHERE id = ?", (now_iso(), paper_id))
|
||||
return {"ok": True}
|
||||
@@ -1,187 +0,0 @@
|
||||
"""services/paper/llm.py —— LLM 调用（httpx 裸调，拷贝自 sdk/app/services/docgen/llm.py）。
|
||||
|
||||
- complete_text(system, user, *, temperature=0.3, max_tokens=8000) -> str
|
||||
- complete_json(...)（response_format json_object + 一次 JSON 修复重试）
|
||||
- 4 次尝试（1.5s / 3s / 6s 指数退避，429 / 5xx / 超时）
|
||||
- env 键：DOCGEN_LLM_BASE_URL / DOCGEN_LLM_API_KEY / DOCGEN_LLM_MODEL / DOCGEN_LLM_TIMEOUT
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_LLM_BASE_URL = "https://api.deepseek.com/v1"
|
||||
DEFAULT_LLM_MODEL = "deepseek-chat"
|
||||
DEFAULT_LLM_TIMEOUT = 300.0
|
||||
BACKOFF_SECONDS = (1.5, 3.0, 6.0)
|
||||
MAX_ATTEMPTS = 4
|
||||
|
||||
|
||||
class LLMError(Exception):
|
||||
"""LLM 调用失败（重试尽/响应异常）。"""
|
||||
|
||||
|
||||
def _llm_config() -> dict[str, Any]:
|
||||
return {
|
||||
"base_url": (os.environ.get("DOCGEN_LLM_BASE_URL") or DEFAULT_LLM_BASE_URL).rstrip("/"),
|
||||
"api_key": os.environ.get("DOCGEN_LLM_API_KEY", ""),
|
||||
"model": os.environ.get("DOCGEN_LLM_MODEL") or DEFAULT_LLM_MODEL,
|
||||
"timeout": float(os.environ.get("DOCGEN_LLM_TIMEOUT") or DEFAULT_LLM_TIMEOUT),
|
||||
}
|
||||
|
||||
|
||||
def _extract_text(payload: dict[str, Any]) -> str:
|
||||
try:
|
||||
choices = payload.get("choices") or []
|
||||
if choices:
|
||||
content = choices[0].get("message", {}).get("content")
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
return "".join(
|
||||
str(c.get("text", "")) if isinstance(c, dict) else str(c) for c in content
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
return ""
|
||||
|
||||
|
||||
async def _chat_once(
|
||||
client: httpx.AsyncClient,
|
||||
cfg: dict[str, Any],
|
||||
system: str,
|
||||
user: str,
|
||||
*,
|
||||
temperature: float,
|
||||
max_tokens: int,
|
||||
json_mode: bool,
|
||||
timeout: float | None = None,
|
||||
) -> str:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if cfg["api_key"]:
|
||||
headers["Authorization"] = f"Bearer {cfg['api_key']}"
|
||||
body: dict[str, Any] = {
|
||||
"model": cfg["model"],
|
||||
"messages": [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
],
|
||||
"temperature": temperature,
|
||||
"max_tokens": max_tokens,
|
||||
}
|
||||
if json_mode:
|
||||
body["response_format"] = {"type": "json_object"}
|
||||
resp = await client.post(
|
||||
f"{cfg['base_url']}/chat/completions",
|
||||
headers=headers,
|
||||
json=body,
|
||||
timeout=timeout if timeout is not None else cfg["timeout"],
|
||||
)
|
||||
resp.raise_for_status()
|
||||
payload = resp.json()
|
||||
if not isinstance(payload, dict):
|
||||
raise LLMError("LLM 响应形态异常（非 JSON 对象）")
|
||||
text = _extract_text(payload)
|
||||
if not text:
|
||||
raise LLMError("LLM 响应缺少 choices[0].message.content")
|
||||
return text
|
||||
|
||||
|
||||
async def _complete(
|
||||
system: str,
|
||||
user: str,
|
||||
*,
|
||||
temperature: float,
|
||||
max_tokens: int,
|
||||
json_mode: bool,
|
||||
timeout: float | None = None,
|
||||
) -> str:
|
||||
cfg = _llm_config()
|
||||
last_exc: BaseException | None = None
|
||||
async with httpx.AsyncClient() as client:
|
||||
for attempt in range(MAX_ATTEMPTS):
|
||||
try:
|
||||
return await _chat_once(
|
||||
client, cfg, system, user,
|
||||
temperature=temperature, max_tokens=max_tokens, json_mode=json_mode,
|
||||
timeout=timeout,
|
||||
)
|
||||
except LLMError:
|
||||
raise
|
||||
except (httpx.HTTPStatusError, httpx.HTTPError, httpx.TimeoutException) as exc:
|
||||
last_exc = exc
|
||||
status = getattr(getattr(exc, "response", None), "status_code", None)
|
||||
retryable = status is None or status == 429 or status >= 500
|
||||
if not retryable or attempt == MAX_ATTEMPTS - 1:
|
||||
raise LLMError(f"LLM 调用失败（HTTP {status or '网络'}）：{exc}") from exc
|
||||
await asyncio.sleep(BACKOFF_SECONDS[min(attempt, len(BACKOFF_SECONDS) - 1)])
|
||||
raise LLMError(f"LLM 调用失败：{last_exc}")
|
||||
|
||||
|
||||
async def complete_text(
|
||||
system: str,
|
||||
user: str,
|
||||
*,
|
||||
temperature: float = 0.3,
|
||||
max_tokens: int = 8000,
|
||||
timeout: float | None = None,
|
||||
) -> str:
|
||||
"""文本补全。失败抛 LLMError。"""
|
||||
return await _complete(
|
||||
system, user, temperature=temperature, max_tokens=max_tokens, json_mode=False,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def _repair_json(text: str) -> str:
|
||||
text = text.strip()
|
||||
if text.startswith("```"):
|
||||
text = text.strip("`")
|
||||
if text.lower().startswith("json"):
|
||||
text = text[4:]
|
||||
text = text.strip()
|
||||
if not text:
|
||||
raise LLMError("LLM 返回空内容")
|
||||
if text.startswith("{"):
|
||||
return text
|
||||
start = text.find("{")
|
||||
end = text.rfind("}")
|
||||
if start >= 0 and end > start:
|
||||
return text[start:end + 1]
|
||||
raise LLMError("LLM 输出中未找到 JSON 对象")
|
||||
|
||||
|
||||
async def complete_json(
|
||||
system: str,
|
||||
user: str,
|
||||
*,
|
||||
temperature: float = 0.3,
|
||||
max_tokens: int = 8000,
|
||||
) -> dict[str, Any]:
|
||||
"""JSON 补全（json_object + 一次修复重试）。"""
|
||||
text = await _complete(
|
||||
system, user, temperature=temperature, max_tokens=max_tokens, json_mode=True
|
||||
)
|
||||
try:
|
||||
return json.loads(text)
|
||||
except ValueError:
|
||||
pass
|
||||
text2 = await _complete(
|
||||
system + "\n（重要：你的回答必须是单个合法 JSON 对象，不要输出其他任何内容）",
|
||||
user,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
json_mode=True,
|
||||
)
|
||||
try:
|
||||
return json.loads(text2)
|
||||
except ValueError:
|
||||
repaired = _repair_json(text2)
|
||||
return json.loads(repaired)
|
||||
@@ -1,113 +0,0 @@
|
||||
"""outline.py —— propose_paper_outline：建 papers 行 → search_refs 注入线索 → LLM 出大纲 → 预填 sections。"""
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from typing import Any
|
||||
|
||||
from ...db import db, dumps_json, now_iso
|
||||
from . import llm, search
|
||||
|
||||
_OUTLINE_SYSTEM = (
|
||||
"你是植物学论文撰写助手。根据用户给出的论文主题与写作要求，生成论文大纲。"
|
||||
"只输出一个合法 JSON 对象，结构为："
|
||||
'{"title": 论文标题, "abstract_expect": 摘要预期内容(一句话), '
|
||||
'"sections": [{"section_id": "s1", "title": "节标题", "key_points": ["要点1", "要点2"], "target_words": 1200}, ...]}。'
|
||||
"章节 5-6 节，覆盖 引言/材料与方法/结果/讨论/结论 等综述常见结构；target_words 每节 1200-2500。"
|
||||
)
|
||||
|
||||
_DEFAULT_SECTIONS = [
|
||||
("s1", "引言"),
|
||||
("s2", "材料与方法"),
|
||||
("s3", "结果"),
|
||||
("s4", "讨论"),
|
||||
("s5", "结论与展望"),
|
||||
]
|
||||
|
||||
|
||||
async def propose_paper_outline(
|
||||
topic: str,
|
||||
requirements: str = "",
|
||||
paper_type: str = "",
|
||||
language: str = "",
|
||||
length: str = "",
|
||||
format_req: str = "",
|
||||
style: str = "",
|
||||
kb_hints: list[dict[str, Any]] | None = None,
|
||||
template_key: str = "",
|
||||
) -> dict[str, Any]:
|
||||
# 1. 知识库线索：优先用调用方传入的 kb_hints，否则现场 search_refs（失败静默降级）
|
||||
hints = list(kb_hints or [])
|
||||
if not hints:
|
||||
try:
|
||||
r = await search.search_refs(query=topic, top_k=8)
|
||||
hints = list(r.get("items") or [])
|
||||
except Exception:
|
||||
hints = []
|
||||
|
||||
# 2. LLM 出大纲
|
||||
user = f"论文主题：{topic}\n"
|
||||
if paper_type:
|
||||
user += f"论文类型：{paper_type}\n"
|
||||
if requirements:
|
||||
user += f"写作要求：{requirements}\n"
|
||||
if language:
|
||||
user += f"语言：{language}\n"
|
||||
if length:
|
||||
user += f"篇幅：{length}\n"
|
||||
if format_req:
|
||||
user += f"格式：{format_req}\n"
|
||||
if style:
|
||||
user += f"风格：{style}\n"
|
||||
if hints:
|
||||
user += "可参考知识库文献线索（仅选题依据，非硬性引用）：\n" + "\n".join(
|
||||
f"- {h.get('title', '')}: {str(h.get('snippet', ''))[:120]}" for h in hints[:5]
|
||||
)
|
||||
try:
|
||||
outline = await llm.complete_json(_OUTLINE_SYSTEM, user)
|
||||
except llm.LLMError as exc:
|
||||
return {"ok": False, "code": "LLM_ERROR", "message": f"大纲生成失败：{exc}"}
|
||||
|
||||
title = str(outline.get("title") or topic).strip() or topic
|
||||
sections = outline.get("sections") or []
|
||||
if not sections:
|
||||
sections = [
|
||||
{"section_id": sid, "title": t, "key_points": [], "target_words": 1500}
|
||||
for sid, t in _DEFAULT_SECTIONS
|
||||
]
|
||||
|
||||
# 3. 落库 papers + paper_sections
|
||||
paper_id = uuid.uuid4().hex[:16]
|
||||
await db.execute(
|
||||
"INSERT INTO papers (id, title, paper_type, language, format_req, style, status, brief_json, outline_json, created_at, updated_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, 'outline', ?, ?, ?, ?)",
|
||||
(
|
||||
paper_id, title, paper_type, language, format_req, style,
|
||||
dumps_json({
|
||||
"topic": topic, "requirements": requirements, "length": length,
|
||||
"kb_hints": hints, "template_key": template_key,
|
||||
}),
|
||||
dumps_json({
|
||||
"title": title,
|
||||
"abstract_expect": outline.get("abstract_expect", ""),
|
||||
"sections": sections,
|
||||
}),
|
||||
now_iso(), now_iso(),
|
||||
),
|
||||
)
|
||||
for s in sections:
|
||||
await db.execute(
|
||||
"INSERT INTO paper_sections (id, paper_id, section_id, title, key_points, status, updated_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, 'pending', ?)",
|
||||
(
|
||||
uuid.uuid4().hex[:16], paper_id,
|
||||
s.get("section_id") or "s1", s.get("title", ""),
|
||||
dumps_json(s.get("key_points") or []), now_iso(),
|
||||
),
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"paper_id": paper_id,
|
||||
"title": title,
|
||||
"abstract_expect": outline.get("abstract_expect", ""),
|
||||
"sections": sections,
|
||||
}
|
||||
@@ -1,101 +0,0 @@
|
||||
"""search.py —— vstore 8094 知识库检索客户端（search_refs 后端）。
|
||||
|
||||
- httpx 裸调 GET /api/vstore/search-chunks
|
||||
- 失败不抛异常：返回 {"ok": False, "error": {...}}，工具层降级为空结果
|
||||
- vstore chunk 条目无 doi/pmid/journal/year（仅 doc_id/doc_name/snippet/score），
|
||||
故 search_refs 的 meta 为尽力而为（通常为空）→ 引用多为 title kind → 降级格式 + disclaimer
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
DEFAULT_VSTORE_BASE_URL = "http://127.0.0.1:8094"
|
||||
RETRY_BACKOFF = (0.5, 1.0, 2.0)
|
||||
MAX_RETRIES = 2
|
||||
DEFAULT_TIMEOUT = 30.0
|
||||
|
||||
_client: httpx.AsyncClient | None = None
|
||||
|
||||
|
||||
def _base_url() -> str:
|
||||
return (os.environ.get("VSTORE_BASE_URL") or DEFAULT_VSTORE_BASE_URL).rstrip("/")
|
||||
|
||||
|
||||
def get_client() -> httpx.AsyncClient:
|
||||
global _client
|
||||
if _client is None:
|
||||
_client = httpx.AsyncClient(
|
||||
base_url=_base_url(), timeout=httpx.Timeout(DEFAULT_TIMEOUT, connect=5.0)
|
||||
)
|
||||
return _client
|
||||
|
||||
|
||||
def set_client(client: httpx.AsyncClient) -> None:
|
||||
"""注入 client（测试用 MockTransport）。"""
|
||||
global _client
|
||||
_client = client
|
||||
|
||||
|
||||
async def close_client() -> None:
|
||||
global _client
|
||||
if _client is not None:
|
||||
await _client.aclose()
|
||||
_client = None
|
||||
|
||||
|
||||
async def search_chunks(
|
||||
query: str,
|
||||
*,
|
||||
top_k: int = 8,
|
||||
min_score: float = 0.30,
|
||||
timeout: float = 12.0,
|
||||
) -> dict[str, Any]:
|
||||
"""GET /api/vstore/search-chunks。失败返回 {ok:False, error:{...}}（不抛）。"""
|
||||
client = get_client()
|
||||
last_exc: BaseException | None = None
|
||||
for attempt in range(MAX_RETRIES + 1):
|
||||
try:
|
||||
resp = await client.get(
|
||||
"/api/vstore/search-chunks",
|
||||
params={"q": query, "top_k": top_k, "min_score": min_score},
|
||||
timeout=timeout,
|
||||
)
|
||||
if resp.status_code < 500:
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError:
|
||||
return {"ok": False, "error": {"code": "BAD_RESPONSE", "message": "vstore 响应非 JSON"}}
|
||||
if not isinstance(payload, dict):
|
||||
return {"ok": False, "error": {"code": "BAD_RESPONSE", "message": "vstore 响应形态异常"}}
|
||||
return payload
|
||||
last_exc = RuntimeError(f"vstore {resp.status_code}")
|
||||
except (httpx.HTTPError, httpx.TimeoutException) as exc:
|
||||
last_exc = exc
|
||||
if attempt < MAX_RETRIES:
|
||||
await asyncio.sleep(RETRY_BACKOFF[min(attempt, len(RETRY_BACKOFF) - 1)])
|
||||
return {"ok": False, "error": {"code": "UPSTREAM_DOWN", "message": f"vstore 不可达：{last_exc}"}}
|
||||
|
||||
|
||||
async def search_refs(paper_id: str = "", query: str = "", top_k: int = 8) -> dict[str, Any]:
|
||||
"""search_refs 工具：透传 vstore 检索，映射为 {doc_id,title,snippet,score,meta}。"""
|
||||
if not query:
|
||||
return {"ok": False, "code": "BAD_REQUEST", "message": "缺少检索词 query"}
|
||||
r = await search_chunks(query, top_k=max(1, min(int(top_k), 20)))
|
||||
if not r.get("ok"):
|
||||
return {"ok": True, "items": [], "warning": (r.get("error") or {}).get("message", "")}
|
||||
items: list[dict[str, Any]] = []
|
||||
for it in r.get("items") or []:
|
||||
if not isinstance(it, dict):
|
||||
continue
|
||||
items.append({
|
||||
"doc_id": it.get("doc_id") or "",
|
||||
"title": it.get("doc_name") or it.get("doc_id") or "",
|
||||
"snippet": it.get("snippet") or "",
|
||||
"score": it.get("score"),
|
||||
"meta": {}, # vstore chunk 无 doi/pmid/journal/year，尽力为空
|
||||
})
|
||||
return {"ok": True, "items": items}
|
||||
@@ -1,251 +0,0 @@
|
||||
"""write.py —— 章节撰写与修订（write_paper_section / revise_paper_section / revise_paper）。"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from ...db import db, dumps_json, loads_json, now_iso
|
||||
from . import llm
|
||||
from .cite import find_cite_numbers, render_gbt7714
|
||||
|
||||
_WRITE_SYSTEM = (
|
||||
"你是植物学论文撰写助手，正在撰写论文的某一章节。"
|
||||
"引用文献一律用 [n] 指代，n 必须取自下方提供的全局编号列表；如需引用新文献，编号按当前最大编号依次 +1。"
|
||||
"你只能引用「知识库线索」中列出的文献；新引用的元数据只能照抄线索里的 title 和 doc_id。"
|
||||
"知识库线索里没有 DOI/PMID/作者/期刊/年份，这些字段一律输出空字符串，禁止编造任何 DOI/PMID/作者/期刊。"
|
||||
"只输出一个合法 JSON 对象：{\"markdown\": 章节正文(不含章标题), \"citations_used\": "
|
||||
"[{\"temp_n\": 编号, \"title\": \"...\", \"authors\": \"\", \"journal\": \"\", "
|
||||
"\"year\": \"\", \"doi\": \"\", \"pmid\": \"\", \"source_doc_id\": \"...\", \"snippet\": \"...\"}]}。"
|
||||
"citations_used 必须列出本节正文中出现的每一个 [n] 引用（既有编号也列出其已有元数据）。"
|
||||
)
|
||||
|
||||
_REVISE_SYSTEM = (
|
||||
"你是植物学论文撰写助手，正在按用户意见修订论文的某一章节。"
|
||||
"保持引用协议不变：[n] 指代，n 只能取自全局编号列表或最大编号 +1；禁止杜撰元数据。"
|
||||
"只输出一个合法 JSON 对象：{\"markdown\": 修订后的章节正文, \"citations_used\": [...]}。"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 读取辅助
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _get_paper(paper_id: str) -> dict[str, Any] | None:
|
||||
row = await db.fetchone("SELECT * FROM papers WHERE id = ?", (paper_id,))
|
||||
return dict(row) if row else None
|
||||
|
||||
|
||||
async def _get_sections(paper_id: str) -> list[dict[str, Any]]:
|
||||
rows = await db.fetchall(
|
||||
"SELECT * FROM paper_sections WHERE paper_id = ? ORDER BY rowid", (paper_id,)
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
|
||||
async def _get_citations(paper_id: str) -> list[dict[str, Any]]:
|
||||
rows = await db.fetchall(
|
||||
"SELECT * FROM citations WHERE paper_id = ? ORDER BY temp_n", (paper_id,)
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
|
||||
def _global_citations_text(citations: list[dict[str, Any]]) -> str:
|
||||
if not citations:
|
||||
return "（当前无已编号文献；新引用从 [1] 开始）"
|
||||
return "\n".join(f"[{c['temp_n']}] {c['ref_text']}" for c in citations)
|
||||
|
||||
|
||||
def _max_temp_n(citations: list[dict[str, Any]]) -> int:
|
||||
return max((c.get("temp_n") or 0 for c in citations), default=0)
|
||||
|
||||
|
||||
def _build_write_user(paper: dict, outline: dict, sec: dict, cit_text: str, style: str, cite_mode: str) -> str:
|
||||
parts: list[str] = []
|
||||
parts.append(f"论文标题：{paper.get('title', '')}")
|
||||
parts.append(f"论文类型：{paper.get('paper_type') or '综述'}")
|
||||
parts.append(f"语言：{paper.get('language') or '中文'}")
|
||||
parts.append(f"格式要求：{paper.get('format_req') or '无'}")
|
||||
parts.append(f"写作风格：{style or paper.get('style') or '学术严谨'}")
|
||||
parts.append("")
|
||||
parts.append("论文大纲：")
|
||||
for s in outline.get("sections") or []:
|
||||
mark = "（当前撰写节）" if s.get("section_id") == sec.get("section_id") else ""
|
||||
parts.append(f"- {s.get('section_id')}: {s.get('title')}{mark} 要点：{'；'.join(s.get('key_points') or [])}")
|
||||
parts.append("")
|
||||
parts.append(f"现在撰写章节「{sec.get('title', '')}」（section_id={sec.get('section_id')}）。")
|
||||
if sec.get("key_points"):
|
||||
parts.append("本节要点：" + "；".join(loads_json(sec.get("key_points"), [])))
|
||||
parts.append("")
|
||||
parts.append("全局已编号文献列表（[n] 引用只能从这里取值，或从最大编号 +1 起新增）：")
|
||||
parts.append(cit_text)
|
||||
brief = loads_json(paper.get("brief_json"), {})
|
||||
kb_hints = brief.get("kb_hints") or []
|
||||
if kb_hints:
|
||||
parts.append("")
|
||||
parts.append("知识库线索（新引用只能从这里选，元数据只照抄 title/doc_id）：")
|
||||
for h in kb_hints:
|
||||
parts.append(f"- title={h.get('title', '')} | doc_id={h.get('doc_id', '')} | snippet={str(h.get('snippet', ''))[:120]}")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _validate_numbers(markdown: str, valid: set[int]) -> list[int]:
|
||||
return [n for n in find_cite_numbers(markdown) if n not in valid]
|
||||
|
||||
|
||||
def _kind_of(meta: dict[str, Any]) -> str:
|
||||
if meta.get("doi"):
|
||||
return "doi"
|
||||
if meta.get("pmid"):
|
||||
return "pmid"
|
||||
return "title"
|
||||
|
||||
|
||||
def _sanitize_citation(meta: dict[str, Any]) -> dict[str, Any]:
|
||||
"""丢弃 LLM 可能杜撰的 doi/pmid/authors/journal/year（知识库线索无这些字段），
|
||||
只保留 title/source_doc_id/snippet，强制走 title kind 降级格式。"""
|
||||
return {
|
||||
"title": (meta.get("title") or "").strip(),
|
||||
"source_doc_id": meta.get("source_doc_id") or "",
|
||||
"snippet": meta.get("snippet") or "",
|
||||
}
|
||||
|
||||
|
||||
async def _persist_new_citations(paper_id: str, section_id: str, used: list[Any], existing_nums: set[int]) -> list[dict[str, Any]]:
|
||||
"""把 citations_used 中的新引用（temp_n 不在既有集合）落库，返回新增摘要。"""
|
||||
new_citations: list[dict[str, Any]] = []
|
||||
seen_new: set[int] = set()
|
||||
for c in used:
|
||||
if not isinstance(c, dict):
|
||||
continue
|
||||
tn = int(c.get("temp_n") or 0)
|
||||
if not tn or tn in existing_nums or tn in seen_new:
|
||||
continue
|
||||
seen_new.add(tn)
|
||||
meta = _sanitize_citation(c)
|
||||
ref_text = render_gbt7714(meta)
|
||||
await db.execute(
|
||||
"INSERT INTO citations (paper_id, section_id, temp_n, kind, ref_text, source_doc_id, snippet, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
paper_id, section_id, tn, _kind_of(meta), ref_text,
|
||||
meta.get("source_doc_id") or "", meta.get("snippet") or "", now_iso(),
|
||||
),
|
||||
)
|
||||
new_citations.append({"temp_n": tn, "ref_text": ref_text})
|
||||
return new_citations
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 写节
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def write_paper_section(paper_id: str, section_id: str, style: str = "", cite_mode: str = "") -> dict[str, Any]:
|
||||
paper = await _get_paper(paper_id)
|
||||
if not paper:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
|
||||
sections = await _get_sections(paper_id)
|
||||
sec = next((s for s in sections if s["section_id"] == section_id), None)
|
||||
if not sec:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "章节不存在"}
|
||||
citations = await _get_citations(paper_id)
|
||||
outline = loads_json(paper.get("outline_json"), {})
|
||||
cit_text = _global_citations_text(citations)
|
||||
existing_nums = {c.get("temp_n") for c in citations}
|
||||
|
||||
user = _build_write_user(paper, outline, sec, cit_text, style, cite_mode)
|
||||
markdown = ""
|
||||
used: list[Any] = []
|
||||
bad: list[int] = []
|
||||
for attempt in range(2):
|
||||
try:
|
||||
res = await llm.complete_json(_WRITE_SYSTEM, user, max_tokens=8000)
|
||||
except llm.LLMError as exc:
|
||||
return {"ok": False, "code": "LLM_ERROR", "message": f"写节 LLM 失败：{exc}"}
|
||||
markdown = res.get("markdown") or ""
|
||||
used = res.get("citations_used") or []
|
||||
used_nums = {int(c.get("temp_n") or 0) for c in used if isinstance(c, dict)}
|
||||
valid = existing_nums | used_nums
|
||||
bad = _validate_numbers(markdown, valid)
|
||||
if not bad:
|
||||
break
|
||||
user += f"\n（上次正文出现的引用编号 {sorted(bad)} 非法；只能使用已有编号或从 {_max_temp_n(citations) + 1} 起的新编号，重试。）"
|
||||
else:
|
||||
return {"ok": False, "code": "CITE_VALIDATION_FAILED", "message": f"引用编号非法：{sorted(bad)}"}
|
||||
|
||||
new_citations = await _persist_new_citations(paper_id, section_id, used, existing_nums)
|
||||
await db.execute(
|
||||
"UPDATE paper_sections SET markdown = ?, word_count = ?, status = 'done', updated_at = ? WHERE paper_id = ? AND section_id = ?",
|
||||
(markdown, len(markdown), now_iso(), paper_id, section_id),
|
||||
)
|
||||
await db.execute("UPDATE papers SET status = 'writing', updated_at = ? WHERE id = ?", (now_iso(), paper_id))
|
||||
return {"ok": True, "section_id": section_id, "status": "done", "markdown": markdown,
|
||||
"citations_used": used, "new_citations": new_citations}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 修订
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def revise_paper_section(paper_id: str, section_id: str, instruction: str) -> dict[str, Any]:
|
||||
paper = await _get_paper(paper_id)
|
||||
if not paper:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
|
||||
sections = await _get_sections(paper_id)
|
||||
sec = next((s for s in sections if s["section_id"] == section_id), None)
|
||||
if not sec:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "章节不存在"}
|
||||
citations = await _get_citations(paper_id)
|
||||
outline = loads_json(paper.get("outline_json"), {})
|
||||
existing_nums = {c.get("temp_n") for c in citations}
|
||||
|
||||
user = _build_write_user(paper, outline, sec, _global_citations_text(citations), "", "")
|
||||
user += "\n\n原章节正文：\n" + (sec.get("markdown") or "")
|
||||
user += f"\n\n修订要求：{instruction}"
|
||||
|
||||
markdown = ""
|
||||
used: list[Any] = []
|
||||
bad: list[int] = []
|
||||
for attempt in range(2):
|
||||
try:
|
||||
res = await llm.complete_json(_REVISE_SYSTEM, user, max_tokens=8000)
|
||||
except llm.LLMError as exc:
|
||||
return {"ok": False, "code": "LLM_ERROR", "message": f"修订 LLM 失败：{exc}"}
|
||||
markdown = res.get("markdown") or ""
|
||||
used = res.get("citations_used") or []
|
||||
used_nums = {int(c.get("temp_n") or 0) for c in used if isinstance(c, dict)}
|
||||
valid = existing_nums | used_nums
|
||||
bad = _validate_numbers(markdown, valid)
|
||||
if not bad:
|
||||
break
|
||||
user += f"\n（引用编号 {sorted(bad)} 非法，重试。）"
|
||||
else:
|
||||
return {"ok": False, "code": "CITE_VALIDATION_FAILED", "message": f"引用编号非法：{sorted(bad)}"}
|
||||
|
||||
new_citations = await _persist_new_citations(paper_id, section_id, used, existing_nums)
|
||||
await db.execute(
|
||||
"UPDATE paper_sections SET markdown = ?, word_count = ?, status = 'done', updated_at = ? WHERE paper_id = ? AND section_id = ?",
|
||||
(markdown, len(markdown), now_iso(), paper_id, section_id),
|
||||
)
|
||||
return {"ok": True, "section_id": section_id, "markdown": markdown,
|
||||
"citations_used": used, "citation_delta": len(new_citations), "new_citations": new_citations}
|
||||
|
||||
|
||||
async def revise_paper(paper_id: str, instruction: str) -> dict[str, Any]:
|
||||
"""LLM 判定受影响章节，返回受影响节清单 + 修订计划（由调用方逐节 revise_paper_section）。"""
|
||||
paper = await _get_paper(paper_id)
|
||||
if not paper:
|
||||
return {"ok": False, "code": "NOT_FOUND", "message": "论文不存在"}
|
||||
sections = await _get_sections(paper_id)
|
||||
outline = loads_json(paper.get("outline_json"), {})
|
||||
|
||||
system = (
|
||||
"你是论文修订计划助手。根据用户修订意见与论文大纲，判定需要修改哪些章节。"
|
||||
"只输出一个合法 JSON 对象：{\"affected_section_ids\": [\"s1\", ...], \"plan\": 修订计划(一句话)}。"
|
||||
)
|
||||
sec_list = "\n".join(f"{s['section_id']}: {s['title']}" for s in sections)
|
||||
user = f"论文标题：{paper.get('title')}\n大纲章节：\n{sec_list}\n\n修订意见：{instruction}"
|
||||
try:
|
||||
res = await llm.complete_json(system, user, max_tokens=2000)
|
||||
except llm.LLMError as exc:
|
||||
return {"ok": False, "code": "LLM_ERROR", "message": f"修订计划 LLM 失败：{exc}"}
|
||||
affected = [s for s in (res.get("affected_section_ids") or []) if any(x["section_id"] == s for x in sections)]
|
||||
return {"ok": True, "affected_section_ids": affected, "plan": res.get("plan", "")}
|
||||
@@ -1,119 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# build.sh —— PAPER 服务编译硬化（复制自 bioflow_service/build.sh，白名单改造）。
|
||||
# 白名单(明文)：app/main.py(uvicorn 入口)、app/config.py(运维可读面)、
|
||||
# app/schemas.py(常量面)、app/db.py(aiosqlite 协程取消热区——Cython 协程断流断言
|
||||
# abort 是既有教训，明文保留)、app/mcp.py(MCP 工具层，mcp 依赖 pydantic + async 工具)、
|
||||
# services/paper/{llm,search,write,outline,complete}.py(长 LLM/httpx/aiosqlite 调用，
|
||||
# 断流崩溃高危点，明文走 CPython 原生协程)+ 全部 __init__.py。
|
||||
# 仅纯同步逻辑(cite.py 等)可编译。
|
||||
# 用法:build.sh <source-root> <output-dir> [python]
|
||||
set -euo pipefail
|
||||
umask 027
|
||||
|
||||
SOURCE_ROOT="${1:?Usage: $0 <source-root> <output-dir> [python]}"
|
||||
OUTPUT_DIR="${2:?Usage: $0 <source-root> <output-dir> [python]}"
|
||||
PYTHON="${3:-python3}"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
VISIBILITY_TOOL="${ELF_VISIBILITY_TOOL:-$SCRIPT_DIR/../tooling/compile/tools/elf_visibility.py}"
|
||||
|
||||
SOURCE_ROOT="$(realpath "$SOURCE_ROOT")"
|
||||
OUTPUT_DIR="$(realpath -m "$OUTPUT_DIR")"
|
||||
|
||||
[[ -d "$SOURCE_ROOT" ]] || { echo "Paper service source missing: $SOURCE_ROOT" >&2; exit 2; }
|
||||
[[ -x "$PYTHON" ]] || { echo "Python is not executable: $PYTHON" >&2; exit 2; }
|
||||
[[ -f "$VISIBILITY_TOOL" ]] || { echo "ELF visibility tool missing: $VISIBILITY_TOOL" >&2; exit 2; }
|
||||
for command_name in gcc strip rsync; do command -v "$command_name" >/dev/null; done
|
||||
|
||||
case "$OUTPUT_DIR" in
|
||||
"/"|"$(realpath -m "$HOME")"|"$SOURCE_ROOT")
|
||||
echo "Unsafe output directory: $OUTPUT_DIR" >&2; exit 2 ;;
|
||||
esac
|
||||
[[ ! -e "$OUTPUT_DIR" ]] || { echo "Output exists; refusing to overwrite: $OUTPUT_DIR" >&2; exit 2; }
|
||||
|
||||
echo "[PAPER BUILD 1/5] Copying source (excluding tests)"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
rsync -a --exclude '__pycache__' --exclude 'test_*.py' --exclude 'data' --exclude '.env' \
|
||||
--exclude 'build.sh' --exclude 'restart.sh' --exclude 'pytest.ini' \
|
||||
"$SOURCE_ROOT/" "$OUTPUT_DIR/"
|
||||
|
||||
WHITELIST_PY=(
|
||||
"app/main.py"
|
||||
"app/config.py"
|
||||
"app/schemas.py"
|
||||
"app/db.py"
|
||||
"app/mcp.py"
|
||||
"app/services/paper/llm.py"
|
||||
"app/services/paper/search.py"
|
||||
"app/services/paper/write.py"
|
||||
"app/services/paper/outline.py"
|
||||
"app/services/paper/complete.py"
|
||||
)
|
||||
_is_whitelisted() {
|
||||
local rel="$1"
|
||||
[[ "$(basename "$rel")" == "__init__.py" ]] && return 0
|
||||
for w in "${WHITELIST_PY[@]}"; do
|
||||
case "$rel" in
|
||||
$w) return 0 ;;
|
||||
esac
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
echo "[PAPER BUILD 2/5] Discovering Python modules"
|
||||
PY_FILES=()
|
||||
while IFS= read -r -d '' path; do PY_FILES+=("$path"); done \
|
||||
< <(find "$OUTPUT_DIR" -type f -name '*.py' -print0)
|
||||
FILTERED=()
|
||||
for py_file in "${PY_FILES[@]}"; do
|
||||
rel="${py_file#"$OUTPUT_DIR"/}"
|
||||
if _is_whitelisted "$rel"; then
|
||||
echo " [whitelist] $rel (明文保留)"
|
||||
continue
|
||||
fi
|
||||
FILTERED+=("$py_file")
|
||||
done
|
||||
PY_FILES=("${FILTERED[@]}")
|
||||
TOTAL="${#PY_FILES[@]}"
|
||||
(( TOTAL > 0 )) || { echo "No paper service modules selected" >&2; exit 3; }
|
||||
|
||||
INCLUDE_PY="$("$PYTHON" -c 'import sysconfig; print(sysconfig.get_config_var("INCLUDEPY") or "")')"
|
||||
EXT_SUFFIX="$("$PYTHON" -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX") or ".so")')"
|
||||
|
||||
echo "[PAPER BUILD 3/5] Cython preflight ($TOTAL modules)"
|
||||
COUNT=0; FAILURES=0
|
||||
for py_file in "${PY_FILES[@]}"; do
|
||||
COUNT=$((COUNT+1))
|
||||
directory="$(dirname "$py_file")"
|
||||
module="$(basename "$py_file" .py)"
|
||||
c_file="$directory/$module.c"
|
||||
echo " [$COUNT/$TOTAL] ${py_file#"$OUTPUT_DIR"/}"
|
||||
if ! "$PYTHON" -m cython -3 \
|
||||
--directive always_allow_keywords=true,annotation_typing=False \
|
||||
--no-docstrings "$py_file" -o "$c_file" 2>/dev/null; then
|
||||
FAILURES=$((FAILURES+1))
|
||||
fi
|
||||
done
|
||||
(( FAILURES == 0 )) || { echo "Paper service Cython failed: $FAILURES" >&2; exit 3; }
|
||||
|
||||
echo "[PAPER BUILD 4/5] GCC hidden visibility + strip"
|
||||
COUNT=0
|
||||
for py_file in "${PY_FILES[@]}"; do
|
||||
COUNT=$((COUNT+1))
|
||||
directory="$(dirname "$py_file")"
|
||||
module="$(basename "$py_file" .py)"
|
||||
c_file="$directory/$module.c"
|
||||
so_file="$directory/$module$EXT_SUFFIX"
|
||||
gcc -shared -pthread -fPIC -O2 -fvisibility=hidden \
|
||||
-I"$INCLUDE_PY" -o "$so_file" "$c_file"
|
||||
strip --strip-unneeded "$so_file"
|
||||
"$PYTHON" -B "$VISIBILITY_TOOL" --hide-non-pyinit --check "$so_file" >/dev/null
|
||||
rm -f -- "$c_file" "$py_file"
|
||||
done
|
||||
|
||||
echo "[PAPER BUILD 5/5] Final checks"
|
||||
SO_COUNT="$(find "$OUTPUT_DIR" -type f -name '*.so' | wc -l)"
|
||||
[[ "$SO_COUNT" -eq "$TOTAL" ]] || { echo "count mismatch: $TOTAL vs $SO_COUNT" >&2; exit 4; }
|
||||
if find "$OUTPUT_DIR" -type f -name '*.c' -print -quit | grep -q .; then
|
||||
echo "Generated C source remains" >&2; exit 4
|
||||
fi
|
||||
echo "Paper service compile+harden complete: modules=$SO_COUNT output=$OUTPUT_DIR"
|
||||
@@ -1,6 +0,0 @@
|
||||
[pytest]
|
||||
python_files = test_*.py
|
||||
markers =
|
||||
paper_unit: 论文服务单元测试（引用协议/大纲/写节/重排/complete）
|
||||
paper_mcp: MCP 协议互操作测试
|
||||
asyncio_mode = auto
|
||||
@@ -1,34 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# restart.sh —— PAPER 服务重启脚本（8097；不触碰生产服务 8090/8091/8092/8093/8094/8098）。
|
||||
# 零硬编码：venv/python/仓库路径全部从脚本位置推导或环境变量覆盖。
|
||||
# 用法: restart.sh [python] [port]
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
PYTHON="${1:-$REPO_ROOT/../plantai-rag-new-venv/bin/python}"
|
||||
PORT="${2:-8097}"
|
||||
HOST="${PAPER_BIND_HOST:-127.0.0.1}"
|
||||
|
||||
[[ -x "$PYTHON" ]] || { echo "Python not found: $PYTHON" >&2; exit 2; }
|
||||
|
||||
APP_MODULE="app.main:app"
|
||||
if command -v lsof >/dev/null 2>&1; then
|
||||
PID="$(lsof -ti tcp:"$PORT" -sTCP:LISTEN 2>/dev/null || true)"
|
||||
if [[ -n "$PID" ]]; then
|
||||
echo "[restart] 停止 $PORT 旧进程 pid=$PID"
|
||||
kill $PID 2>/dev/null || true
|
||||
sleep 2
|
||||
fi
|
||||
fi
|
||||
|
||||
cd "$SCRIPT_DIR"
|
||||
# 加载运行时配置（.env 不入 git）：services/paper/llm.py 直接读 DOCGEN_LLM_* 裸键，
|
||||
# search.py 读 VSTORE_BASE_URL——需 set -a 显式导出（同 bioflow_service/restart.sh 先例）。
|
||||
set -a
|
||||
[ -f .env ] && source .env
|
||||
set +a
|
||||
echo "[restart] 启动论文撰写服务 :$PORT（$PYTHON）"
|
||||
nohup "$PYTHON" -m uvicorn "$APP_MODULE" --host "$HOST" --port "$PORT" \
|
||||
> "$SCRIPT_DIR/paper.log" 2>&1 &
|
||||
echo "[restart] 完成；日志 paper_writer_service/paper.log"
|
||||
@@ -1,27 +0,0 @@
|
||||
# review_service/.env.example —— 运行时配置模板
|
||||
# 用法: cp .env.example .env 后填写（.env 已被仓库 .gitignore 忽略，勿提交）
|
||||
# restart.sh 会 `set -a; source .env` 导出给 uvicorn 进程。
|
||||
|
||||
# ---- LLM（DeepSeek；REVIEW_LLM_* 优先 → DOCGEN_LLM_* 回退 → 内置默认，见 services/llm.py）----
|
||||
# 最简做法：从部署树抄一行即可复用同一把 key：
|
||||
# grep '^DOCGEN_LLM_API_KEY=' /home/xzq/5090/teamWorkspace/zh/PlantAI-rag-new/sdk/.env >> .env
|
||||
REVIEW_LLM_BASE_URL=https://api.deepseek.com/v1
|
||||
REVIEW_LLM_API_KEY=
|
||||
REVIEW_LLM_MODEL=deepseek-chat
|
||||
#REVIEW_LLM_TIMEOUT=300
|
||||
|
||||
# ---- 上游（只读消费）----
|
||||
#REVIEW_VSTORE_BASE_URL=http://127.0.0.1:8094
|
||||
#REVIEW_METHODOLOGY_BASE_URL=http://127.0.0.1:8092 # 预留（MinerU 重链路，当前未用）
|
||||
|
||||
# ---- PDF 解析（经 vstore）----
|
||||
#REVIEW_PARSE_TIMEOUT=180 # ingest→取文→清理 总读超时（秒）
|
||||
#REVIEW_PARSE_DELETE_AFTER=1 # 1=解析后删除 vstore 临时文档（0=保留便于排查）
|
||||
|
||||
# ---- 自审稿 ----
|
||||
#REVIEW_PRESUBMIT_READ_TIMEOUT=360 # LLM 长调用读超时（秒）
|
||||
#REVIEW_MAX_TEXT_CHARS=60000 # 送 LLM 的正文截断上限
|
||||
#REVIEW_MAX_PDF_BYTES=52428800 # 上传 PDF 上限 50MB
|
||||
|
||||
# ---- 服务（restart.sh 读取）----
|
||||
#REVIEW_BIND_HOST=127.0.0.1
|
||||
@@ -1,7 +0,0 @@
|
||||
"""review_service —— 科研投稿与智能审稿助手（8100）。
|
||||
|
||||
独立部署树（模式同 paper_writer_service 8097 / bioflow_service 8098）：
|
||||
- BFF(sdk 8090) 经 routers/review_proxy.py 条件反代 /api/review/*（REVIEW_ENABLED）。
|
||||
- 上游只读消费：vstore 8094（PDF 解析）、DeepSeek（LLM，REVIEW_LLM_*/DOCGEN_LLM_*）。
|
||||
- 约束见根目录 AGENTS.md：审稿相关新代码只允许落在本目录。
|
||||
"""
|
||||
@@ -1 +0,0 @@
|
||||
"""review 服务（8100）应用包。"""
|
||||
@@ -1,112 +0,0 @@
|
||||
"""config.py —— review 服务（8100）环境变量驱动配置（pydantic-settings）。
|
||||
|
||||
全部键可用环境变量覆盖（前缀 REVIEW_）。LLM 沿用 docgen 的裸键模式：
|
||||
优先 REVIEW_LLM_*，回退 DOCGEN_LLM_*（与 paper_writer_service 共享同一套
|
||||
DeepSeek 凭据，由 services/llm.py 直接读 os.environ）。零硬编码宿主路径。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(
|
||||
env_file=".env",
|
||||
env_file_encoding="utf-8",
|
||||
extra="ignore",
|
||||
env_prefix="REVIEW_",
|
||||
)
|
||||
|
||||
app_name: str = "plantai-review-service"
|
||||
version: str = "0.1.0"
|
||||
|
||||
# ---- 上游（只读消费；不修改 vstore/methodology 任何代码）----
|
||||
# vstore 8094：POST /api/vstore/ingest + GET/DELETE /api/vstore/documents/{id}
|
||||
# （pymupdf 段落解析，同步秒级；methodology 8092 的 MinerU 链路是
|
||||
# 分钟级异步 job，不适配同步自审稿，仅预留地址供后续扩展）
|
||||
vstore_base_url: str = "http://127.0.0.1:8094"
|
||||
methodology_base_url: str = "http://127.0.0.1:8092"
|
||||
|
||||
# ---- PDF 解析 ----
|
||||
parse_timeout: float = 180.0 # vstore ingest→fetch→delete 读超时（秒）
|
||||
parse_delete_after: bool = True # 解析完成后删除 vstore 临时文档（不污染私有库）
|
||||
|
||||
# ---- 自审稿（presubmit）----
|
||||
presubmit_read_timeout: float = 360.0 # LLM 长调用读超时（秒）
|
||||
max_text_chars: int = 150000 # 送 LLM 的正文截断上限（REVIEW_MAX_TEXT_CHARS 覆盖；
|
||||
# token 超 48K 自动降 100K，LLM 400/413 再降 80K 重试）
|
||||
max_pdf_bytes: int = 50 * 1024 * 1024 # 上传 PDF 上限 50MB（对齐 extraction FR-EXTRACT-004）
|
||||
|
||||
# ---- 期刊情报与投稿推荐（recommend）----
|
||||
db_path: str = "review.db" # 独立 SQLite（相对 review_service/；REVIEW_DB_PATH 覆盖）
|
||||
journals_seed_path: str = "data/journals/journals.json" # 种子（相对 review_service/）
|
||||
journal_directory_seed_path: str = "data/journals/source_v4.xlsx" # 目录源 xlsx（相对 review_service/）
|
||||
seed_on_startup: bool = True # 启动自动 seed journal_directory（空表才导；false 关闭）
|
||||
vstore_match_timeout: float = 30.0 # scope 入库/向量检索读超时（秒）
|
||||
recommend_read_timeout: float = 60.0 # LLM 重排读超时（秒）
|
||||
recommend_query_chars: int = 3000 # 送 vstore 的检索 query 截断（防嵌入模型超长）
|
||||
recommend_scope_top_k: int = 10 # search-chunks top_k（vstore 上限 20）
|
||||
|
||||
# ---- 论文重排版（reformat，P1：.tex 源码级变换，不编译）----
|
||||
max_tex_bytes: int = 2 * 1024 * 1024 # 上传 .tex 上限 2MB（PDF 的 50MB 不适用）
|
||||
reformat_dir: str = "data/reformat_jobs" # 产物落盘（相对 review_service/）
|
||||
|
||||
# ---- LaTeX 编译 PDF（compile，P3：tectonic 单文件二进制）----
|
||||
tectonic_path: str = "tools/tectonic" # 相对 review_service/；也可绝对路径
|
||||
tectonic_cache_dir: str = "data/tectonic-cache" # 宏包缓存（部署时预热，运行期只读命中）
|
||||
compile_timeout_s: float = 60.0 # 单次编译墙钟上限，超时整进程组 kill
|
||||
max_compile_pdf_bytes: int = 50 * 1024 * 1024 # 产物 PDF 上限 50MB
|
||||
compile_queue_limit: int = 5 # 等待槽位上限（>5 → QUEUE_FULL 429）
|
||||
compile_retention_days: int = 7 # 编译产物保留天数（后台清扫）
|
||||
|
||||
@property
|
||||
def _service_root(self) -> "Path":
|
||||
# review_service/app/config.py → review_service/（相对本文件定位，不依赖 cwd）
|
||||
return Path(__file__).resolve().parent.parent
|
||||
|
||||
@property
|
||||
def db_path_abs(self) -> "Path":
|
||||
# 绝对 db_path 会被 Path 除法原样返回；相对路径落在 review_service/ 下
|
||||
return self._service_root / self.db_path
|
||||
|
||||
@property
|
||||
def journals_seed_abs(self) -> "Path":
|
||||
return self._service_root / self.journals_seed_path
|
||||
|
||||
@property
|
||||
def journal_directory_seed_abs(self) -> "Path":
|
||||
return self._service_root / self.journal_directory_seed_path
|
||||
|
||||
@property
|
||||
def reformat_dir_abs(self) -> "Path":
|
||||
p = Path(self.reformat_dir)
|
||||
return p if p.is_absolute() else self._service_root / p
|
||||
|
||||
@property
|
||||
def tectonic_abs(self) -> "Path":
|
||||
p = Path(self.tectonic_path)
|
||||
return p if p.is_absolute() else self._service_root / p
|
||||
|
||||
@property
|
||||
def tectonic_cache_abs(self) -> "Path":
|
||||
p = Path(self.tectonic_cache_dir)
|
||||
return p if p.is_absolute() else self._service_root / p
|
||||
|
||||
@property
|
||||
def vstore_base(self) -> str:
|
||||
return self.vstore_base_url.rstrip("/")
|
||||
|
||||
@property
|
||||
def methodology_base(self) -> str:
|
||||
return self.methodology_base_url.rstrip("/")
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_settings() -> Settings:
|
||||
return Settings()
|
||||
|
||||
|
||||
settings = get_settings()
|
||||
@@ -1,191 +0,0 @@
|
||||
"""app/db.py —— review 服务独立 SQLite（aiosqlite + WAL，无 ORM）：journals 表。
|
||||
|
||||
- 单连接 + asyncio 写锁（模式对齐 paper_writer_service/bioflow，但表结构与种子
|
||||
逻辑为本模块自研）；路径默认 review_service/review.db（REVIEW_DB_PATH 覆盖）。
|
||||
- 种子：data/journals/journals.json → journals 表（幂等 upsert，vstore_doc_id 列
|
||||
永不被种子覆盖——该列记录 scope_desc 在 vstore 8094 的 document_id）。
|
||||
- 只在 review_service/ 内使用，不触碰 BFF/生产服务数据库。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
import aiosqlite
|
||||
|
||||
from .config import get_settings
|
||||
|
||||
_SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS journals (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
scope_desc TEXT NOT NULL,
|
||||
impact_factor REAL,
|
||||
review_time_weeks REAL,
|
||||
acceptance_rate REAL,
|
||||
submission_deadline TEXT,
|
||||
official_url TEXT,
|
||||
last_updated TEXT NOT NULL,
|
||||
vstore_doc_id TEXT
|
||||
);
|
||||
"""
|
||||
|
||||
# 种子 upsert 列（刻意不含 vstore_doc_id：种子刷新不得清掉向量库映射）
|
||||
_SEED_COLS = (
|
||||
"id",
|
||||
"name",
|
||||
"scope_desc",
|
||||
"impact_factor",
|
||||
"review_time_weeks",
|
||||
"acceptance_rate",
|
||||
"submission_deadline",
|
||||
"official_url",
|
||||
"last_updated",
|
||||
)
|
||||
_UPSERT_SQL = (
|
||||
"INSERT INTO journals (" + ", ".join(_SEED_COLS) + ") VALUES ("
|
||||
+ ", ".join("?" for _ in _SEED_COLS) + ") "
|
||||
+ "ON CONFLICT(id) DO UPDATE SET "
|
||||
+ ", ".join(f"{c}=excluded.{c}" for c in _SEED_COLS if c != "id")
|
||||
)
|
||||
|
||||
|
||||
class SeedError(Exception):
|
||||
"""种子文件缺失/损坏（journals.json 不可读）。"""
|
||||
|
||||
|
||||
class Database:
|
||||
"""aiosqlite 单连接封装（WAL；写串行化）。"""
|
||||
|
||||
def __init__(self, path: str | Path | None = None) -> None:
|
||||
self._path: Path | None = Path(path) if path is not None else None
|
||||
self._conn: aiosqlite.Connection | None = None
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
@property
|
||||
def path(self) -> Path:
|
||||
if self._path is None:
|
||||
self._path = get_settings().db_path_abs
|
||||
return self._path
|
||||
|
||||
def set_path(self, path: str | Path) -> None:
|
||||
"""测试注入：在进入 TestClient lifespan 之前预置库路径（须尚未 connect）。"""
|
||||
if self._conn is not None:
|
||||
raise RuntimeError("Database 已连接，无法再改路径（先 close_db()）")
|
||||
self._path = Path(path)
|
||||
|
||||
@property
|
||||
def connected(self) -> bool:
|
||||
return self._conn is not None
|
||||
|
||||
async def connect(self) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._conn = await aiosqlite.connect(self.path)
|
||||
self._conn.row_factory = aiosqlite.Row
|
||||
await self._conn.execute("PRAGMA journal_mode=WAL")
|
||||
await self._conn.execute("PRAGMA busy_timeout=5000")
|
||||
await self._conn.executescript(_SCHEMA)
|
||||
await self._conn.commit()
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._conn is not None:
|
||||
await self._conn.close()
|
||||
self._conn = None
|
||||
|
||||
@property
|
||||
def conn(self) -> aiosqlite.Connection:
|
||||
if self._conn is None:
|
||||
raise RuntimeError("Database 未初始化：先 await init_db()")
|
||||
return self._conn
|
||||
|
||||
async def fetchall(self, sql: str, params: Iterable[Any] = ()) -> list[dict[str, Any]]:
|
||||
async with self.conn.execute(sql, tuple(params)) as cur:
|
||||
rows = await cur.fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
async def fetchone(self, sql: str, params: Iterable[Any] = ()) -> dict[str, Any] | None:
|
||||
async with self.conn.execute(sql, tuple(params)) as cur:
|
||||
row = await cur.fetchone()
|
||||
return dict(row) if row is not None else None
|
||||
|
||||
async def execute(self, sql: str, params: Iterable[Any] = ()) -> int:
|
||||
async with self._lock:
|
||||
cur = await self.conn.execute(sql, tuple(params))
|
||||
await self._conn.commit() # type: ignore[union-attr]
|
||||
return cur.lastrowid or 0
|
||||
|
||||
async def executemany(self, sql: str, seq: list[tuple[Any, ...]]) -> None:
|
||||
async with self._lock:
|
||||
await self.conn.executemany(sql, seq)
|
||||
await self._conn.commit() # type: ignore[union-attr]
|
||||
|
||||
|
||||
db = Database()
|
||||
|
||||
|
||||
async def init_db(path: str | Path | None = None) -> None:
|
||||
"""幂等初始化：已连接则跳过（测试可在 lifespan 前 set_path 注入路径）。"""
|
||||
if path is not None:
|
||||
db.set_path(path)
|
||||
if db.connected:
|
||||
return
|
||||
await db.connect()
|
||||
|
||||
|
||||
async def close_db() -> None:
|
||||
await db.close()
|
||||
|
||||
|
||||
def set_path(path: str | Path) -> None:
|
||||
"""同步注入库路径（测试 fixture 用；须在 init_db 之前调用）。"""
|
||||
db.set_path(path)
|
||||
|
||||
|
||||
async def ensure_journals_seeded() -> int:
|
||||
"""装载种子期刊（幂等）。返回 journals 表行数。"""
|
||||
row = await db.fetchone("SELECT COUNT(*) AS c FROM journals")
|
||||
if row and int(row["c"]) > 0:
|
||||
return int(row["c"])
|
||||
|
||||
seed_file = get_settings().journals_seed_abs
|
||||
try:
|
||||
payload = json.loads(Path(seed_file).read_text(encoding="utf-8"))
|
||||
except FileNotFoundError as exc:
|
||||
raise SeedError(f"期刊种子文件不存在：{seed_file}") from exc
|
||||
except (OSError, ValueError) as exc:
|
||||
raise SeedError(f"期刊种子文件不可读：{exc}") from exc
|
||||
|
||||
journals = payload.get("journals") if isinstance(payload, dict) else None
|
||||
if not isinstance(journals, list) or not journals:
|
||||
raise SeedError(f"期刊种子文件缺少 journals 数组：{seed_file}")
|
||||
|
||||
rows: list[tuple[Any, ...]] = []
|
||||
for item in journals:
|
||||
if not isinstance(item, dict) or not item.get("id") or not item.get("name"):
|
||||
continue
|
||||
rows.append(tuple(item.get(c) for c in _SEED_COLS))
|
||||
if not rows:
|
||||
raise SeedError(f"期刊种子文件无有效条目：{seed_file}")
|
||||
|
||||
await db.executemany(_UPSERT_SQL, rows)
|
||||
return len(rows)
|
||||
|
||||
|
||||
async def list_journals() -> list[dict[str, Any]]:
|
||||
"""全部期刊（impact_factor 降序，NULL 靠后）。"""
|
||||
return await db.fetchall(
|
||||
"SELECT * FROM journals ORDER BY impact_factor DESC, name ASC"
|
||||
)
|
||||
|
||||
|
||||
async def get_journal(journal_id: str) -> dict[str, Any] | None:
|
||||
return await db.fetchone("SELECT * FROM journals WHERE id = ?", (journal_id,))
|
||||
|
||||
|
||||
async def set_vstore_doc_id(journal_id: str, doc_id: str) -> None:
|
||||
"""记录 scope_desc 在 vstore 的 document_id（向量白名单检索用）。"""
|
||||
await db.execute(
|
||||
"UPDATE journals SET vstore_doc_id = ? WHERE id = ?", (doc_id, journal_id)
|
||||
)
|
||||
@@ -1,124 +0,0 @@
|
||||
"""main.py —— review 服务应用入口（uvicorn 入口）。
|
||||
|
||||
- 独立部署树 review_service/（科研投稿与智能审稿助手，端口 8100）。
|
||||
- 端点：/health（GET，BFF 探活反代）+ /api/review/*（自审稿等业务路由）。
|
||||
- 不依赖 BFF/生产服务（8090 等绝不探测/连接）；上游仅消费 vstore 8094。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
import httpx
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from .config import get_settings
|
||||
from .db import close_db, db, init_db
|
||||
from .routers import (
|
||||
compile,
|
||||
conferences,
|
||||
format_check,
|
||||
journal_directory,
|
||||
journals,
|
||||
presubmit,
|
||||
reformat,
|
||||
rebuttal,
|
||||
)
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
settings = get_settings()
|
||||
app = FastAPI(title=settings.app_name, version=settings.version)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=False,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
@asynccontextmanager
|
||||
async def _lifespan(_app: FastAPI):
|
||||
# journals 推荐依赖独立 SQLite：启动即建表（幂等；测试可先 set_path 注入
|
||||
# tmp 路径，本函数沿用注入路径）
|
||||
await init_db()
|
||||
# rebuttal Phase 1：独立表，启动幂等建表（自含连接，不走 db.py）
|
||||
try:
|
||||
from .services.rebuttal_store import ensure_table
|
||||
ensure_table()
|
||||
except Exception: # noqa: BLE001
|
||||
logger.warning("rebuttal 建表失败（忽略）", exc_info=True)
|
||||
# 编译产物过期清扫：启动跑一次（失败不阻塞启动）
|
||||
try:
|
||||
from .services.latex_compile import sweep_expired
|
||||
sweep_expired()
|
||||
except Exception: # noqa: BLE001
|
||||
logger.warning("compile sweep 启动清扫失败（忽略）", exc_info=True)
|
||||
# journal_directory 自动 seed：空表才导（xlsx 解析约 1-3s，不堵 event loop；
|
||||
# 路径沿用 db.path 注入链——生产=settings.db_path_abs，测试=tmp）
|
||||
if settings.seed_on_startup:
|
||||
try:
|
||||
from .services.journal_seed import ensure_journal_directory_seeded
|
||||
_t0 = time.time()
|
||||
_result = await asyncio.to_thread(
|
||||
ensure_journal_directory_seeded, db.path)
|
||||
if _result.get("seeded"):
|
||||
_c = _result.get("counts") or {}
|
||||
logger.info(
|
||||
"seeded %s rows (en=%s, zh=%s, %.1fs)",
|
||||
_result["rows"], _c.get("en"), _c.get("zh"),
|
||||
time.time() - _t0)
|
||||
elif _result.get("error"):
|
||||
logger.warning(
|
||||
"journal_directory 启动 seed 失败（忽略，不阻塞启动）: %s",
|
||||
_result["error"])
|
||||
else:
|
||||
logger.info("%s", _result.get("reason") or "journal_directory seed skipped")
|
||||
except Exception: # noqa: BLE001
|
||||
logger.warning("journal_directory 启动 seed 失败（忽略，不阻塞启动）", exc_info=True)
|
||||
else:
|
||||
logger.info("journal_directory seed skipped (REVIEW_SEED_ON_STARTUP=false)")
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await close_db()
|
||||
|
||||
app.router.lifespan_context = _lifespan
|
||||
|
||||
@app.get("/health")
|
||||
async def health() -> dict[str, object]:
|
||||
vstore_ok = False
|
||||
vstore_error: str | None = None
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=2.0) as client:
|
||||
resp = await client.get(f"{settings.vstore_base}/api/vstore/health")
|
||||
resp.raise_for_status()
|
||||
vstore_ok = bool(resp.json().get("ok"))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
vstore_error = str(exc)
|
||||
return {
|
||||
"ok": True,
|
||||
"status": "ok" if vstore_ok else "degraded",
|
||||
"upstreams": {"vstore": {"connected": vstore_ok, "error": vstore_error}},
|
||||
"version": settings.version,
|
||||
}
|
||||
|
||||
app.include_router(presubmit.router)
|
||||
app.include_router(rebuttal.router)
|
||||
app.include_router(journals.router)
|
||||
app.include_router(format_check.router)
|
||||
app.include_router(conferences.router)
|
||||
app.include_router(journal_directory.router)
|
||||
app.include_router(reformat.router)
|
||||
app.include_router(compile.router)
|
||||
return app
|
||||
|
||||
|
||||
app = create_app()
|
||||
@@ -1 +0,0 @@
|
||||
"""review 服务路由包。"""
|
||||
@@ -1,184 +0,0 @@
|
||||
"""routers/compile.py —— LaTeX 编译 PDF（P3a：tectonic）。
|
||||
|
||||
端点：
|
||||
POST /api/review/compile multipart：file(.tex 可选) | job_id(可选，二选一)
|
||||
→ {ok, compile_id, pdf_url, log_url, meta}
|
||||
GET /api/review/compile/{cid} 编译元信息（meta.json）
|
||||
GET /api/review/compile/{cid}/pdf output.pdf（FileResponse）
|
||||
GET /api/review/compile/{cid}/log output.log（text/plain）
|
||||
|
||||
信封：业务错误 = HTTP 200 + {ok:false,error:{code,message,detail?,degraded:false}}
|
||||
（QUEUE_FULL 例外：HTTP 429 + 同款信封体，前端仍按 ok/error 单一口径处理）。
|
||||
detail = log 尾 200 行（COMPILE_FAILED / TIMEOUT）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, File, Form, UploadFile
|
||||
from fastapi.responses import FileResponse, JSONResponse, PlainTextResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..config import get_settings
|
||||
from ..services import latex_compile as lc
|
||||
from ..services.latex_compile import (
|
||||
CompileError,
|
||||
QueueFullError,
|
||||
ToolUnavailableError,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-compile"])
|
||||
|
||||
_CID_RE = re.compile(r"^[0-9a-f]{12}$")
|
||||
|
||||
|
||||
class CompileResponse(BaseModel):
|
||||
ok: bool
|
||||
compile_id: str | None = None
|
||||
pdf_url: str | None = None
|
||||
log_url: str | None = None
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str, detail: str = "") -> CompileResponse:
|
||||
err: dict[str, Any] = {"code": code, "message": message, "degraded": False}
|
||||
if detail:
|
||||
err["detail"] = detail
|
||||
return CompileResponse(ok=False, error=err)
|
||||
|
||||
|
||||
def _jobs_root() -> Path:
|
||||
d = get_settings().reformat_dir_abs
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
@router.post("/compile", response_model=CompileResponse)
|
||||
async def compile_tex(
|
||||
file: UploadFile | None = File(None, description="LaTeX 源码（.tex，与 job_id 二选一）"),
|
||||
job_id: str = Form("", description="P1 重排版任务 id（优先于 file）"),
|
||||
) -> CompileResponse:
|
||||
settings = get_settings()
|
||||
|
||||
# ---- 输入解析：job_id 优先，否则直接上传的 .tex ----
|
||||
jid = (job_id or "").strip()
|
||||
if jid:
|
||||
if not _CID_RE.fullmatch(jid):
|
||||
return _error("VALIDATION", "job_id 格式非法（应为 12 位 hex）")
|
||||
src = _jobs_root() / f"{jid}.tex"
|
||||
if not src.is_file():
|
||||
return _error("VALIDATION", "job_id 对应的重排版结果不存在，请重新排版")
|
||||
try:
|
||||
source = src.read_text("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return _error("VALIDATION", "任务源码不是合法 UTF-8")
|
||||
anchor_id = jid
|
||||
else:
|
||||
if file is None:
|
||||
return _error("VALIDATION", "必须提供 file（.tex）或 job_id")
|
||||
filename = file.filename or "paper.tex"
|
||||
if Path(filename).name != filename or ".." in filename:
|
||||
return _error("VALIDATION", "非法文件名")
|
||||
if not filename.lower().endswith(".tex"):
|
||||
return _error("VALIDATION", "仅支持 .tex 文件")
|
||||
content = await file.read()
|
||||
if not content:
|
||||
return _error("VALIDATION", "上传文件为空")
|
||||
if len(content) > settings.max_tex_bytes:
|
||||
return _error("VALIDATION", "源码超过 2MB 上限")
|
||||
try:
|
||||
source = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return _error("VALIDATION", "源码必须为 UTF-8 编码")
|
||||
anchor_id = uuid.uuid4().hex[:12]
|
||||
|
||||
# ---- 编译（隔离目录；失败信封带 log 尾）----
|
||||
lc.prepare_input(anchor_id, source)
|
||||
try:
|
||||
result = await lc.compile_tex(anchor_id)
|
||||
except ToolUnavailableError as exc:
|
||||
return _error("TOOL_UNAVAILABLE", str(exc))
|
||||
except QueueFullError as exc:
|
||||
return JSONResponse(
|
||||
status_code=429,
|
||||
content={
|
||||
"ok": False,
|
||||
"compile_id": None, "pdf_url": None, "log_url": None, "meta": None,
|
||||
"error": {"code": "QUEUE_FULL", "message": str(exc), "degraded": False},
|
||||
},
|
||||
)
|
||||
except CompileError as exc:
|
||||
return _error(exc.code, str(exc), detail=exc.log_tail)
|
||||
|
||||
meta_path = lc.compile_dir(anchor_id) / "meta.json"
|
||||
try:
|
||||
meta = json.loads(meta_path.read_text("utf-8"))
|
||||
except (OSError, ValueError):
|
||||
meta = {"compile_id": anchor_id, "seconds": result["seconds"]}
|
||||
return CompileResponse(
|
||||
ok=True,
|
||||
compile_id=anchor_id,
|
||||
pdf_url=f"/api/review/compile/{anchor_id}/pdf",
|
||||
log_url=f"/api/review/compile/{anchor_id}/log",
|
||||
meta=meta,
|
||||
)
|
||||
|
||||
|
||||
def _cid_dir(cid: str) -> Path | None:
|
||||
if not _CID_RE.fullmatch(cid or ""):
|
||||
return None
|
||||
d = lc.compile_dir(cid)
|
||||
return d if d.is_dir() else None
|
||||
|
||||
|
||||
@router.get("/compile/{compile_id}", response_model=CompileResponse)
|
||||
async def get_compile(compile_id: str) -> CompileResponse:
|
||||
cdir = _cid_dir(compile_id)
|
||||
meta_path = cdir / "meta.json" if cdir else None
|
||||
if meta_path is None or not meta_path.is_file():
|
||||
return _error("NOT_FOUND", "编译任务不存在或已过期，请重新编译")
|
||||
try:
|
||||
meta = json.loads(meta_path.read_text("utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return _error("NOT_FOUND", "编译产物损坏，请重新编译")
|
||||
return CompileResponse(
|
||||
ok=True,
|
||||
compile_id=compile_id,
|
||||
pdf_url=f"/api/review/compile/{compile_id}/pdf",
|
||||
log_url=f"/api/review/compile/{compile_id}/log",
|
||||
meta=meta,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/compile/{compile_id}/pdf", response_model=None)
|
||||
async def download_compile_pdf(compile_id: str) -> FileResponse | JSONResponse:
|
||||
cdir = _cid_dir(compile_id)
|
||||
pdf = cdir / "output.pdf" if cdir else None
|
||||
if pdf is None or not pdf.is_file():
|
||||
return JSONResponse(
|
||||
status_code=200,
|
||||
content={"ok": False, "error": {
|
||||
"code": "NOT_FOUND", "message": "PDF 不存在或已过期", "degraded": False}},
|
||||
)
|
||||
return FileResponse(pdf, media_type="application/pdf", filename=f"{compile_id}.pdf")
|
||||
|
||||
|
||||
@router.get("/compile/{compile_id}/log", response_model=None)
|
||||
async def download_compile_log(compile_id: str) -> PlainTextResponse | JSONResponse:
|
||||
cdir = _cid_dir(compile_id)
|
||||
log = cdir / "output.log" if cdir else None
|
||||
if log is None or not log.is_file():
|
||||
return JSONResponse(
|
||||
status_code=200,
|
||||
content={"ok": False, "error": {
|
||||
"code": "NOT_FOUND", "message": "日志不存在或已过期", "degraded": False}},
|
||||
)
|
||||
return PlainTextResponse(log.read_text("utf-8", errors="replace"))
|
||||
@@ -1,209 +0,0 @@
|
||||
"""routers/conferences.py —— 会议投稿时间线（只读静态种子）。
|
||||
|
||||
- GET /api/review/conferences?domain=&ccf=&upcoming_days=180
|
||||
data/journals/conferences.json → 过滤/排序 → 列表 + meta。
|
||||
- 数据诚实性铁律：每条会议必须带 deadline_source ∈ {placeholder, official}、
|
||||
last_verified 与 deadline_confidence ∈ {high, medium, low}，缺失/非法条目加载时
|
||||
跳过并记日志（宁缺毋造）；本端点无任何生成日期的逻辑，日期只来自种子常量。
|
||||
v2 共 27/27 全 placeholder，meta.data_source_note 显式声明占位与置信度，
|
||||
由前端渲染「⚠ 日期待核验」及置信度提示。
|
||||
- 过滤语义：domain 对 domain_tags 精确匹配（大小写不敏感）；ccf ∈ {A,B,C}；
|
||||
upcoming_days=0 不过滤，>0 仅保留 deadline_full_paper ∈ [UTC now, now+N 天]，
|
||||
缺失/不可解析截稿日期在过滤开启时排除。排序：deadline_full_paper 升序，
|
||||
缺失/不可解析恒排最后。
|
||||
- 只读无写接口；文件不可读 → ok=false + DATA_UNAVAILABLE 信封；参数非法 →
|
||||
VALIDATION 信封（均 HTTP 200，对齐本服务"错误信封永远 200"约定）。
|
||||
- 静态文件首次请求加载进内存（lru_cache），进程内 O(1)；不触碰 SQLite/BFF。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-conferences"])
|
||||
|
||||
# review_service/app/routers/ → parents[2] = review_service（对齐 format_check 写法）
|
||||
_CONFERENCES_PATH = (
|
||||
Path(__file__).resolve().parents[2] / "data" / "journals" / "conferences.json"
|
||||
)
|
||||
|
||||
_CCF_OK = frozenset({"A", "B", "C"})
|
||||
_REQUIRED = ("id", "name", "deadline_source", "last_verified", "deadline_confidence")
|
||||
_CONFIDENCE_OK = frozenset({"high", "medium", "low"})
|
||||
_SOURCE_NOTE = (
|
||||
"会议截稿/通知日期首版均为占位数据（deadline_source=placeholder），"
|
||||
"接入官方渠道前须人工核验；deadline_confidence 标注日期置信度"
|
||||
"（high=官方已公布/历年极稳定，medium=按历年规律推算，low=待核验）；"
|
||||
"所有时间按 AoE (UTC-12) 时区。"
|
||||
)
|
||||
|
||||
|
||||
# ---- 请求/响应模型 ----
|
||||
|
||||
class ConferenceItem(BaseModel):
|
||||
"""会议公开字段（种子全字段白名单；未知键丢弃）。"""
|
||||
id: str
|
||||
name: str
|
||||
full_name: str = ""
|
||||
domain_tags: list[str] = []
|
||||
application_tags: list[str] = []
|
||||
ccf_rank: str | None = None
|
||||
deadline_abstract: str | None = None
|
||||
deadline_full_paper: str | None = None
|
||||
notification_date: str | None = None
|
||||
conference_date: str | None = None
|
||||
location: str | None = None
|
||||
official_url: str | None = None
|
||||
timezone_note: str | None = None
|
||||
deadline_source: str
|
||||
last_verified: str
|
||||
deadline_confidence: str = ""
|
||||
|
||||
|
||||
class ConferencesResponse(BaseModel):
|
||||
ok: bool
|
||||
items: list[ConferenceItem] = []
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str) -> ConferencesResponse:
|
||||
return ConferencesResponse(
|
||||
ok=False, items=[], meta=None,
|
||||
error={"code": code, "message": message, "degraded": False},
|
||||
)
|
||||
|
||||
|
||||
# ---- 种子加载（首次调用读盘，进程内缓存）----
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _load_conferences() -> list[dict[str, Any]] | None:
|
||||
"""读 conferences.json → 合法会议列表；文件不可读 → None（DATA_UNAVAILABLE）。"""
|
||||
try:
|
||||
payload = json.loads(_CONFERENCES_PATH.read_text("utf-8"))
|
||||
except (OSError, ValueError) as exc:
|
||||
logger.warning("conferences: 种子文件不可读 %s: %s", _CONFERENCES_PATH, exc)
|
||||
return None
|
||||
raw = payload.get("conferences") if isinstance(payload, dict) else None
|
||||
if not isinstance(raw, list):
|
||||
logger.warning("conferences: 种子缺少 conferences 数组：%s", _CONFERENCES_PATH)
|
||||
return None
|
||||
items: list[dict[str, Any]] = []
|
||||
for conf in raw:
|
||||
if not isinstance(conf, dict):
|
||||
continue
|
||||
missing = [k for k in _REQUIRED if not conf.get(k)]
|
||||
if missing:
|
||||
logger.warning("conferences: 条目缺字段 %s（id=%s），跳过", missing, conf.get("id"))
|
||||
continue
|
||||
if conf.get("deadline_source") not in ("placeholder", "official"):
|
||||
logger.warning(
|
||||
"conferences: deadline_source 非法（id=%s），跳过", conf.get("id")
|
||||
)
|
||||
continue
|
||||
if str(conf.get("deadline_confidence") or "").strip().lower() not in _CONFIDENCE_OK:
|
||||
logger.warning(
|
||||
"conferences: deadline_confidence 非法（id=%s），跳过", conf.get("id")
|
||||
)
|
||||
continue
|
||||
items.append(conf)
|
||||
return items
|
||||
|
||||
|
||||
def _parse_deadline(value: Any) -> datetime | None:
|
||||
"""截稿时间 → aware datetime（naive 视为 UTC）；非法 → None。"""
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
return None
|
||||
try:
|
||||
dt = datetime.fromisoformat(value.strip())
|
||||
except ValueError:
|
||||
return None
|
||||
return dt.replace(tzinfo=timezone.utc) if dt.tzinfo is None else dt
|
||||
|
||||
|
||||
# ---- GET /api/review/conferences ----
|
||||
|
||||
@router.get("/conferences", response_model=ConferencesResponse)
|
||||
async def conferences_list(
|
||||
domain: str = "",
|
||||
ccf: str = "",
|
||||
upcoming_days: int = 180,
|
||||
) -> ConferencesResponse:
|
||||
"""只读：会议投稿时间线（无写/删/改接口）。
|
||||
|
||||
- ?domain= 对 domain_tags 精确匹配（大小写不敏感），空=不过滤；
|
||||
- ?ccf= ccf_rank ∈ A/B/C，空=不过滤，非法值 → VALIDATION 信封；
|
||||
- ?upcoming_days= int≥0，默认 180；0=不过滤，>0 仅保留截稿在
|
||||
[UTC now, now+N 天] 内的会议，负数 → VALIDATION 信封；
|
||||
- 排序按 deadline_full_paper 升序，缺失/不可解析恒排最后；
|
||||
- 空结果 → items=[] + meta.count=0（HTTP 200，不报错）。
|
||||
"""
|
||||
ccf_norm = (ccf or "").strip().upper()
|
||||
if ccf_norm and ccf_norm not in _CCF_OK:
|
||||
return _error("VALIDATION", f"不支持的 ccf={ccf!r}，可选：A / B / C")
|
||||
if upcoming_days < 0:
|
||||
return _error(
|
||||
"VALIDATION", f"upcoming_days 必须 ≥ 0，收到 {upcoming_days}"
|
||||
)
|
||||
|
||||
all_items = _load_conferences()
|
||||
if all_items is None:
|
||||
return _error("DATA_UNAVAILABLE", "会议种子数据不可读，请联系管理员")
|
||||
|
||||
items = list(all_items)
|
||||
|
||||
dom = (domain or "").strip().casefold()
|
||||
if dom:
|
||||
items = [
|
||||
c for c in items
|
||||
if any(
|
||||
str(t).strip().casefold() == dom
|
||||
for t in (c.get("domain_tags") or [])
|
||||
)
|
||||
]
|
||||
if ccf_norm:
|
||||
items = [
|
||||
c for c in items
|
||||
if str(c.get("ccf_rank") or "").strip().upper() == ccf_norm
|
||||
]
|
||||
if upcoming_days > 0:
|
||||
now = datetime.now(timezone.utc)
|
||||
limit = now + timedelta(days=upcoming_days)
|
||||
items = [
|
||||
c for c in items
|
||||
if (dl := _parse_deadline(c.get("deadline_full_paper"))) is not None
|
||||
and now <= dl <= limit
|
||||
]
|
||||
|
||||
def _sort_key(c: dict[str, Any]) -> tuple[bool, datetime]:
|
||||
dl = _parse_deadline(c.get("deadline_full_paper"))
|
||||
if dl is None:
|
||||
return (True, datetime.max.replace(tzinfo=timezone.utc))
|
||||
return (False, dl)
|
||||
|
||||
items.sort(key=_sort_key)
|
||||
|
||||
out = [ConferenceItem(**c) for c in items]
|
||||
verified = max(
|
||||
(str(c.get("last_verified") or "") for c in all_items), default=""
|
||||
)
|
||||
meta: dict[str, Any] = {
|
||||
"count": len(out),
|
||||
"last_verified": verified or None,
|
||||
"data_source_note": _SOURCE_NOTE,
|
||||
"query": {
|
||||
"domain": (domain or "").strip(),
|
||||
"ccf": ccf_norm,
|
||||
"upcoming_days": upcoming_days,
|
||||
},
|
||||
}
|
||||
return ConferencesResponse(ok=True, items=out, meta=meta, error=None)
|
||||
@@ -1,81 +0,0 @@
|
||||
"""routers/format_check.py —— 版式与规范检查（POST /api/review/format-check）。
|
||||
|
||||
纯规则引擎（services/format_check.py）：PDF/Word 本地解档取版式元数据 +
|
||||
journals.json 规则集（format_rules_default + 每条 format_rules 带 rules_source）。
|
||||
无 LLM / 无 vstore，同步执行通常 <1s；恒 200 + {ok, checks, summary, meta|error} 信封。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, File, Form, UploadFile
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..config import get_settings
|
||||
from ..services import format_check
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-format"])
|
||||
|
||||
|
||||
class CheckItem(BaseModel):
|
||||
key: str
|
||||
name: str
|
||||
status: str # pass | warn | fail
|
||||
detail: str
|
||||
suggestion: str = ""
|
||||
|
||||
|
||||
class FormatCheckResponse(BaseModel):
|
||||
ok: bool
|
||||
checks: list[CheckItem] | None = None
|
||||
summary: dict[str, int] | None = None
|
||||
meta: dict[str, object] | None = None
|
||||
error: dict[str, object] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str, *, degraded: bool = False) -> FormatCheckResponse:
|
||||
return FormatCheckResponse(
|
||||
ok=False, error={"code": code, "message": message, "degraded": degraded},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/format-check", response_model=FormatCheckResponse)
|
||||
async def format_check_endpoint(
|
||||
file: UploadFile = File(..., description="待检查的 PDF / Word(.docx) / LaTeX(.tex) 文档"),
|
||||
target_journal: str = Form("", description="目标期刊（可选，传 journals 的 id）"),
|
||||
) -> FormatCheckResponse:
|
||||
settings = get_settings()
|
||||
|
||||
filename = file.filename or ""
|
||||
low = filename.lower()
|
||||
if low.endswith(".pdf"):
|
||||
kind = "pdf"
|
||||
elif low.endswith(".docx"):
|
||||
kind = "docx"
|
||||
elif low.endswith(".tex"):
|
||||
kind = "tex"
|
||||
else:
|
||||
return _error("VALIDATION", "仅支持 PDF / Word（.docx）/ LaTeX（.tex）文件")
|
||||
|
||||
content = await file.read()
|
||||
if not content:
|
||||
return _error("VALIDATION", "上传文件为空")
|
||||
if len(content) > settings.max_pdf_bytes:
|
||||
return _error("VALIDATION", "上传文件超过 50MB 上限")
|
||||
|
||||
try:
|
||||
result = format_check.check_document(content, kind, (target_journal or "").strip())
|
||||
except format_check.FormatCheckError as exc:
|
||||
logger.warning("format-check: 解档失败：%s", exc)
|
||||
return _error("PARSE_FAILED", str(exc))
|
||||
|
||||
meta = dict(result.get("meta") or {})
|
||||
meta["filename"] = filename
|
||||
return FormatCheckResponse(
|
||||
ok=True,
|
||||
checks=[CheckItem(**c) for c in result["checks"]],
|
||||
summary=result["summary"],
|
||||
meta=meta,
|
||||
)
|
||||
@@ -1,280 +0,0 @@
|
||||
"""routers/journal_directory.py —— 期刊目录浏览（1056 本，只读分页）。
|
||||
|
||||
- GET /api/review/journals/directory：journal_directory 表（scripts/import_journals.py
|
||||
导入的 v4 目录：2025 中科院分区 + 2025 JIF（2026 年 JCR 发布）+ 年报 2025 版
|
||||
（中文统计年 2024）+ 官网网址/核对状态）→ 过滤/排序/分页 → 列表 + meta。
|
||||
- 与 GET /journals（journals.json 精选 5 本，供 FormatPanel 下拉/recommend 语义）物理
|
||||
分离（方案 A）；本端点绝不读写 journals 表，recommend/journal_kb/journal_rank 零触碰。
|
||||
- 读路径只 SELECT；表由导入脚本拥有（db.py 零改动）；表不存在 → data_imported=false
|
||||
优雅降级（HTTP 200 空列表）。
|
||||
|
||||
读时归一化（拍板细节 1-3，均在 API 层做，不动脚本/DB 原值）：
|
||||
细节1 lang=zh → cas_partition 恒 null + cas_partition_note="不按中科院分区口径列示"；
|
||||
细节2 jif_note 源文本映射（v4 值域已是用户可读文本，旧值映射保留兜底）；
|
||||
细节3 缺值即矛盾清零：en 且 jif=null → jif_source_url=null。
|
||||
（zh 的 IF 载体是 composite/comprehensive，其 jif_source_url 本恒 null。）
|
||||
决策2：响应不下发"来源"文本字段（partition_source/jif_source/data_source_label
|
||||
已移除）；jif_source_url/data_source_url 保留供展开行链接。
|
||||
决策4（来源展示优化）：partition_source_url 不再下发——DB 原值分毫未动
|
||||
（仍 SELECT * 读出后被模型丢弃），响应改为常量字段 partition_source_note
|
||||
文案，前端以灰色说明文字替代原"分区来源"超链接。
|
||||
新增 official_check_status / jcr_release_year。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import sqlite3
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..db import db
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-directory"])
|
||||
|
||||
_LANGS = frozenset({"en", "zh"})
|
||||
# 排序列白名单（列名不可参数化 → 白名单防注入）
|
||||
_FIELDS = frozenset({
|
||||
"name",
|
||||
"jif",
|
||||
"composite_impact_factor",
|
||||
"comprehensive_impact_factor",
|
||||
"research_direction",
|
||||
"cas_partition",
|
||||
})
|
||||
_ORDERS = frozenset({"asc", "desc"})
|
||||
_PARTS = {"1": "1区", "2": "2区", "3": "3区", "4": "4区"}
|
||||
_MAX_SIZE = 200
|
||||
|
||||
# 细节2：源核对状态 → 用户可读 note
|
||||
_JIF_NOTE_MAP = {
|
||||
"源表标N/A": "未被2024 JCR收录",
|
||||
"所用年度来源未检得": "本次汇编未查到，建议查阅最新 JCR",
|
||||
}
|
||||
_ZH_PART_NOTE = "不按中科院分区口径列示"
|
||||
# 决策4：分区来源改为常量说明文案（不下发 partition_source_url；DB 原值不动）
|
||||
_PARTITION_SOURCE_NOTE = (
|
||||
"2025 中科院官方表（2026 年起停更；官方入口 fenqubiao.com 需机构登录）"
|
||||
)
|
||||
|
||||
_NOTE_EN = (
|
||||
"分区：2025年中科院公开表（多表比对，2026年起停止更新）；"
|
||||
"JIF：2025年度指标（2026年JCR发布，两份公开汇编一致或已核对出版社官方指标）；"
|
||||
"整理日期2026-10-10。空白=来源未给值≠0；研究方向/期刊标记/文章类型为导航标签，非官方评级。"
|
||||
)
|
||||
_NOTE_ZH = (
|
||||
"指标：国内自然科学期刊年报2025版（统计年2024，official，含复合/综合IF两口径）；"
|
||||
"中文期刊不按中科院分区口径列示；国内IF与JIF统计源不同，不可混排。"
|
||||
)
|
||||
_UNIMPORTED_NOTE = "目录数据未导入（请先运行 scripts/import_journals.py）"
|
||||
|
||||
|
||||
# ---- 响应模型 ----
|
||||
|
||||
class DirectoryItem(BaseModel):
|
||||
"""目录公开字段（DB 全列白名单；imported_at 等内部列丢弃）。"""
|
||||
id: str
|
||||
lang: str
|
||||
name: str
|
||||
research_direction: str | None = None
|
||||
cas_partition: str | None = None
|
||||
cas_partition_note: str | None = None
|
||||
cas_partition_year: int | None = None
|
||||
jif: float | None = None
|
||||
jif_year: int | None = None
|
||||
jif_note: str | None = None
|
||||
jif_check_status: str | None = None
|
||||
composite_impact_factor: float | None = None
|
||||
comprehensive_impact_factor: float | None = None
|
||||
metric_year: int | None = None
|
||||
metric_report_year: int | None = None
|
||||
metric_caliber: str | None = None
|
||||
annual_report_page: int | None = None
|
||||
top_flag: str | None = None
|
||||
publisher_series: str | None = None
|
||||
journal_mark: str | None = None
|
||||
article_type: str | None = None
|
||||
issn: str | None = None
|
||||
eissn: str | None = None
|
||||
jcr_disciplines: list[str] = []
|
||||
supplement_note: str | None = None
|
||||
evidence_note: str | None = None
|
||||
official_url: str | None = None
|
||||
official_check_status: str | None = None
|
||||
jcr_release_year: int | None = None
|
||||
jif_source_url: str | None = None
|
||||
partition_source_note: str = _PARTITION_SOURCE_NOTE # 决策4：常量，替代分区来源链接
|
||||
data_source_url: str | None = None
|
||||
|
||||
|
||||
class DirectoryResponse(BaseModel):
|
||||
ok: bool
|
||||
items: list[DirectoryItem] = []
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str) -> DirectoryResponse:
|
||||
return DirectoryResponse(
|
||||
ok=False, items=[], meta=None,
|
||||
error={"code": code, "message": message, "degraded": False},
|
||||
)
|
||||
|
||||
|
||||
# ---- 读时归一化（拍板细节 1-3）----
|
||||
|
||||
def _present(raw: dict[str, Any], lang: str) -> dict[str, Any]:
|
||||
d = dict(raw)
|
||||
# jcr_disciplines：JSON 数组字符串 → list
|
||||
jcr = d.get("jcr_disciplines")
|
||||
if jcr:
|
||||
try:
|
||||
parsed = json.loads(jcr)
|
||||
d["jcr_disciplines"] = parsed if isinstance(parsed, list) else []
|
||||
except (TypeError, ValueError):
|
||||
d["jcr_disciplines"] = []
|
||||
else:
|
||||
d["jcr_disciplines"] = []
|
||||
|
||||
# 细节2：jif_note 源文本 → 用户可读
|
||||
note = d.get("jif_note")
|
||||
if note in _JIF_NOTE_MAP:
|
||||
d["jif_note"] = _JIF_NOTE_MAP[note]
|
||||
|
||||
# 细节1：zh 分区归一（DB 原值"不按此口径列示"只留在库里）
|
||||
if lang == "zh":
|
||||
d["cas_partition"] = None
|
||||
d["cas_partition_note"] = _ZH_PART_NOTE
|
||||
|
||||
# 细节3：缺值 → 来源链接清零（消除 null 值却挂来源链接的矛盾；
|
||||
# 分区来源链接已按决策4 移出响应，改为常量 partition_source_note）
|
||||
if lang == "en":
|
||||
if d.get("jif") is None:
|
||||
d["jif_source_url"] = None
|
||||
return d
|
||||
|
||||
|
||||
def _escape_like(s: str) -> str:
|
||||
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
|
||||
|
||||
# ---- GET /api/review/journals/directory ----
|
||||
|
||||
@router.get("/journals/directory", response_model=DirectoryResponse)
|
||||
async def journals_directory(
|
||||
lang: str = "en",
|
||||
domain: str = "",
|
||||
cas_partition: str = "",
|
||||
q: str = "",
|
||||
page: int = 1,
|
||||
size: int = 50,
|
||||
field: str = "",
|
||||
order: str = "",
|
||||
) -> DirectoryResponse:
|
||||
"""只读：期刊目录分页浏览（无写接口）。
|
||||
|
||||
- ?lang=en|zh（默认 en；**强制单语，保证国际/中文永不混排**）；
|
||||
- ?domain= 研究方向精确匹配；?cas_partition=1|2|3|4（仅 lang=en，zh 忽略）；
|
||||
- ?q= 期刊名模糊（ASCII 大小写不敏感）；?page≥1 ?size 1-200（默认 50）；
|
||||
- ?field= 排序白名单（name/jif/composite_impact_factor/
|
||||
comprehensive_impact_factor/research_direction/cas_partition），?order=asc|desc；
|
||||
未传 field → 默认排序 en=jif 降序 / zh=复合IF 降序（NULL 恒排最后）；
|
||||
- 参数非法 → VALIDATION 信封；表不存在 → data_imported=false 空列表（HTTP 200）。
|
||||
"""
|
||||
lang_n = (lang or "").strip().lower()
|
||||
if lang_n not in _LANGS:
|
||||
return _error("VALIDATION", f"不支持的 lang={lang!r}，可选：en / zh")
|
||||
if page < 1:
|
||||
return _error("VALIDATION", f"page 必须 ≥ 1，收到 {page}")
|
||||
if size < 1 or size > _MAX_SIZE:
|
||||
return _error("VALIDATION", f"size 必须在 1-{_MAX_SIZE}，收到 {size}")
|
||||
field_n = (field or "").strip()
|
||||
if field_n and field_n not in _FIELDS:
|
||||
return _error(
|
||||
"VALIDATION",
|
||||
f"不支持的 field={field!r}，可选：{', '.join(sorted(_FIELDS))}",
|
||||
)
|
||||
order_n = (order or "").strip().lower()
|
||||
if order_n and order_n not in _ORDERS:
|
||||
return _error("VALIDATION", f"不支持的 order={order!r}，可选：asc / desc")
|
||||
part_q = (cas_partition or "").strip()
|
||||
if part_q:
|
||||
part_q = _PARTS.get(part_q, _PARTS.get(part_q.rstrip("区").strip(), ""))
|
||||
if not part_q:
|
||||
return _error(
|
||||
"VALIDATION",
|
||||
f"不支持的 cas_partition={cas_partition!r}，可选：1 / 2 / 3 / 4",
|
||||
)
|
||||
|
||||
# 过滤条件（白名单拼接；参数全部占位符）
|
||||
where = ["lang = ?"]
|
||||
params: list[Any] = [lang_n]
|
||||
domain_n = (domain or "").strip()
|
||||
if domain_n:
|
||||
where.append("research_direction = ?")
|
||||
params.append(domain_n)
|
||||
if part_q and lang_n == "en": # zh 不分区：忽略（meta 回显）
|
||||
where.append("cas_partition = ?")
|
||||
params.append(part_q)
|
||||
q_n = (q or "").strip()
|
||||
if q_n:
|
||||
where.append("name LIKE ? ESCAPE '\\'")
|
||||
params.append(f"%{_escape_like(q_n)}%")
|
||||
where_sql = " AND ".join(where)
|
||||
|
||||
# 排序（列名出自白名单；NULL 恒排最后）
|
||||
if field_n:
|
||||
dir_sql = "DESC" if order_n == "desc" else "ASC"
|
||||
order_sql = f"{field_n} IS NULL, {field_n} {dir_sql}"
|
||||
else:
|
||||
col = "jif" if lang_n == "en" else "composite_impact_factor"
|
||||
order_sql = f"{col} IS NULL, {col} DESC"
|
||||
|
||||
try:
|
||||
total_row = await db.fetchone(
|
||||
f"SELECT COUNT(*) AS c FROM journal_directory WHERE {where_sql}", params # noqa: S608
|
||||
)
|
||||
total = int(total_row["c"]) if total_row else 0
|
||||
rows = await db.fetchall(
|
||||
f"SELECT * FROM journal_directory WHERE {where_sql} " # noqa: S608
|
||||
f"ORDER BY {order_sql} LIMIT ? OFFSET ?",
|
||||
[*params, size, (page - 1) * size],
|
||||
)
|
||||
dom_rows = await db.fetchall(
|
||||
"SELECT research_direction AS d, COUNT(*) AS c FROM journal_directory "
|
||||
"WHERE lang = ? AND research_direction IS NOT NULL "
|
||||
"GROUP BY 1 ORDER BY c DESC, d ASC",
|
||||
[lang_n],
|
||||
)
|
||||
except sqlite3.OperationalError as exc:
|
||||
# 表不存在（脚本未跑）→ 优雅降级，HTTP 200
|
||||
logger.warning("journal_directory: 表不可用（目录未导入？）: %s", exc)
|
||||
return DirectoryResponse(
|
||||
ok=True, items=[],
|
||||
meta={
|
||||
"total": 0, "page": page, "size": size, "lang": lang_n,
|
||||
"domain": domain_n, "cas_partition": part_q, "q": q_n,
|
||||
"domains": [], "data_source_note": _UNIMPORTED_NOTE,
|
||||
"data_imported": False,
|
||||
},
|
||||
error=None,
|
||||
)
|
||||
|
||||
items = [DirectoryItem(**_present(dict(r), lang_n)) for r in rows]
|
||||
meta: dict[str, Any] = {
|
||||
"total": total,
|
||||
"page": page,
|
||||
"size": size,
|
||||
"lang": lang_n,
|
||||
"domain": domain_n,
|
||||
"cas_partition": part_q if lang_n == "en" else "",
|
||||
"q": q_n,
|
||||
"domains": [r["d"] for r in dom_rows if r["d"]],
|
||||
"data_source_note": _NOTE_EN if lang_n == "en" else _NOTE_ZH,
|
||||
"data_imported": True,
|
||||
}
|
||||
return DirectoryResponse(ok=True, items=items, meta=meta, error=None)
|
||||
@@ -1,354 +0,0 @@
|
||||
"""routers/journals.py —— 期刊情报与智能投稿推荐。
|
||||
|
||||
- GET /api/review/journals 只读：journals 表全量列表（?field=/&order= 排序），
|
||||
meta 含 count / last_updated / data_source_note。无任何写接口；空表返回
|
||||
空数组 + meta.count=0（HTTP 200，绝不 500）。
|
||||
- POST /api/review/recommend 智能投稿推荐：vstore 向量匹配（摘要 vs
|
||||
scope_desc，白名单检索）→ LLM 按 id 白名单重排（幻觉期刊丢弃、缺漏规则
|
||||
补位）→ 三档输出：挑战型 / 适配型 / 稳妥型，附匹配理由与风险提示。
|
||||
|
||||
降级哲学（同 presubmit）：向量失败 → 全量候选 + vector_ok=false；
|
||||
LLM 失败 → 规则分档 + llm_ok=false；推荐永不 5xx。仅入参非法返回
|
||||
VALIDATION 错误信封（HTTP 200，与本服务其余端点一致）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..config import get_settings
|
||||
from ..db import SeedError, ensure_journals_seeded, list_journals
|
||||
from ..services import journal_kb, journal_rank, llm
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-journals"])
|
||||
|
||||
|
||||
# ---- 请求/响应模型 ----
|
||||
|
||||
class Preferences(BaseModel):
|
||||
max_review_weeks: float | None = Field(default=None, ge=1, le=104)
|
||||
min_impact_factor: float | None = Field(default=None, ge=0, le=200)
|
||||
tier_focus: Literal["all", "challenge", "fit", "safe"] = "all"
|
||||
|
||||
|
||||
class RecommendRequest(BaseModel):
|
||||
abstract: str
|
||||
preferences: Preferences = Preferences()
|
||||
|
||||
|
||||
class JournalItem(BaseModel):
|
||||
id: str
|
||||
name: str
|
||||
impact_factor: float | None = None
|
||||
review_time_weeks: float | None = None
|
||||
acceptance_rate: float | None = None
|
||||
official_url: str | None = None
|
||||
similarity: float | None = None
|
||||
tier: str
|
||||
match_reason: str
|
||||
risk_note: str
|
||||
|
||||
|
||||
class TierBox(BaseModel):
|
||||
label: str
|
||||
items: list[JournalItem]
|
||||
|
||||
|
||||
class RecommendResponse(BaseModel):
|
||||
ok: bool
|
||||
tiers: dict[str, TierBox] | None = None
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
# ---- GET /api/review/journals（只读列表）----
|
||||
|
||||
class JournalBrief(BaseModel):
|
||||
"""期刊公开字段（journals 表全列；不含 vstore_doc_id 内部映射）。
|
||||
|
||||
标签/分区 6 字段不进 SQLite：GET 时按 id 从 journals.json 内存缓存合并；
|
||||
JSON 缺该 id → 空数组/null 默认值。
|
||||
"""
|
||||
id: str
|
||||
name: str
|
||||
scope_desc: str = ""
|
||||
impact_factor: float | None = None
|
||||
review_time_weeks: float | None = None
|
||||
acceptance_rate: float | None = None
|
||||
submission_deadline: str | None = None
|
||||
official_url: str | None = None
|
||||
last_updated: str = ""
|
||||
scope_tags: list[str] = []
|
||||
application_tags: list[str] = []
|
||||
cas_partition: str | None = None
|
||||
jcr_quartile: str | None = None
|
||||
partition_source: str | None = None
|
||||
cross_domain: list[str] = []
|
||||
|
||||
|
||||
class JournalsListResponse(BaseModel):
|
||||
ok: bool
|
||||
items: list[JournalBrief] = []
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
# ?field= 排序白名单（列名无法参数化，白名单防注入；非法值 → VALIDATION 信封）
|
||||
_SORT_FIELDS = frozenset({
|
||||
"id",
|
||||
"name",
|
||||
"impact_factor",
|
||||
"review_time_weeks",
|
||||
"acceptance_rate",
|
||||
"last_updated",
|
||||
})
|
||||
_DATA_SOURCE_NOTE = (
|
||||
"IF/审稿周期为公开近似值，acceptance_rate 为 null 表示暂无公开数据，需人工核验。"
|
||||
"CAS/JCR 分区为首版占位数据，需核验。"
|
||||
)
|
||||
|
||||
# journals.json 新增字段（标签/分区；刻意不进 SQLite，见 C-2 方案）
|
||||
_TAG_FIELDS = (
|
||||
"scope_tags",
|
||||
"application_tags",
|
||||
"cas_partition",
|
||||
"jcr_quartile",
|
||||
"partition_source",
|
||||
"cross_domain",
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _load_tag_extras() -> dict[str, dict[str, Any]]:
|
||||
"""journals.json 标签/分区字段 → {id: {6 新字段}}。
|
||||
|
||||
首次请求读盘后进程内缓存，后续 GET O(1) 按 id 合并（不做每请求磁盘 IO）。
|
||||
JSON 缺该 id → 合并时用空数组/null 默认；文件不可读 → 空 dict 降级（不 500）。
|
||||
"""
|
||||
seed = get_settings().journals_seed_abs
|
||||
try:
|
||||
payload = json.loads(Path(seed).read_text("utf-8"))
|
||||
except (OSError, ValueError) as exc:
|
||||
logger.warning("journals: 标签扩展文件不可读，合并降级为默认值：%s", exc)
|
||||
return {}
|
||||
raw = payload.get("journals") if isinstance(payload, dict) else None
|
||||
if not isinstance(raw, list):
|
||||
return {}
|
||||
out: dict[str, dict[str, Any]] = {}
|
||||
for j in raw:
|
||||
if isinstance(j, dict) and j.get("id"):
|
||||
out[str(j["id"])] = {k: j.get(k) for k in _TAG_FIELDS}
|
||||
return out
|
||||
|
||||
|
||||
def _tag_fields(extra: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""缓存条目 → JournalBrief 关键字参数（JSON 缺该 id → 空数组/null 默认）。"""
|
||||
ex = extra or {}
|
||||
return {
|
||||
"scope_tags": list(ex.get("scope_tags") or []),
|
||||
"application_tags": list(ex.get("application_tags") or []),
|
||||
"cas_partition": ex.get("cas_partition"),
|
||||
"jcr_quartile": ex.get("jcr_quartile"),
|
||||
"partition_source": ex.get("partition_source"),
|
||||
"cross_domain": list(ex.get("cross_domain") or []),
|
||||
}
|
||||
|
||||
|
||||
def _sort_rows(rows: list[dict[str, Any]], field: str, order: str) -> list[dict[str, Any]]:
|
||||
"""按白名单字段排序；NULL 恒排最后；次序不稳定时按 db 默认序（稳定排序保留）。"""
|
||||
present = [r for r in rows if r.get(field) is not None]
|
||||
missing = [r for r in rows if r.get(field) is None]
|
||||
present.sort(key=lambda r: r.get(field), reverse=(order == "desc"))
|
||||
return present + missing
|
||||
|
||||
|
||||
def _list_error(message: str) -> JournalsListResponse:
|
||||
return JournalsListResponse(
|
||||
ok=False, items=[], meta=None,
|
||||
error={"code": "VALIDATION", "message": message, "degraded": False},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/journals", response_model=JournalsListResponse)
|
||||
async def journals_list(
|
||||
field: str = "name",
|
||||
order: str = "asc",
|
||||
) -> JournalsListResponse:
|
||||
"""只读：journals 表全量列表（无写/删/改接口）。
|
||||
|
||||
- ?field= 排序字段白名单：id/name/impact_factor/review_time_weeks/
|
||||
acceptance_rate/last_updated，默认 name；
|
||||
- ?order=asc|desc，默认 asc；
|
||||
- 空表 → items=[] + meta.count=0（HTTP 200，不报错）。种子装载由
|
||||
POST /recommend 首次调用幂等完成，本端点严格只读。
|
||||
"""
|
||||
col = (field or "").strip()
|
||||
if col not in _SORT_FIELDS:
|
||||
return _list_error(
|
||||
f"不支持的排序字段 field={field!r}，可选：{', '.join(sorted(_SORT_FIELDS))}"
|
||||
)
|
||||
ord_norm = (order or "asc").strip().lower()
|
||||
if ord_norm not in ("asc", "desc"):
|
||||
return _list_error(f"不支持的排序方向 order={order!r}，可选：asc, desc")
|
||||
|
||||
rows = await list_journals()
|
||||
rows = _sort_rows(rows, col, ord_norm)
|
||||
|
||||
tag_extras = _load_tag_extras()
|
||||
items = [
|
||||
JournalBrief(
|
||||
id=str(r.get("id") or ""),
|
||||
name=str(r.get("name") or ""),
|
||||
scope_desc=str(r.get("scope_desc") or ""),
|
||||
impact_factor=r.get("impact_factor"),
|
||||
review_time_weeks=r.get("review_time_weeks"),
|
||||
acceptance_rate=r.get("acceptance_rate"),
|
||||
submission_deadline=r.get("submission_deadline"),
|
||||
official_url=r.get("official_url"),
|
||||
last_updated=str(r.get("last_updated") or ""),
|
||||
**_tag_fields(tag_extras.get(str(r.get("id") or ""))),
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
latest = max((str(r.get("last_updated") or "") for r in rows), default="")
|
||||
meta: dict[str, Any] = {
|
||||
"count": len(items),
|
||||
"last_updated": latest or None,
|
||||
"data_source_note": _DATA_SOURCE_NOTE,
|
||||
}
|
||||
return JournalsListResponse(ok=True, items=items, meta=meta, error=None)
|
||||
|
||||
|
||||
def _error(code: str, message: str, *, degraded: bool = False) -> RecommendResponse:
|
||||
return RecommendResponse(
|
||||
ok=False, tiers=None, meta=None,
|
||||
error={"code": code, "message": message, "degraded": degraded},
|
||||
)
|
||||
|
||||
|
||||
def _item_sort_key(it: JournalItem) -> tuple[int, float, float]:
|
||||
return (
|
||||
1 if it.similarity is None else 0,
|
||||
-(it.similarity or 0.0),
|
||||
-(it.impact_factor or 0.0),
|
||||
)
|
||||
|
||||
|
||||
def _apply_soft_preferences(item: JournalItem, prefs: Preferences) -> None:
|
||||
"""软偏好：不满足只加风险提示，不删条目。"""
|
||||
weeks = item.review_time_weeks
|
||||
if prefs.max_review_weeks is not None and weeks is not None and weeks > prefs.max_review_weeks:
|
||||
prefix = f"⚠ 审稿约 {weeks:g} 周，超出你 {prefs.max_review_weeks:g} 周的周期偏好；"
|
||||
item.risk_note = prefix + ("无" if item.risk_note == "无" else item.risk_note)
|
||||
inf = item.impact_factor
|
||||
if prefs.min_impact_factor is not None:
|
||||
if inf is None:
|
||||
item.risk_note = "⚠ 影响因子未收录，无法确认满足你的要求；" + item.risk_note
|
||||
elif inf < prefs.min_impact_factor:
|
||||
prefix = f"⚠ 影响因子 {inf:g} 低于你 {prefs.min_impact_factor:g} 的要求；"
|
||||
item.risk_note = prefix + ("无" if item.risk_note == "无" else item.risk_note)
|
||||
|
||||
|
||||
@router.post("/recommend", response_model=RecommendResponse)
|
||||
async def recommend(req: RecommendRequest) -> RecommendResponse:
|
||||
settings = get_settings()
|
||||
|
||||
# ---- 入参校验（手动，错误信封与本服务其余端点一致）----
|
||||
abstract = (req.abstract or "").strip()
|
||||
if len(abstract) < 50:
|
||||
return _error("VALIDATION", "abstract 至少需要 50 个字符（请提供摘要或正文片段）")
|
||||
if len(abstract) > 20000:
|
||||
return _error("VALIDATION", "abstract 超过 20000 字符上限")
|
||||
|
||||
prefs = req.preferences
|
||||
|
||||
# ---- 种子 + 期刊元数据 ----
|
||||
try:
|
||||
await ensure_journals_seeded()
|
||||
journals = await list_journals()
|
||||
except SeedError as exc:
|
||||
logger.error("recommend: 种子装载失败：%s", exc)
|
||||
return _error("SEED_UNAVAILABLE", str(exc), degraded=True)
|
||||
if not journals:
|
||||
return _error("SEED_UNAVAILABLE", "期刊库为空", degraded=True)
|
||||
|
||||
# ---- ① vstore 向量匹配（摘要 vs scope_desc）----
|
||||
try:
|
||||
candidates, vector_ok = await journal_kb.match(abstract, journals)
|
||||
except journal_kb.MatchError as exc:
|
||||
logger.warning("recommend: 向量匹配降级：%s", exc)
|
||||
candidates, vector_ok = [], False
|
||||
if not candidates:
|
||||
# 无有效语义分 → 全量候选（similarity=None），LLM 仍可按 scope 文本重排
|
||||
candidates = [{"journal": j, "similarity": None} for j in journals]
|
||||
vector_ok = False
|
||||
|
||||
# ---- ② LLM 重排（id 白名单硬校验）----
|
||||
prefs_dict = prefs.model_dump(exclude_none=True)
|
||||
llm_ok = True
|
||||
dropped = 0
|
||||
repaired = 0
|
||||
try:
|
||||
rankings, dropped, repaired = await journal_rank.rerank(abstract, candidates, prefs_dict)
|
||||
except (llm.LLMError, journal_rank.RankError, ValueError) as exc:
|
||||
# ValueError：complete_json 最终 json.loads 修复仍失败（同 presubmit 宽容口径）
|
||||
logger.warning("recommend: LLM 重排降级为规则分档：%s", exc)
|
||||
rankings = journal_rank.fallback_rankings(candidates)
|
||||
llm_ok = False
|
||||
|
||||
# ---- ③ 三档组装（名称/指标全部来自 DB，不采信 LLM 文本）----
|
||||
by_id = {str(c["journal"]["id"]): c for c in candidates}
|
||||
tiers: dict[str, list[JournalItem]] = {t: [] for t in journal_rank.TIERS}
|
||||
for r in rankings:
|
||||
cand = by_id.get(r.journal_id)
|
||||
if cand is None:
|
||||
continue
|
||||
j = cand["journal"]
|
||||
sim = cand.get("similarity")
|
||||
item = JournalItem(
|
||||
id=str(j["id"]),
|
||||
name=str(j.get("name") or j["id"]),
|
||||
impact_factor=j.get("impact_factor"),
|
||||
review_time_weeks=j.get("review_time_weeks"),
|
||||
acceptance_rate=j.get("acceptance_rate"),
|
||||
official_url=j.get("official_url"),
|
||||
similarity=round(float(sim), 4) if sim is not None else None,
|
||||
tier=r.tier,
|
||||
match_reason=r.match_reason,
|
||||
risk_note=r.risk_note,
|
||||
)
|
||||
_apply_soft_preferences(item, prefs)
|
||||
tiers.setdefault(r.tier, []).append(item)
|
||||
|
||||
# 档内排序：相似度（None 靠后）→ 影响因子
|
||||
for box in tiers.values():
|
||||
box.sort(key=_item_sort_key)
|
||||
|
||||
# tier_focus：软展示聚焦（默认 all；聚焦时其余档 items 置空，label 保留）
|
||||
out_tiers: dict[str, TierBox] = {}
|
||||
for t in journal_rank.TIERS:
|
||||
items = tiers.get(t, [])
|
||||
if prefs.tier_focus != "all" and t != prefs.tier_focus:
|
||||
items = []
|
||||
out_tiers[t] = TierBox(label=journal_rank.TIER_LABELS[t], items=items)
|
||||
|
||||
meta: dict[str, Any] = {
|
||||
"model": llm._llm_config()["model"],
|
||||
"vector_ok": vector_ok,
|
||||
"llm_ok": llm_ok,
|
||||
"seed_count": len(journals),
|
||||
"candidate_count": len(candidates),
|
||||
"dropped_hallucinated": dropped,
|
||||
"repaired": repaired,
|
||||
"preferences": prefs_dict,
|
||||
"abstract_chars": len(abstract),
|
||||
}
|
||||
return RecommendResponse(ok=True, tiers=out_tiers, meta=meta, error=None)
|
||||
@@ -1,269 +0,0 @@
|
||||
"""routers/presubmit.py —— 投稿前自审稿（POST /api/review/presubmit）。
|
||||
|
||||
流程：上传 PDF → paper_parse（vstore 8094 解析）→ 段落感知装箱 + token 双层保护
|
||||
→ LLM 结构化审稿（DeepSeek json_object，System Prompt 为植物基因组学六维专属
|
||||
清单，见 services/self_review.py）→ coerce 归一化 → 关键词扫描器证伪硬性缺项
|
||||
（services/paper_scanner.py）→ 规则引擎计算 recommendation。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
|
||||
from fastapi import APIRouter, File, Form, UploadFile
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..config import get_settings
|
||||
from ..services import llm, paper_parse, paper_scanner, self_review
|
||||
from ..services.recommend_engine import decide_recommendation, resolve_tier
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-presubmit"])
|
||||
|
||||
# ---- token 双层保护（2026-10-12）----
|
||||
# 第一层：估算 token 超阈值 → 自动降到 TOKEN_GUARD_REPACK_CHARS 字符重装箱
|
||||
# 第二层：LLM 返回 HTTP 400/413 → 降到 LLM_REJECT_RETRY_CHARS 字符重试一次
|
||||
_EST_TOKENS_DIVISOR = 1.5 # 粗略上界：英文 ~4 字符/token，1.5 偏保守
|
||||
_TOKEN_GUARD_THRESHOLD = 48000
|
||||
_TOKEN_GUARD_REPACK_CHARS = 100000
|
||||
_LLM_REJECT_RETRY_CHARS = 80000
|
||||
# 无节标题可辨时的尾窗（捞页尾 Data Availability 块）
|
||||
_TAIL_FALLBACK_CHARS = 5000
|
||||
|
||||
# 节标题（行首、允许编号前缀、独占一行）；键 → 装箱分组名
|
||||
_SECTION_HEADER_RE = re.compile(
|
||||
r"^[ \t]*(?:\d+(?:\.\d+)*\.?\s+)?"
|
||||
r"(Abstract|Summary|Introduction|Results|Discussion|Methods"
|
||||
r"|Materials\s+and\s+Methods"
|
||||
r"|Data\s+(?:and\s+(?:code\s+)?)?[Aa]vailability"
|
||||
r"|Availability\s+of\s+[Dd]ata"
|
||||
r"|Data\s+and\s+[Mm]aterials\s+[Aa]vailability"
|
||||
r"|References|Acknowledg(?:e)?ments?)\s*$",
|
||||
re.IGNORECASE | re.MULTILINE,
|
||||
)
|
||||
_SECTION_GROUP = {
|
||||
"abstract": "abstract", "summary": "abstract",
|
||||
"introduction": "introduction",
|
||||
"results": "results",
|
||||
"discussion": "discussion",
|
||||
"methods": "methods", "materials and methods": "methods",
|
||||
"data availability": "data_availability",
|
||||
"data and code availability": "data_availability",
|
||||
"data availability statement": "data_availability",
|
||||
"availability of data": "data_availability",
|
||||
"data and materials availability": "data_availability",
|
||||
"references": "references", "acknowledgements": "acknowledgements",
|
||||
"acknowledgments": "acknowledgements",
|
||||
}
|
||||
# 装箱优先级（首条命中优先；methods 限 60% 预算——六维清单大量依赖 Methods）
|
||||
_PACK_PRIORITY = ("abstract", "data_availability", "methods", "results", "discussion")
|
||||
_METHODS_CAP_RATIO = 0.6
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""粗略 token 上界估计（len/1.5），无需 tokenizer 依赖。"""
|
||||
return int(len(text) / _EST_TOKENS_DIVISOR)
|
||||
|
||||
|
||||
def _split_segments(text: str) -> list[tuple[str, str]] | None:
|
||||
"""按节标题切段 → [(group, content)...]。无可辨标题 → None。"""
|
||||
matches = list(_SECTION_HEADER_RE.finditer(text))
|
||||
if not matches:
|
||||
return None
|
||||
segments: list[tuple[str, str]] = []
|
||||
if matches[0].start() > 0:
|
||||
segments.append(("preamble", text[:matches[0].start()]))
|
||||
for i, m in enumerate(matches):
|
||||
end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
|
||||
group = _SECTION_GROUP.get(m.group(1).casefold(), m.group(1).casefold())
|
||||
segments.append((group, text[m.start():end]))
|
||||
return segments
|
||||
|
||||
|
||||
def pack_feed_text(text: str, budget: int) -> tuple[str, list[str]]:
|
||||
"""段落感知装箱 → (feed_text, dropped_sections)。
|
||||
|
||||
预算内：原文返回。超预算：按优先级保留
|
||||
① Abstract ② Data Availability 全文 ③ Methods 前 60% ④ Results
|
||||
⑤ Discussion，其余按原序填满；未收段记入 dropped。
|
||||
无节标题可辨 → 头 + 尾窗拼接（捞页尾 availability 块），不优于现状亦不劣于。
|
||||
"""
|
||||
if len(text) <= budget:
|
||||
return text, []
|
||||
segments = _split_segments(text)
|
||||
if not segments:
|
||||
sep = "\n[...]\n"
|
||||
head = max(0, budget - _TAIL_FALLBACK_CHARS - len(sep))
|
||||
return (
|
||||
text[:head] + sep + text[-_TAIL_FALLBACK_CHARS:],
|
||||
["[无节标题回退：保留头+尾窗]"],
|
||||
)
|
||||
methods_cap = int(budget * _METHODS_CAP_RATIO)
|
||||
chosen: dict[int, str] = {}
|
||||
used = 0
|
||||
for group in _PACK_PRIORITY:
|
||||
for i, (g, content) in enumerate(segments):
|
||||
if g != group or i in chosen:
|
||||
continue
|
||||
room = budget - used
|
||||
if room <= 0:
|
||||
break
|
||||
piece = content[:methods_cap] if group == "methods" else content
|
||||
if len(piece) > room:
|
||||
piece = piece[:room]
|
||||
if not piece:
|
||||
break
|
||||
chosen[i] = piece
|
||||
used += len(piece)
|
||||
break
|
||||
for i, (_, content) in enumerate(segments):
|
||||
if i in chosen or used >= budget:
|
||||
continue
|
||||
room = budget - used
|
||||
chosen[i] = content if len(content) <= room else content[:room]
|
||||
used += len(chosen[i])
|
||||
feed = "".join(chosen[i] for i in sorted(chosen))
|
||||
dropped: list[str] = []
|
||||
for i, (g, _) in enumerate(segments):
|
||||
if i not in chosen and g not in dropped:
|
||||
dropped.append(g)
|
||||
return feed, dropped
|
||||
|
||||
|
||||
def _fit_feed_text(text: str, budget: int) -> tuple[str, list[str]]:
|
||||
"""装箱 + 第一层 token 保护（估算超阈值自动降到 100K 字符重装）。"""
|
||||
feed, dropped = pack_feed_text(text, budget)
|
||||
if estimate_tokens(feed) > _TOKEN_GUARD_THRESHOLD and len(feed) > _TOKEN_GUARD_REPACK_CHARS:
|
||||
logger.info(
|
||||
"presubmit: token 估算 %d > %d，降到 %d 字符重装箱",
|
||||
estimate_tokens(feed), _TOKEN_GUARD_THRESHOLD, _TOKEN_GUARD_REPACK_CHARS,
|
||||
)
|
||||
feed, dropped = pack_feed_text(text, _TOKEN_GUARD_REPACK_CHARS)
|
||||
return feed, dropped
|
||||
|
||||
|
||||
class PresubmitResponse(BaseModel):
|
||||
ok: bool
|
||||
report: self_review.SelfReviewReport | None = None
|
||||
error: dict[str, object] | None = None
|
||||
meta: dict[str, object] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str, *, degraded: bool = False) -> PresubmitResponse:
|
||||
return PresubmitResponse(
|
||||
ok=False, report=None,
|
||||
error={"code": code, "message": message, "degraded": degraded},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/presubmit", response_model=PresubmitResponse)
|
||||
async def presubmit_review(
|
||||
file: UploadFile = File(..., description="待审稿 PDF"),
|
||||
target_journal: str = Form("", description="目标期刊（可选，影响格式类意见）"),
|
||||
) -> PresubmitResponse:
|
||||
settings = get_settings()
|
||||
|
||||
filename = file.filename or "paper.pdf"
|
||||
if not filename.lower().endswith(".pdf"):
|
||||
return _error("VALIDATION", "仅支持 PDF 文件")
|
||||
|
||||
content = await file.read()
|
||||
if not content:
|
||||
return _error("VALIDATION", "上传文件为空")
|
||||
if len(content) > settings.max_pdf_bytes:
|
||||
return _error("VALIDATION", "PDF 超过 50MB 上限")
|
||||
|
||||
# ---- 解析（vstore 8094）----
|
||||
try:
|
||||
text = await paper_parse.parse_pdf_bytes(content, filename=filename)
|
||||
except paper_parse.ParseError as exc:
|
||||
logger.warning("presubmit: 解析失败：%s", exc)
|
||||
return _error("PARSE_FAILED", str(exc), degraded=True)
|
||||
|
||||
truncated = len(text) > settings.max_text_chars
|
||||
feed_text, dropped_sections = _fit_feed_text(text, settings.max_text_chars)
|
||||
|
||||
# ---- LLM 结构化自审（植物基因组学六维清单）----
|
||||
# 第二层保护：HTTP 400/413（疑似超长/形态拒绝）→ 降 80K 字符重试一次
|
||||
user_prompt = self_review.build_user_prompt(feed_text, (target_journal or "").strip())
|
||||
try:
|
||||
try:
|
||||
payload = await llm.complete_json(
|
||||
self_review.SYSTEM_PROMPT,
|
||||
user_prompt,
|
||||
temperature=0.2,
|
||||
max_tokens=8000,
|
||||
timeout=settings.presubmit_read_timeout,
|
||||
)
|
||||
except llm.LLMError as exc:
|
||||
msg = str(exc)
|
||||
if ("HTTP 400" in msg or "HTTP 413" in msg) and len(feed_text) > _LLM_REJECT_RETRY_CHARS:
|
||||
logger.warning("presubmit: LLM 拒绝输入（%s），降到 %d 字符重试",
|
||||
msg[:120], _LLM_REJECT_RETRY_CHARS)
|
||||
feed_text, dropped_sections = pack_feed_text(text, _LLM_REJECT_RETRY_CHARS)
|
||||
truncated = True
|
||||
payload = await llm.complete_json(
|
||||
self_review.SYSTEM_PROMPT,
|
||||
self_review.build_user_prompt(feed_text, (target_journal or "").strip()),
|
||||
temperature=0.2,
|
||||
max_tokens=8000,
|
||||
timeout=settings.presubmit_read_timeout,
|
||||
)
|
||||
else:
|
||||
raise
|
||||
except llm.LLMError as exc:
|
||||
logger.error("presubmit: LLM 审稿失败：%s", exc)
|
||||
return _error("UPSTREAM_DOWN", f"审稿模型调用失败：{exc}", degraded=True)
|
||||
|
||||
report = self_review.coerce_report(payload)
|
||||
|
||||
# ---- 关键词扫描器：证伪 LLM 的硬性缺项（有 X 却报缺 X → 真降级至 minor）----
|
||||
# A2（2026-10-12）：被冲突的 major 不再留在 major 列表里只剥标签，而是移出
|
||||
# major、append 到 minor（scanner_downgraded=True）——major_count/引擎输入真实
|
||||
# 减少；fatal 条目例外（只记冲突不降级，须人工过目）。
|
||||
signals = paper_scanner.scan_paper_signals(feed_text)
|
||||
kept_majors, demoted_minors, scanner_conflicts = paper_scanner.reconcile_hard_rules(
|
||||
report.major_weaknesses, signals
|
||||
)
|
||||
report.major_weaknesses = kept_majors
|
||||
if demoted_minors:
|
||||
report.minor_weaknesses = list(report.minor_weaknesses) + list(demoted_minors)
|
||||
|
||||
# recommendation 由规则引擎计算（LLM 输出的同名字段已被 coerce 忽略）
|
||||
tier, tier_source = resolve_tier((target_journal or "").strip())
|
||||
decision = decide_recommendation(
|
||||
majors=report.major_weaknesses,
|
||||
minors=report.minor_weaknesses,
|
||||
presentations=report.presentation_format,
|
||||
tier=tier,
|
||||
)
|
||||
rationale = decision["rationale"]
|
||||
rationale["scanner_conflicts"] = scanner_conflicts
|
||||
if scanner_conflicts:
|
||||
rationale["explanation"] += (
|
||||
f" 关键词扫描器与审稿意见存在 {len(scanner_conflicts)} 处冲突，"
|
||||
f"已降级为小修级问题，请人工核对。"
|
||||
)
|
||||
report.recommendation = decision["recommendation"]
|
||||
report.recommendation_rationale = rationale
|
||||
return PresubmitResponse(
|
||||
ok=True,
|
||||
report=report,
|
||||
meta={
|
||||
"filename": filename,
|
||||
"target_journal": (target_journal or "").strip(),
|
||||
"text_chars": len(feed_text),
|
||||
"truncated": truncated,
|
||||
"dropped_sections": dropped_sections,
|
||||
"scanner": {
|
||||
"accession_found": bool((signals.get("accession") or {}).get("found")),
|
||||
"busco_level": (signals.get("busco") or {}).get("level"),
|
||||
"hic_found": bool((signals.get("hic") or {}).get("found")),
|
||||
},
|
||||
"model": llm._llm_config()["model"],
|
||||
"tier": tier,
|
||||
"tier_label": decision["rationale"]["tier_label"],
|
||||
"tier_source": tier_source,
|
||||
},
|
||||
)
|
||||
@@ -1,288 +0,0 @@
|
||||
"""routers/rebuttal.py —— Rebuttal Phase 1（审稿回复助手，10 端点）。
|
||||
|
||||
信封与其余 review 端点一致：业务错误 = HTTP 200 + {ok:false,error:{code,message,degraded}}。
|
||||
- 文本粘贴主入口 / PDF 次要入口（PDF 经 vstore 8094 解析）；
|
||||
- 逐条 response → draft（校验C前置）→ confirm（后端重算 human_ack）→ letter 拼装；
|
||||
- letter.md 下载门槛 = human_ack；DELETE 手动清（无 TTL，决策 5）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, Literal
|
||||
|
||||
from fastapi import APIRouter, File, UploadFile
|
||||
from fastapi.responses import PlainTextResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..config import get_settings
|
||||
from ..services import llm, paper_parse
|
||||
from ..services.rebuttal_draft import PART_TYPES, assemble_letter, generate_draft, validate_parts
|
||||
from ..services.rebuttal_parse import parse_review_text
|
||||
from ..services import rebuttal_store
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review/rebuttal", tags=["review-rebuttal"])
|
||||
|
||||
_MAX_RAW_CHARS = 200_000 # 粘贴文本上限（审稿意见通常 <20K，保守给大）
|
||||
|
||||
|
||||
# ---------- 信封 ----------
|
||||
class RebuttalResponse(BaseModel):
|
||||
ok: bool
|
||||
job: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
meta: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str) -> RebuttalResponse:
|
||||
return RebuttalResponse(ok=False, error={"code": code, "message": message, "degraded": False})
|
||||
|
||||
|
||||
def _ok(job: dict[str, Any], **meta: Any) -> RebuttalResponse:
|
||||
return RebuttalResponse(ok=True, job=job, meta=meta or None)
|
||||
|
||||
|
||||
# ---------- 请求体 ----------
|
||||
class JobCreateBody(BaseModel):
|
||||
text: str = Field(min_length=1)
|
||||
|
||||
|
||||
class PartBody(BaseModel):
|
||||
type: Literal["new_data", "reanalysis", "location", "clarification", "rebuttal"]
|
||||
content: str = ""
|
||||
location_hint: str | None = None
|
||||
|
||||
|
||||
class ResponseBody(BaseModel):
|
||||
parts: list[PartBody] = Field(default_factory=list)
|
||||
|
||||
|
||||
class StructureBody(BaseModel):
|
||||
action: Literal["merge", "split"]
|
||||
# merge: comment_ids（含当前 cid，≥2 条）；split: items [{title, body}]
|
||||
comment_ids: list[str] | None = None
|
||||
items: list[dict[str, str]] | None = None
|
||||
|
||||
|
||||
class ConfirmBody(BaseModel):
|
||||
confirmed: bool
|
||||
|
||||
|
||||
# ---------- 内部工具 ----------
|
||||
async def _messages_llm(messages: list[dict[str, str]], *, temperature: float = 0.3) -> dict[str, Any]:
|
||||
"""适配 rebuttal_parse/draft 的 messages 形态 → llm.complete_json。"""
|
||||
system = next((m["content"] for m in messages if m.get("role") == "system"), "")
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
return await llm.complete_json(system, user, temperature=temperature)
|
||||
|
||||
|
||||
def _find_comment(job: dict[str, Any], cid: str) -> tuple[dict, dict] | None:
|
||||
for r in job["reviewers"]:
|
||||
for c in r["comments"]:
|
||||
if c["id"] == cid:
|
||||
return r, c
|
||||
return None
|
||||
|
||||
|
||||
async def _build_job(raw_text: str, *, kind: str, filename: str | None) -> RebuttalResponse:
|
||||
raw_text = (raw_text or "").strip()
|
||||
if not raw_text:
|
||||
return _error("VALIDATION", "审稿意见文本为空")
|
||||
if len(raw_text) > _MAX_RAW_CHARS:
|
||||
return _error("VALIDATION", f"文本超过 {_MAX_RAW_CHARS} 字符上限")
|
||||
reviewers = await parse_review_text(raw_text, _messages_llm)
|
||||
if not reviewers:
|
||||
return _error("PARSE_FAILED", "未能从文本中解析出任何审稿意见条目")
|
||||
job = {
|
||||
"id": rebuttal_store.new_id("rbt_"),
|
||||
"status": "ready",
|
||||
"source": {"kind": kind, "filename": filename, "raw_text": raw_text},
|
||||
"template": "generic",
|
||||
"reviewers": reviewers,
|
||||
"letter_md": None,
|
||||
"validation": None,
|
||||
"human_ack": False,
|
||||
}
|
||||
rebuttal_store.insert_job(job)
|
||||
n_comments = sum(len(r["comments"]) for r in reviewers)
|
||||
return _ok(job, reviewers=len(reviewers), comments=n_comments)
|
||||
|
||||
|
||||
# ---------- 端点 ----------
|
||||
@router.post("/jobs", response_model=RebuttalResponse)
|
||||
async def create_job(body: JobCreateBody) -> RebuttalResponse:
|
||||
"""文本粘贴主入口。"""
|
||||
return await _build_job(body.text, kind="text", filename=None)
|
||||
|
||||
|
||||
@router.post("/jobs/pdf", response_model=RebuttalResponse)
|
||||
async def create_job_pdf(file: UploadFile = File(...)) -> RebuttalResponse:
|
||||
"""PDF 次要入口（批注版审稿 PDF；经 vstore 8094 解析）。"""
|
||||
settings = get_settings()
|
||||
filename = file.filename or "review.pdf"
|
||||
if not filename.lower().endswith(".pdf"):
|
||||
return _error("VALIDATION", "仅支持 PDF 文件（推荐直接粘贴文本）")
|
||||
data = await file.read()
|
||||
if not data:
|
||||
return _error("VALIDATION", "上传文件为空")
|
||||
if len(data) > settings.max_pdf_bytes:
|
||||
return _error("VALIDATION", f"PDF 超过 {settings.max_pdf_bytes // (1024 * 1024)}MB 上限")
|
||||
try:
|
||||
text = await paper_parse.parse_pdf_bytes(data, filename=filename)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return _error("UPSTREAM_DOWN", f"PDF 解析失败：{exc}")
|
||||
return await _build_job(text, kind="pdf", filename=filename)
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}", response_model=RebuttalResponse)
|
||||
async def get_job(job_id: str) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
return _ok(job)
|
||||
|
||||
|
||||
@router.patch("/jobs/{job_id}/comments/{cid}/response", response_model=RebuttalResponse)
|
||||
async def patch_response(job_id: str, cid: str, body: ResponseBody) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
found = _find_comment(job, cid)
|
||||
if found is None:
|
||||
return _error("NOT_FOUND", f"comment {cid} 不存在")
|
||||
_, comment = found
|
||||
for p in body.parts:
|
||||
if p.type not in PART_TYPES:
|
||||
return _error("VALIDATION", f"未知 part.type: {p.type}")
|
||||
comment["response"] = {"parts": [p.model_dump() for p in body.parts]}
|
||||
# 改回复 → 旧 draft/validation 作废（保守：不保留过期草稿）
|
||||
comment["draft"] = None
|
||||
comment["validation"] = None
|
||||
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
|
||||
return _ok(job)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/comments/{cid}/draft", response_model=RebuttalResponse)
|
||||
async def post_draft(job_id: str, cid: str) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
found = _find_comment(job, cid)
|
||||
if found is None:
|
||||
return _error("NOT_FOUND", f"comment {cid} 不存在")
|
||||
_, comment = found
|
||||
parts = (comment.get("response") or {}).get("parts") or []
|
||||
if not parts or not any(str(p.get("content") or "").strip() for p in parts):
|
||||
# 决策 4：生成前必须填 user_response；LLM 调用次数=0
|
||||
return _error("MISSING_RESPONSE", "请先填写回复要点（user_response）再生成草稿")
|
||||
validation = validate_parts(parts)
|
||||
comment["validation"] = validation
|
||||
if not validation["passed"]:
|
||||
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
|
||||
return RebuttalResponse(
|
||||
ok=False, job=job,
|
||||
error={"code": "VALIDATION_FAILED", "message": "反幻觉校验未通过",
|
||||
"degraded": False},
|
||||
meta={"validation": validation},
|
||||
)
|
||||
comment["draft"] = await generate_draft(_messages_llm, comment)
|
||||
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
|
||||
return _ok(job, validation=validation)
|
||||
|
||||
|
||||
@router.patch("/jobs/{job_id}/comments/{cid}/structure", response_model=RebuttalResponse)
|
||||
async def patch_structure(job_id: str, cid: str, body: StructureBody) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
found = _find_comment(job, cid)
|
||||
if found is None:
|
||||
return _error("NOT_FOUND", f"comment {cid} 不存在")
|
||||
reviewer, comment = found
|
||||
comments = reviewer["comments"]
|
||||
idx = comments.index(comment)
|
||||
if body.action == "merge":
|
||||
ids = body.comment_ids or []
|
||||
if cid not in ids:
|
||||
ids = [cid] + ids
|
||||
if len(ids) < 2:
|
||||
return _error("VALIDATION", "merge 至少需要 2 条 comment")
|
||||
targets = [c for c in comments if c["id"] in ids]
|
||||
if len(targets) < 2:
|
||||
return _error("VALIDATION", "merge 目标中有效 comment 不足 2 条")
|
||||
merged_body = "\n\n".join(c["body"] for c in targets)
|
||||
comment["body"] = merged_body
|
||||
comment["title"] = f"{comment['title']} (+{len(targets) - 1} merged)"
|
||||
reviewer["comments"] = [
|
||||
c for c in comments
|
||||
if c is comment or c["id"] not in ids
|
||||
]
|
||||
else: # split
|
||||
items = body.items or []
|
||||
items = [i for i in items if str(i.get("body") or "").strip()]
|
||||
if len(items) < 2:
|
||||
return _error("VALIDATION", "split 至少需要 2 个非空子项")
|
||||
group_id = comment.get("group_id") or f"g_{cid}"
|
||||
new_comments = []
|
||||
for n, it in enumerate(items, start=1):
|
||||
c = dict(comment)
|
||||
c["id"] = rebuttal_store.new_id(f"{cid}s{n}")
|
||||
c["group_id"] = group_id
|
||||
c["title"] = str(it.get("title") or f"{comment['title']} ({n})")
|
||||
c["body"] = str(it.get("body") or "").strip()
|
||||
c["sub_items"] = []
|
||||
c["response"] = {"parts": []}
|
||||
c["draft"] = None
|
||||
c["validation"] = None
|
||||
c["human_confirmed"] = False
|
||||
new_comments.append(c)
|
||||
comments[idx:idx + 1] = new_comments
|
||||
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
|
||||
return _ok(job)
|
||||
|
||||
|
||||
@router.patch("/jobs/{job_id}/comments/{cid}/confirm", response_model=RebuttalResponse)
|
||||
async def patch_confirm(job_id: str, cid: str, body: ConfirmBody) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
found = _find_comment(job, cid)
|
||||
if found is None:
|
||||
return _error("NOT_FOUND", f"comment {cid} 不存在")
|
||||
_, comment = found
|
||||
comment["human_confirmed"] = bool(body.confirmed)
|
||||
# human_ack 由 store 重算（决策 4：全勾才置位，不信前端）
|
||||
job = rebuttal_store.update_job(job_id, reviewers=job["reviewers"]) or job
|
||||
return _ok(job, human_ack=job["human_ack"])
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/letter", response_model=RebuttalResponse)
|
||||
async def post_letter(job_id: str) -> RebuttalResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
job = rebuttal_store.update_job(job_id, letter_md=assemble_letter(job)) or job
|
||||
return _ok(job)
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}/letter.md")
|
||||
async def download_letter(job_id: str) -> PlainTextResponse:
|
||||
job = rebuttal_store.get_job(job_id)
|
||||
if job is None:
|
||||
return PlainTextResponse(
|
||||
"job not found", status_code=404, media_type="text/plain; charset=utf-8")
|
||||
if not job.get("human_ack"):
|
||||
# 信封口径与下载端点适配：GET 下载返回纯文本，业务拒绝用 400 + 明文
|
||||
return PlainTextResponse(
|
||||
"HUMAN_ACK_REQUIRED: 请逐条勾选确认后再下载 letter.md",
|
||||
status_code=400, media_type="text/plain; charset=utf-8")
|
||||
md = job.get("letter_md") or assemble_letter(job)
|
||||
return PlainTextResponse(md, media_type="text/markdown; charset=utf-8")
|
||||
|
||||
|
||||
@router.delete("/jobs/{job_id}", response_model=RebuttalResponse)
|
||||
async def delete_job(job_id: str) -> RebuttalResponse:
|
||||
if not rebuttal_store.delete_job(job_id):
|
||||
return _error("NOT_FOUND", f"job {job_id} 不存在或已删除")
|
||||
return RebuttalResponse(ok=True, job=None, meta={"deleted": job_id})
|
||||
@@ -1,165 +0,0 @@
|
||||
"""routers/reformat.py —— 论文自动排版 P1（.tex 源码级重排版，不编译）。
|
||||
|
||||
端点：
|
||||
POST /api/review/reformat multipart 上传 .tex + target_journal →
|
||||
修改后源码 + 逐行 diff 报告（job 落盘）
|
||||
GET /api/review/reformat/{id} 取完整报告（重启后仍在）
|
||||
GET /api/review/reformat/{id}/file 下载修改后的 .tex
|
||||
|
||||
信封与其余 review 端点一致：业务错误 = HTTP 200 + {ok:false,error:{code,message,degraded:false}}，
|
||||
前端经 review.js unwrap 归一为 ApiError。本次改动无上游依赖（零 LLM/零 vstore），
|
||||
degraded 恒 false。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, File, Form, UploadFile
|
||||
from fastapi.responses import FileResponse, JSONResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..config import get_settings
|
||||
from ..services.reformat import ReformatError, reformat_tex, resolve_rules
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/review", tags=["review-reformat"])
|
||||
|
||||
_JOB_ID_RE = re.compile(r"^[0-9a-f]{12}$")
|
||||
_ENGINE = "reformat-p1/1.0"
|
||||
|
||||
|
||||
class ReformatResponse(BaseModel):
|
||||
ok: bool
|
||||
job_id: str | None = None
|
||||
modified_tex: str | None = None
|
||||
changes: list[dict[str, Any]] | None = None
|
||||
warnings: list[dict[str, Any]] | None = None
|
||||
meta: dict[str, Any] | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def _error(code: str, message: str) -> ReformatResponse:
|
||||
return ReformatResponse(
|
||||
ok=False,
|
||||
error={"code": code, "message": message, "degraded": False},
|
||||
)
|
||||
|
||||
|
||||
def _jobs_dir() -> Path:
|
||||
d = get_settings().reformat_dir_abs
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
@router.post("/reformat", response_model=ReformatResponse)
|
||||
async def reformat(
|
||||
file: UploadFile = File(..., description="LaTeX 源码（.tex 单文件）"),
|
||||
target_journal: str = Form("", description="目标期刊 id/名称，或 default=通用规则"),
|
||||
) -> ReformatResponse:
|
||||
settings = get_settings()
|
||||
|
||||
# ---- 入参校验（拍板 5：只收 .tex，docx/pdf 走 format-check 审核不重排）----
|
||||
filename = file.filename or "paper.tex"
|
||||
if not filename.lower().endswith(".tex"):
|
||||
return _error("VALIDATION", "仅支持 .tex 文件（docx/pdf 请走格式审核）")
|
||||
target = (target_journal or "").strip()
|
||||
if not target:
|
||||
return _error("VALIDATION", "必须指定目标期刊（或选择「通用规则」）")
|
||||
|
||||
content = await file.read()
|
||||
if not content:
|
||||
return _error("VALIDATION", "上传文件为空")
|
||||
if len(content) > settings.max_tex_bytes:
|
||||
return _error("VALIDATION", "源码超过 2MB 上限")
|
||||
try:
|
||||
source = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return _error("VALIDATION", "源码必须为 UTF-8 编码")
|
||||
|
||||
# ---- 规则解析（journals.json / format_cache / journal_directory 只读回链）----
|
||||
resolved = await resolve_rules(target)
|
||||
|
||||
# ---- 变换（纯规则引擎；结构畸形 → PARSER_FAILED）----
|
||||
try:
|
||||
result = reformat_tex(source, resolved["rules"])
|
||||
except ReformatError as exc:
|
||||
logger.warning("reformat: 结构守卫拒收 %s: %s", filename, exc)
|
||||
return _error(exc.code, str(exc))
|
||||
|
||||
# ---- 落盘（.tex + .json 报告；重启后 GET 仍可用）----
|
||||
job_id = uuid.uuid4().hex[:12]
|
||||
base = _jobs_dir()
|
||||
(base / f"{job_id}.tex").write_text(result["modified_tex"], encoding="utf-8")
|
||||
|
||||
meta: dict[str, Any] = {
|
||||
"journal_id": resolved["journal_id"],
|
||||
"journal_name": resolved["journal_name"],
|
||||
"rules": resolved["rules"],
|
||||
"rules_source": resolved["rules_source"],
|
||||
"rules_note": resolved["rules_note"],
|
||||
"filename": filename,
|
||||
"original_sha256": hashlib.sha256(content).hexdigest(),
|
||||
"engine": _ENGINE,
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||
}
|
||||
payload = {
|
||||
"ok": True,
|
||||
"job_id": job_id,
|
||||
"modified_tex": result["modified_tex"],
|
||||
"changes": result["changes"],
|
||||
"warnings": result["warnings"],
|
||||
"meta": meta,
|
||||
}
|
||||
(base / f"{job_id}.json").write_text(
|
||||
json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
|
||||
)
|
||||
logger.info(
|
||||
"reformat: %s → job %s（%d 处修改, %d 条提示, 规则 %s）",
|
||||
filename, job_id,
|
||||
sum(1 for c in result["changes"] if c["action"] == "modified"),
|
||||
sum(1 for c in result["changes"] if c["action"] == "warning"),
|
||||
resolved["rules_source"],
|
||||
)
|
||||
return ReformatResponse(**payload)
|
||||
|
||||
|
||||
def _load_job(job_id: str) -> dict[str, Any] | None:
|
||||
if not _JOB_ID_RE.fullmatch(job_id or ""):
|
||||
return None
|
||||
path = _jobs_dir() / f"{job_id}.json"
|
||||
try:
|
||||
payload = json.loads(path.read_text("utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
return payload if isinstance(payload, dict) and payload.get("ok") else None
|
||||
|
||||
|
||||
_NOT_FOUND = {"ok": False, "error": {"code": "NOT_FOUND",
|
||||
"message": "任务不存在或已过期，请重新上传源码",
|
||||
"degraded": False}}
|
||||
|
||||
|
||||
@router.get("/reformat/{job_id}", response_model=ReformatResponse)
|
||||
async def get_reformat_job(job_id: str) -> ReformatResponse:
|
||||
payload = _load_job(job_id)
|
||||
if payload is None:
|
||||
return _error("NOT_FOUND", "任务不存在或已过期，请重新上传源码")
|
||||
return ReformatResponse(**payload)
|
||||
|
||||
|
||||
@router.get("/reformat/{job_id}/file", response_model=None)
|
||||
async def download_reformat_file(job_id: str) -> FileResponse | JSONResponse:
|
||||
"""下载修改后的 .tex；缺件/非法 id → 200 信封（前端单一错误口径）。"""
|
||||
valid = bool(_JOB_ID_RE.fullmatch(job_id or ""))
|
||||
tex = _jobs_dir() / f"{job_id}.tex" if valid else None
|
||||
if tex is None or not tex.is_file():
|
||||
return JSONResponse(status_code=200, content=_NOT_FOUND)
|
||||
return FileResponse(tex, media_type="application/x-tex", filename=f"{job_id}.tex")
|
||||
@@ -1 +0,0 @@
|
||||
"""review 服务服务层包。"""
|
||||
@@ -1,788 +0,0 @@
|
||||
"""services/format_check.py —— 版式与规范规则引擎（⑤ 论文排版与规范）。
|
||||
|
||||
纯规则、无 LLM、无 vstore、同步执行（通常 <1s）：
|
||||
- PDF → pymupdf 本地解档（AGENTS.md 约束 6 例外：vstore 只回文本，字体/边距/
|
||||
行距/页数等版式元数据拿不到；正文提取仍走 vstore，本服务不碰正文语义）；
|
||||
- Word → python-docx（页边距/行距精确，页数取 docProps/app.xml 的 <Pages>）；
|
||||
- 规则集 → data/journals/journals.json：
|
||||
顶层 format_rules_default（通用默认值，rules_source="default"）
|
||||
每条目可选 format_rules（按期刊覆盖，必须带 rules_source: placeholder|official|default）
|
||||
⚠ 诚实原则：不知道期刊真实投稿指南 → 该条只写 {"rules_source": "placeholder"}，
|
||||
所有具体数字回退 default 通用值，绝不编造期刊专属数字。
|
||||
|
||||
输出：{checks[{name,key,status,detail,suggestion}], summary{pass,warn,fail}, meta{...}}
|
||||
status ∈ pass|warn|fail。参考文献样式宽容化：仅"明确不符"（如 [J] 标引但规则要求 APA）
|
||||
才 fail，"疑似但不确定/检测不到"一律 warn，避免误伤。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import statistics
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import fitz # pymupdf（venv 已装 1.28.x）
|
||||
from docx import Document
|
||||
from docx.shared import Length
|
||||
|
||||
# review_service/app/services/ → parents[2] = review_service
|
||||
_JOURNALS_PATH = Path(__file__).resolve().parents[2] / "data" / "journals" / "journals.json"
|
||||
|
||||
_MAX_SCAN_PAGES = 80 # 版式扫描页数上限（正文/字体/边距；稿件通常远小于此）
|
||||
_MAX_REFS_SCAN = 40 # 从文末向前找参考文献标题的页数窗口
|
||||
_MAX_PARAS = 8000 # docx 段落扫描上限
|
||||
_PT_TO_MM = 25.4 / 72.0
|
||||
_EMU_TO_MM = 1.0 / 36000.0
|
||||
|
||||
_HEADING_RE = re.compile(
|
||||
r"^\s*(references|bibliography|literature cited|works cited|参考文献|文献)\s*[:：]?\s*$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_FIG_RE = re.compile(r"(?im)^\s*(figure|fig\.?|图)\s*\d+")
|
||||
_TAB_RE = re.compile(r"(?im)^\s*(table|表)\s*\d+")
|
||||
_GB_TAG_RE = re.compile(r"\[[A-Z]{1,3}\d*\]") # [J] [M] [D] [C] …（GB/T 7714 标引）
|
||||
_PAREN_YEAR_RE = re.compile(r"\((?:19|20)\d{2}\)") # (2024) 作者-年份制
|
||||
_BRACKET_NUM_RE = re.compile(r"^\[\d+\]") # [1] 编号制
|
||||
_DOI_RE = re.compile(r"doi:?\.?org|doi:\s*1\.", re.IGNORECASE)
|
||||
|
||||
_STYLE_LABEL = {
|
||||
"gb7714": "GB/T 7714",
|
||||
"author_year": "作者-年份制（类 APA）",
|
||||
"numbered": "编号制（顺序编码）",
|
||||
"unknown": "无法确定",
|
||||
}
|
||||
|
||||
|
||||
class FormatCheckError(Exception):
|
||||
"""解档/规则加载失败（路由器转 PARSE_FAILED）。"""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 规则集加载
|
||||
|
||||
def load_ruleset(target_journal: str) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
"""返回 (rules, ruleset_meta)。target_journal 传 id 或 name 均可；未知 → default。"""
|
||||
try:
|
||||
data = json.loads(_JOURNALS_PATH.read_text("utf-8"))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise FormatCheckError(f"规则集加载失败：{exc}") from exc
|
||||
|
||||
default = data.get("format_rules_default") or {}
|
||||
jid = (target_journal or "").strip()
|
||||
entry = None
|
||||
if jid:
|
||||
for j in data.get("journals") or []:
|
||||
if j.get("id") == jid or j.get("name") == jid:
|
||||
entry = j
|
||||
break
|
||||
|
||||
if entry is None:
|
||||
rules = dict(default)
|
||||
meta = {
|
||||
"journal_id": None,
|
||||
"journal_name": None,
|
||||
"ruleset_source": "default",
|
||||
"rules_source": str(default.get("rules_source", "default")),
|
||||
}
|
||||
else:
|
||||
journal_rules = entry.get("format_rules") or {}
|
||||
rules = {**default, **journal_rules}
|
||||
meta = {
|
||||
"journal_id": entry.get("id"),
|
||||
"journal_name": entry.get("name"),
|
||||
"ruleset_source": "journal",
|
||||
# 占位规则（无官方核验）→ 前端黄字提示
|
||||
"rules_source": str(journal_rules.get("rules_source", "placeholder")),
|
||||
}
|
||||
rules.setdefault("rules_source", "default")
|
||||
return rules, meta
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- PDF 解档（pymupdf）
|
||||
|
||||
def _norm_font(name: str) -> str:
|
||||
n = (name or "").split("+")[-1].strip()
|
||||
return n or "unknown"
|
||||
|
||||
|
||||
def extract_profile_pdf(data: bytes) -> dict[str, Any]:
|
||||
try:
|
||||
doc = fitz.open(stream=data, filetype="pdf")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise FormatCheckError(f"PDF 解档失败：{exc}") from exc
|
||||
if getattr(doc, "needs_pass", False):
|
||||
doc.close()
|
||||
raise FormatCheckError("PDF 已加密，无法解析")
|
||||
|
||||
pages_total = doc.page_count
|
||||
scan = min(pages_total, _MAX_SCAN_PAGES)
|
||||
margin_rows: list[tuple[float, float, float, float]] = [] # t r b l (mm)
|
||||
font_w: dict[str, int] = {}
|
||||
size_w: dict[float, int] = {}
|
||||
ratios: list[float] = []
|
||||
img_ids: set[int] = set()
|
||||
fig_caps = tab_caps = 0
|
||||
|
||||
for pno in range(scan):
|
||||
page = doc[pno]
|
||||
rect = page.rect
|
||||
blocks = page.get_text("blocks")
|
||||
content = [b for b in blocks if b[4].strip()]
|
||||
if content:
|
||||
left = min(b[0] for b in content)
|
||||
top = min(b[1] for b in content)
|
||||
right = rect.width - max(b[2] for b in content)
|
||||
bottom = rect.height - max(b[3] for b in content)
|
||||
margin_rows.append(
|
||||
(
|
||||
max(0.0, top) * _PT_TO_MM,
|
||||
max(0.0, right) * _PT_TO_MM,
|
||||
max(0.0, bottom) * _PT_TO_MM,
|
||||
max(0.0, left) * _PT_TO_MM,
|
||||
)
|
||||
)
|
||||
|
||||
for img in page.get_images(full=True):
|
||||
img_ids.add(img[0])
|
||||
|
||||
text = page.get_text()
|
||||
fig_caps += len(_FIG_RE.findall(text))
|
||||
tab_caps += len(_TAB_RE.findall(text))
|
||||
|
||||
d = page.get_text("dict")
|
||||
for block in d.get("blocks", []):
|
||||
if block.get("type") != 0:
|
||||
continue
|
||||
line_meta: list[tuple[float, float]] = [] # (基线 y, 字号)
|
||||
for line in block.get("lines", []):
|
||||
spans = [s for s in line.get("spans", []) if s.get("text", "").strip()]
|
||||
if not spans:
|
||||
continue
|
||||
row_size = max(s.get("size", 0.0) for s in spans)
|
||||
if row_size <= 0:
|
||||
continue
|
||||
for s in spans:
|
||||
size = round(float(s.get("size", 0.0)), 1)
|
||||
chars = len(s.get("text", ""))
|
||||
if size > 0:
|
||||
norm = _norm_font(s.get("font", ""))
|
||||
font_w[norm] = font_w.get(norm, 0) + chars
|
||||
size_w[size] = size_w.get(size, 0) + chars
|
||||
# 行距基准用 span.origin 基线 y，不用 bbox 顶缘：bbox 顶随字形高低
|
||||
# 漂移，表格/图注等同基线并列文本会产出假性小 delta，混入中位数会把
|
||||
# 行距拉成 0.1x 的不可能值。
|
||||
origin = spans[0].get("origin")
|
||||
line_meta.append((origin[1] if origin else line["bbox"][1], row_size))
|
||||
for i in range(1, len(line_meta)):
|
||||
delta = line_meta[i][0] - line_meta[i - 1][0]
|
||||
size = line_meta[i][1]
|
||||
# 行距倍数 = 相邻行基线 y 差 / 当前行字号；
|
||||
# 过滤物理不可能值：<0.5（行重叠/同基线并列表格）、>4.0（异常结构）
|
||||
ratio = delta / size
|
||||
if 0.5 <= ratio <= 4.0:
|
||||
ratios.append(ratio)
|
||||
|
||||
refs_text, refs_found = _find_refs_pdf(doc, pages_total)
|
||||
doc.close()
|
||||
|
||||
if margin_rows:
|
||||
med = [statistics.median(col) for col in zip(*margin_rows)]
|
||||
margins = {"top": med[0], "right": med[1], "bottom": med[2], "left": med[3]}
|
||||
else:
|
||||
margins = None
|
||||
|
||||
return {
|
||||
"source_kind": "pdf",
|
||||
"pages": pages_total,
|
||||
"margins_mm": margins,
|
||||
"font_weights": font_w,
|
||||
"size_weights": size_w,
|
||||
"line_spacing_median": statistics.median(ratios) if ratios else None,
|
||||
"figures": max(len(img_ids), fig_caps),
|
||||
"tables": tab_caps,
|
||||
"refs_text": refs_text,
|
||||
"refs_found": refs_found,
|
||||
}
|
||||
|
||||
|
||||
def _find_refs_pdf(doc: "fitz.Document", pages_total: int) -> tuple[str | None, bool]:
|
||||
"""从文末向前找参考文献标题行；找到 → 标题行至文末文本；找不到 → (None, False)。"""
|
||||
buf: list[str] = []
|
||||
start = max(0, pages_total - _MAX_REFS_SCAN)
|
||||
for pno in range(pages_total - 1, start - 1, -1):
|
||||
page_text = doc[pno].get_text()
|
||||
lines = page_text.splitlines()
|
||||
for idx in range(len(lines) - 1, -1, -1):
|
||||
if _HEADING_RE.match(lines[idx].strip()):
|
||||
tail = "\n".join(lines[idx + 1:])
|
||||
joined = "\n".join([tail] + buf).strip()
|
||||
return (joined if len(joined) >= 50 else None), bool(joined)
|
||||
buf.insert(0, page_text)
|
||||
return None, False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- Word 解档（python-docx）
|
||||
|
||||
def _docx_app_pages(data: bytes) -> int | None:
|
||||
try:
|
||||
with zipfile.ZipFile(io.BytesIO(data)) as zf:
|
||||
xml = zf.read("docProps/app.xml").decode("utf-8", "ignore")
|
||||
m = re.search(r"<Pages>(\d+)</Pages>", xml)
|
||||
return int(m.group(1)) if m else None
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
|
||||
|
||||
def extract_profile_docx(data: bytes) -> dict[str, Any]:
|
||||
try:
|
||||
doc = Document(io.BytesIO(data))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise FormatCheckError(f"Word 文件解析失败：{exc}") from exc
|
||||
|
||||
side_lists: dict[str, list[float]] = {"top": [], "right": [], "bottom": [], "left": []}
|
||||
for sec in doc.sections:
|
||||
for k, v in (
|
||||
("top", sec.top_margin),
|
||||
("right", sec.right_margin),
|
||||
("bottom", sec.bottom_margin),
|
||||
("left", sec.left_margin),
|
||||
):
|
||||
if v is not None:
|
||||
side_lists[k].append(float(v) * _EMU_TO_MM)
|
||||
margins = None
|
||||
if all(side_lists.values()):
|
||||
margins = {k: statistics.median(v) for k, v in side_lists.items()}
|
||||
|
||||
try:
|
||||
normal = doc.styles["Normal"].font
|
||||
default_name = normal.name
|
||||
default_size = float(normal.size.pt) if normal.size is not None else None
|
||||
except Exception: # noqa: BLE001
|
||||
default_name, default_size = None, None
|
||||
|
||||
font_w: dict[str, int] = {}
|
||||
size_w: dict[float, int] = {}
|
||||
ratios: list[float] = []
|
||||
fig_caps = tab_caps = 0
|
||||
paras = doc.paragraphs
|
||||
scan_count = min(len(paras), _MAX_PARAS)
|
||||
|
||||
for i in range(scan_count):
|
||||
para = paras[i]
|
||||
text = para.text
|
||||
if not text.strip():
|
||||
continue
|
||||
ls = para.paragraph_format.line_spacing
|
||||
if isinstance(ls, Length):
|
||||
# 精确行距（exactly）以 EMU 存储，Length 是 int 子类——直接当倍数会得出
|
||||
# 十万级的假值；用正文默认字号换算成倍数
|
||||
if default_size:
|
||||
r = float(ls.pt) / default_size
|
||||
if 0.5 <= r <= 4.0:
|
||||
ratios.append(r)
|
||||
elif isinstance(ls, (int, float)) and not isinstance(ls, bool) and ls > 0:
|
||||
r = float(ls) # multiple 行距本身就是倍数
|
||||
if 0.5 <= r <= 4.0:
|
||||
ratios.append(r)
|
||||
if _FIG_RE.match(text):
|
||||
fig_caps += 1
|
||||
elif _TAB_RE.match(text):
|
||||
tab_caps += 1
|
||||
for run in para.runs:
|
||||
if not run.text:
|
||||
continue
|
||||
name = run.font.name or default_name or "unknown"
|
||||
pt = run.font.size.pt if run.font.size is not None else default_size
|
||||
chars = len(run.text)
|
||||
font_w[str(name)] = font_w.get(str(name), 0) + chars
|
||||
if pt:
|
||||
size_w[round(float(pt), 1)] = size_w.get(round(float(pt), 1), 0) + chars
|
||||
|
||||
# 参考文献节：标题段落之后全部文本
|
||||
refs_text, refs_found = None, False
|
||||
for i, para in enumerate(paras):
|
||||
if _HEADING_RE.match((para.text or "").strip()):
|
||||
tail = "\n".join(p.text for p in paras[i + 1:] if (p.text or "").strip())
|
||||
if len(tail) >= 50:
|
||||
refs_text, refs_found = tail, True
|
||||
break
|
||||
|
||||
return {
|
||||
"source_kind": "docx",
|
||||
"pages": _docx_app_pages(data), # Word 未记录 → None（检查项降级为 warn）
|
||||
"margins_mm": margins,
|
||||
"font_weights": font_w,
|
||||
"size_weights": size_w,
|
||||
"line_spacing_median": statistics.median(ratios) if ratios else None,
|
||||
"figures": max(len(doc.inline_shapes), fig_caps),
|
||||
"tables": max(len(doc.tables), tab_caps),
|
||||
"refs_text": refs_text,
|
||||
"refs_found": refs_found,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- LaTeX 解档（正则+状态机，零新依赖）
|
||||
#
|
||||
# 决策前提：只做"解析 + 检查 + 建议代码片段"，不自动改写 LaTeX（用户可能有自定义宏，
|
||||
# 自动改写会引错；LaTeX 应由用户自己编辑）。不引入 pylatexenc/texsoup。
|
||||
|
||||
# 会相互覆盖的字体类宏包：≥2 个同时引入 → warn（times + mathptmx 类冲突）
|
||||
_TEX_FONT_PKGS = {"times", "mathptmx", "newtx", "newpx", "txfonts", "pxfonts", "mathpazo"}
|
||||
|
||||
|
||||
def _strip_tex_comments(src: str) -> str:
|
||||
r"""去行内 % 注释（r-string，避免 \% 被解释为转义序列）。"""
|
||||
return re.sub(r"(?<!\\)%.*", "", src)
|
||||
|
||||
|
||||
def _parse_geometry(src: str) -> dict[str, float] | None:
|
||||
"""\\usepackage[...]{geometry} 或 \\geometry{...} → {边: mm}（可缺边，检查端容缺）。"""
|
||||
m = re.search(r"\\usepackage\s*\[([^\]]*)\]\s*\{\s*geometry\s*\}", src)
|
||||
if not m:
|
||||
m = re.search(r"\\geometry\s*\{([^}]*)\}", src)
|
||||
if not m:
|
||||
return None
|
||||
opts = m.group(1)
|
||||
|
||||
def to_mm(val: str) -> float | None:
|
||||
mm = re.match(r"\s*([\d.]+)\s*(mm|cm|in|pt)?\s*$", val)
|
||||
if not mm:
|
||||
return None
|
||||
x = float(mm.group(1))
|
||||
unit = mm.group(2) or "pt" # geometry 裸数字按 TeX 惯例为 pt
|
||||
return {"mm": x, "cm": x * 10, "in": x * 25.4, "pt": x * 25.4 / 72.0}[unit]
|
||||
|
||||
kv = {}
|
||||
for pair in opts.split(","):
|
||||
if "=" in pair:
|
||||
k, v = pair.split("=", 1)
|
||||
kv[k.strip().lower()] = v.strip()
|
||||
out: dict[str, float] = {}
|
||||
if "margin" in kv:
|
||||
v = to_mm(kv["margin"])
|
||||
if v is not None:
|
||||
return {"top": v, "right": v, "bottom": v, "left": v}
|
||||
for k in ("top", "right", "bottom", "left"):
|
||||
if k in kv:
|
||||
v = to_mm(kv[k])
|
||||
if v is not None:
|
||||
out[k] = v
|
||||
if "inner" in kv:
|
||||
v = to_mm(kv["inner"])
|
||||
if v is not None:
|
||||
out.setdefault("left", v)
|
||||
if "outer" in kv:
|
||||
v = to_mm(kv["outer"])
|
||||
if v is not None:
|
||||
out.setdefault("right", v)
|
||||
return out or None
|
||||
|
||||
|
||||
def extract_tex_profile(data: bytes) -> dict[str, Any]:
|
||||
"""正则 + 轻量状态机解析 .tex 源文件 → 与 DocProfile 同构（+tex 扩展字段）。"""
|
||||
try:
|
||||
text = data.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
text = data.decode("latin-1", errors="replace")
|
||||
src = _strip_tex_comments(text)
|
||||
|
||||
# \documentclass[10pt,a4paper]{article}
|
||||
m_cls = re.search(r"\\documentclass\s*(?:\[[^\]]*\])?\s*\{\s*([\w*]+)\s*\}", src)
|
||||
cls = m_cls.group(1) if m_cls else None
|
||||
m_opt = re.search(r"\\documentclass\s*\[([^\]]*)\]", src)
|
||||
cls_opts = m_opt.group(1) if m_opt else ""
|
||||
m_pt = re.search(r"(\d+)\s*pt", cls_opts)
|
||||
# 标准类未写选项时 LaTeX 默认 10pt；无 documentclass → None（检查项降级 warn）
|
||||
fontsize = int(m_pt.group(1)) if m_pt else (10 if cls else None)
|
||||
paper = "a4paper" if "a4paper" in cls_opts else ("letterpaper" if "letterpaper" in cls_opts else None)
|
||||
|
||||
# \usepackage 列表（支持 {a,b} 逗号多包）
|
||||
packages: list[str] = []
|
||||
for m in re.finditer(r"\\usepackage\s*(?:\[[^\]]*\])?\s*\{\s*([^}]+?)\s*\}", src):
|
||||
packages += [p.strip().lower() for p in m.group(1).split(",") if p.strip()]
|
||||
|
||||
# 行距：\linespread{...} 或 \setstretch{...}（值直接按"行距倍数"口径评估——
|
||||
# 按步骤4 规格：低于 min_line_spacing 即 fail；LaTeX 实际倍数还含 baselinestretch
|
||||
# 的字号比，本引擎取显式设置值做保守判定）
|
||||
line_spread: float | None = None
|
||||
m = re.search(r"\\linespread\s*\{\s*([\d.]+)\s*\}", src)
|
||||
if not m:
|
||||
m = re.search(r"\\setstretch\s*\{\s*([\d.]+)\s*\}", src)
|
||||
if m:
|
||||
line_spread = float(m.group(1))
|
||||
|
||||
# 页边距（geometry）
|
||||
margins = _parse_geometry(src)
|
||||
|
||||
# 计数
|
||||
figures = len(re.findall(r"\\begin\s*\{\s*figure\*?\s*\}", src))
|
||||
tables = len(re.findall(r"\\begin\s*\{\s*table\*?\s*\}", src))
|
||||
cites = len(re.findall(r"\\cite\w*\s*\{", src))
|
||||
|
||||
# 参考文献：\bibliography{...}（BibTeX）/ \printbibliography（biblatex）/ thebibliography
|
||||
has_bibtex = bool(re.search(r"\\bibliography\s*\{", src))
|
||||
has_printbib = bool(re.search(r"\\printbibliography", src))
|
||||
has_thebib = bool(re.search(r"\\begin\s*\{\s*thebibliography\s*\}", src))
|
||||
has_bibstyle = bool(re.search(r"\\bibliographystyle\s*\{", src)) # BibTeX 必需；biblatex 不需要
|
||||
has_bib = has_bibtex or has_printbib or has_thebib
|
||||
refs_text: str | None = None
|
||||
m_bib = re.search(r"\\begin\s*\{\s*thebibliography\s*\}[\s\S]*?\\end\s*\{\s*thebibliography\s*\}", src)
|
||||
if m_bib:
|
||||
refs_text = m_bib.group(0)
|
||||
|
||||
# 宏表（\newcommand{\xxx} / \renewcommand）——用于解析准确性（识别自定义宏）
|
||||
macros = re.findall(r"\\(?:new|re)command\s*\{\s*\\(\w+)\s*\}", src)
|
||||
|
||||
# 匿名评审命令探测（规则集 anonymous:true 时检查）
|
||||
m_anon = re.search(r"\\(?:anonymi[sz]e|anon\b|hideauthor|ifdoubleblind)", src, re.IGNORECASE)
|
||||
|
||||
return {
|
||||
"source_kind": "tex",
|
||||
"pages": None, # LaTeX 源文件页数需编译后才有 → 检查项降级 warn（宽容化）
|
||||
"margins_mm": margins,
|
||||
"font_weights": {},
|
||||
"size_weights": {float(fontsize): 100} if fontsize else {},
|
||||
"line_spacing_median": line_spread,
|
||||
"figures": figures,
|
||||
"tables": tables,
|
||||
"refs_text": refs_text,
|
||||
"refs_found": has_bib,
|
||||
"tex": {
|
||||
"class": cls,
|
||||
"fontsize_pt": fontsize,
|
||||
"paper": paper,
|
||||
"packages": packages,
|
||||
"cites": cites,
|
||||
"macros": macros,
|
||||
"anon_cmd": m_anon.group(0) if m_anon else None,
|
||||
"bib_bibtex": has_bibtex,
|
||||
"bib_style": has_bibstyle,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 参考文献样式检测
|
||||
|
||||
def detect_refs_style(text: str | None) -> tuple[str, str]:
|
||||
"""宽容化检测：返回 (detected, sample_line)。含糊 → unknown（后续只 warn）。"""
|
||||
if not text:
|
||||
return "unknown", ""
|
||||
lines = [l.strip() for l in text.splitlines() if l.strip()][:40]
|
||||
if not lines:
|
||||
return "unknown", ""
|
||||
n = len(lines)
|
||||
gb = sum(1 for l in lines if _GB_TAG_RE.search(l))
|
||||
year = sum(1 for l in lines if _PAREN_YEAR_RE.search(l))
|
||||
num = sum(1 for l in lines if _BRACKET_NUM_RE.match(l))
|
||||
doi = sum(1 for l in lines if _DOI_RE.search(l))
|
||||
sample = lines[0][:80]
|
||||
|
||||
if gb / n >= 0.4:
|
||||
return "gb7714", sample
|
||||
if num / n >= 0.4 and year / n < 0.3:
|
||||
return "numbered", sample
|
||||
if year / n >= 0.3 or doi / n >= 0.3:
|
||||
return "author_year", sample
|
||||
return "unknown", sample
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 检查项构建
|
||||
|
||||
def _chk(key: str, name: str, status: str, detail: str, suggestion: str = "") -> dict[str, str]:
|
||||
return {"key": key, "name": name, "status": status, "detail": detail, "suggestion": suggestion}
|
||||
|
||||
|
||||
def _check_pages(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
limit = int(rules.get("max_pages", 10))
|
||||
pages = profile.get("pages")
|
||||
kind = profile.get("source_kind") or "doc"
|
||||
if pages is None:
|
||||
# 宽容化：Word/LaTeX 页数缺失绝不 fail
|
||||
if kind == "tex":
|
||||
return _chk(
|
||||
"pages", "页数", "warn",
|
||||
f"LaTeX 源无法直接统计页数（规则上限 {limit} 页），需编译后核对",
|
||||
"编译后确认（如 xelatex main.tex）总页数 ≤ 上限",
|
||||
)
|
||||
return _chk(
|
||||
"pages", "页数", "warn",
|
||||
"Word 未记录页数，请自行核对",
|
||||
f"在 Word 中按 Ctrl+End 查看总页数，或导出 PDF 后重查（规则上限 {limit} 页）",
|
||||
)
|
||||
if pages <= limit:
|
||||
return _chk("pages", "页数", "pass", f"共 {pages} 页，规则上限 {limit} 页")
|
||||
return _chk(
|
||||
"pages", "页数", "fail", f"共 {pages} 页，超出规则上限 {limit} 页",
|
||||
f"压缩至 ≤{limit} 页，或将方法/图表移入补充材料",
|
||||
)
|
||||
|
||||
|
||||
def _check_fonts(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
min_font = float(rules.get("min_font_pt", 9))
|
||||
# LaTeX：字号来自 \documentclass 选项（10/11/12pt，粒度粗）
|
||||
if profile.get("source_kind") == "tex":
|
||||
tex = profile.get("tex") or {}
|
||||
cls = tex.get("class") or "documentclass"
|
||||
fs = tex.get("fontsize_pt")
|
||||
pkgs = tex.get("packages") or []
|
||||
font_pkgs = sorted(p for p in pkgs if p in _TEX_FONT_PKGS or p in ("helvet", "courier", "charter", "fourier"))
|
||||
detail = (
|
||||
f"\\documentclass 选项字号 {fs}pt（类 {cls}）；"
|
||||
f"字体宏包：{'、'.join(font_pkgs) if font_pkgs else '未引入'}"
|
||||
+ (f"；纸张 {tex['paper']}" if tex.get("paper") else "")
|
||||
)
|
||||
if fs is None:
|
||||
return _chk("fonts", "字体字号", "warn", detail + "\n建议：\n \\documentclass[11pt]{article}", "补充 \\documentclass 声明")
|
||||
if fs < min_font:
|
||||
# LaTeX 粒度粗 → 按步骤4 规格降为 warn（非 fail）
|
||||
return _chk(
|
||||
"fonts", "字体字号", "warn", detail + f"\n建议：\n \\documentclass[11pt]{{{cls}}}",
|
||||
f"正文字号低于下限 {min_font:.0f}pt，请改用 11pt/12pt 选项",
|
||||
)
|
||||
return _chk("fonts", "字体字号", "pass", detail)
|
||||
|
||||
size_w: dict[float, int] = profile.get("size_weights") or {}
|
||||
font_w: dict[str, int] = profile.get("font_weights") or {}
|
||||
if not size_w:
|
||||
return _chk("fonts", "字体字号", "warn", "未能提取字体信息", "确认文档含可选中文本（扫描件无法检查字体）")
|
||||
min_font = float(rules.get("min_font_pt", 9))
|
||||
max_fonts = int(rules.get("max_fonts", 6))
|
||||
total = sum(size_w.values())
|
||||
body_pt = max(size_w, key=size_w.get)
|
||||
body_ratio = int(size_w[body_pt] / total * 100)
|
||||
distinct = len(font_w)
|
||||
top_fonts = sorted(font_w.items(), key=lambda kv: -kv[1])[:3]
|
||||
top_text = "、".join(f"{k}({v}字)" for k, v in top_fonts)
|
||||
detail = f"主要字号 {body_pt}pt（约 {body_ratio}%），共 {distinct} 种字体；高频字体：{top_text}"
|
||||
if body_pt < min_font:
|
||||
return _chk(
|
||||
"fonts", "字体字号", "fail", detail,
|
||||
f"将正文字号提高至 ≥{min_font}pt（当前主字号 {body_pt}pt）",
|
||||
)
|
||||
if distinct > max_fonts:
|
||||
return _chk(
|
||||
"fonts", "字体字号", "warn", detail,
|
||||
f"字体种类偏多（{distinct} > {max_fonts}），建议全文统一为 1–2 种正文字体",
|
||||
)
|
||||
return _chk("fonts", "字体字号", "pass", detail)
|
||||
|
||||
|
||||
def _check_margins(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
limit = float(rules.get("min_margin_mm", 15))
|
||||
margins = profile.get("margins_mm")
|
||||
# LaTeX：页边距来自 geometry；低于下限按步骤4 规格给 warn（非 fail），附建议片段
|
||||
if profile.get("source_kind") == "tex":
|
||||
if not margins:
|
||||
return _chk(
|
||||
"margins", "页边距", "warn",
|
||||
"未通过 geometry 设置页边距（按类默认评估）"
|
||||
+ f"\n建议：\n \\usepackage[margin=25mm]{{geometry}}",
|
||||
"在导言区显式声明页边距",
|
||||
)
|
||||
lo = min(margins.values())
|
||||
detail = f"geometry 页边距最小 {lo:.0f}mm（" + "、".join(f"{k}{v:.0f}" for k, v in margins.items()) + f"），规则下限 {limit:.0f}mm"
|
||||
if lo < limit:
|
||||
return _chk(
|
||||
"margins", "页边距", "warn",
|
||||
detail + f"\n建议：\n \\usepackage[margin={limit:.0f}mm]{{geometry}}",
|
||||
f"调大页边距至 ≥{limit:.0f}mm",
|
||||
)
|
||||
return _chk("margins", "页边距", "pass", detail)
|
||||
if not margins:
|
||||
return _chk("margins", "页边距", "warn", "未能提取页边距信息", "检查文档是否含可定位文本")
|
||||
vals = [margins.get(k) for k in ("top", "right", "bottom", "left")]
|
||||
vals = [v for v in vals if v is not None]
|
||||
if not vals:
|
||||
return _chk("margins", "页边距", "warn", "未能提取页边距信息", "检查文档是否含可定位文本")
|
||||
t, r = margins.get("top"), margins.get("right")
|
||||
b, l = margins.get("bottom"), margins.get("left")
|
||||
lo = min(vals)
|
||||
sides = "、".join(
|
||||
f"{name}{v:.0f}" for name, v in (("上", t), ("右", r), ("下", b), ("左", l)) if v is not None
|
||||
)
|
||||
detail = f"四边最小 {lo:.0f}mm（{sides}），规则下限 {limit:.0f}mm"
|
||||
if lo >= limit:
|
||||
return _chk("margins", "页边距", "pass", detail)
|
||||
if lo >= limit * 0.8:
|
||||
return _chk("margins", "页边距", "warn", detail, f"将页边距调大至 ≥{limit:.0f}mm")
|
||||
return _chk("margins", "页边距", "fail", detail, f"页边距过小，请调至 ≥{limit:.0f}mm（版式设置）")
|
||||
|
||||
|
||||
def _check_spacing(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
lo = float(rules.get("min_line_spacing", 1.15))
|
||||
ratio = profile.get("line_spacing_median")
|
||||
# LaTeX：\linespread/\setstretch 显式设置值；低于下限按步骤4 规格直接 fail
|
||||
if profile.get("source_kind") == "tex":
|
||||
if ratio is None:
|
||||
return _chk(
|
||||
"spacing", "行距", "pass",
|
||||
"未显式设置 \\linespread/\\setstretch，LaTeX 类默认约 1.2 倍（如 10pt→12pt baselineskip）",
|
||||
)
|
||||
detail = f"\\linespread/\\setstretch = {ratio:.2f}，规则下限 {lo:.2f}"
|
||||
if ratio < lo:
|
||||
return _chk(
|
||||
"spacing", "行距", "fail",
|
||||
detail + f"\n建议：\n \\linespread{{{lo:.2f}}} % 或 \\setstretch{{{lo:.2f}}}（导言区）",
|
||||
"在导言区调大行距设置",
|
||||
)
|
||||
return _chk("spacing", "行距", "pass", detail)
|
||||
if ratio is None:
|
||||
return _chk("spacing", "行距", "warn", "未能提取行距信息", "确认段落使用固定/多倍行距设置")
|
||||
lo = float(rules.get("min_line_spacing", 1.15))
|
||||
detail = f"正文中位行距约 {ratio:.2f} 倍，规则下限 {lo:.2f} 倍"
|
||||
if ratio >= lo:
|
||||
return _chk("spacing", "行距", "pass", detail)
|
||||
if ratio >= lo * 0.9:
|
||||
return _chk("spacing", "行距", "warn", detail, f"段落行距设为 ≥{lo:.2f} 倍（如 1.5 倍行距）")
|
||||
return _chk("spacing", "行距", "fail", detail, f"行距过密，请设为 ≥{lo:.2f} 倍行距")
|
||||
|
||||
|
||||
def _check_figures(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
figures = int(profile.get("figures") or 0)
|
||||
tables = int(profile.get("tables") or 0)
|
||||
min_f = int(rules.get("min_figures", 1))
|
||||
detail = f"检测到图 {figures} 张、表 {tables} 个（图注/表注正则 + 内嵌对象启发式）"
|
||||
if figures >= min_f:
|
||||
return _chk("figures", "图表数量", "pass", detail)
|
||||
# 启发式检测不可靠 → 低于阈值只 warn 不 fail
|
||||
return _chk(
|
||||
"figures", "图表数量", "warn", detail,
|
||||
f"常规研究论文建议 ≥{min_f} 图；若为综述/短文可忽略（本项为启发式检测）",
|
||||
)
|
||||
|
||||
|
||||
# 仅"明确不符"才 fail 的 (expected, detected) 组合（宽容化：其余含糊组合一律 warn）
|
||||
_REF_FAIL_PAIRS = {
|
||||
("apa", "gb7714"), # 找到 [J] 标引但规则要求 APA
|
||||
("gb7714", "author_year"), # 无 [J] 标引的作者-年份制，但规则要求 GB/T 7714
|
||||
("apa", "numbered"), # 编号制引文但规则要求 APA
|
||||
}
|
||||
|
||||
|
||||
def _check_refs(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
expected = str(rules.get("refs_style", "apa")).lower()
|
||||
exp_label = _STYLE_LABEL.get(expected, expected)
|
||||
if not profile.get("refs_found"):
|
||||
return _chk(
|
||||
"refs", "参考文献格式", "warn",
|
||||
"未找到参考文献节（References/参考文献 / \\bibliography / thebibliography）",
|
||||
"确认存在参考文献声明后重查",
|
||||
)
|
||||
if profile.get("source_kind") == "tex":
|
||||
tex = profile.get("tex") or {}
|
||||
# BibTeX：\bibliography 存在但缺 \bibliographystyle → warn（样式引擎未知）
|
||||
if tex.get("bib_bibtex") and not tex.get("bib_style"):
|
||||
return _chk(
|
||||
"refs", "参考文献格式", "warn",
|
||||
f"规则要求 {exp_label}；检测到 \\bibliography 但缺少 \\bibliographystyle（BibTeX 无法确定引用样式）"
|
||||
+ "\n建议：\n \\bibliographystyle{plain} % 按期刊/模板指定实际样式",
|
||||
"在 \\bibliography 前补 \\bibliographystyle",
|
||||
)
|
||||
if not profile.get("refs_text"):
|
||||
# 条目在 .bbl/编译产物，源文件无文本 → 宽容化只 warn
|
||||
return _chk(
|
||||
"refs", "参考文献格式", "warn",
|
||||
f"规则要求 {exp_label}；检测到参考文献声明（条目由编译生成，源文件无条目文本）",
|
||||
"编译后核对参考文献条目样式",
|
||||
)
|
||||
detected, sample = detect_refs_style(profile.get("refs_text"))
|
||||
det_label = _STYLE_LABEL[detected]
|
||||
detail = f"规则要求 {exp_label}；检测到 {det_label}；样例：{sample}"
|
||||
# 宽容化：仅"明确不符"（_REF_FAIL_PAIRS）才 fail；含糊/不确定一律 warn
|
||||
if (expected, detected) in _REF_FAIL_PAIRS:
|
||||
return _chk("refs", "参考文献格式", "fail", detail, f"按 {exp_label} 调整全部文献条目（可用文献管理工具一键转换）")
|
||||
if detected == "unknown":
|
||||
return _chk("refs", "参考文献格式", "warn", detail, f"请人工核对是否符合 {exp_label}")
|
||||
if expected == "gb7714" and detected == "numbered":
|
||||
return _chk("refs", "参考文献格式", "warn", detail, "GB/T 7714 顺序编码制也用编号，需人工确认条目含 [J] 等类型标引")
|
||||
if expected == "author-year-journal" and detected in ("gb7714", "numbered"):
|
||||
return _chk("refs", "参考文献格式", "warn", detail, f"请对照期刊投稿指南核对是否符合 {exp_label}")
|
||||
return _chk("refs", "参考文献格式", "pass", detail)
|
||||
|
||||
|
||||
def _check_font_packages(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
"""LaTeX 特有：多个相互覆盖的字体宏包（times + mathptmx 类）→ warn。"""
|
||||
pkgs = set((profile.get("tex") or {}).get("packages") or [])
|
||||
clash = sorted(pkgs & _TEX_FONT_PKGS)
|
||||
if len(clash) >= 2:
|
||||
detail = (
|
||||
f"检测到多个相互覆盖的字体宏包：{'、'.join(clash)}（后加载者覆盖前者，易产生冲突）"
|
||||
+ "\n建议：\n %只保留一套字体方案，例如：\n \\usepackage{newtx} % newtxtext + newtxmath 一体"
|
||||
)
|
||||
return _chk("fontpkgs", "字体包兼容", "warn", detail, "移除重复/冲突的字体宏包，只保留一套")
|
||||
return _chk(
|
||||
"fontpkgs", "字体包兼容", "pass",
|
||||
"字体宏包：" + ("、".join(clash) if clash else "未引入冲突类字体包"),
|
||||
)
|
||||
|
||||
|
||||
def _check_anonymous(profile: dict[str, Any], rules: dict[str, Any]) -> dict[str, str]:
|
||||
"""LaTeX 特有且规则集 anonymous:true 时才出：源文件需含匿名处理命令。"""
|
||||
tex = profile.get("tex") or {}
|
||||
cmd = tex.get("anon_cmd")
|
||||
if cmd:
|
||||
return _chk("anonymous", "匿名评审", "pass", f"检测到匿名处理命令 {cmd}")
|
||||
return _chk(
|
||||
"anonymous", "匿名评审", "warn",
|
||||
"规则要求匿名评审，源文件未检测到匿名处理命令"
|
||||
+ "\n建议：\n %隐去作者/致谢/自引，或定义匿名宏：\n \\newcommand{\\anon}[1]{[ANON]}",
|
||||
"按期刊要求对作者信息做匿名处理",
|
||||
)
|
||||
|
||||
|
||||
def build_checks(profile: dict[str, Any], rules: dict[str, Any]) -> list[dict[str, str]]:
|
||||
checks = [
|
||||
_check_pages(profile, rules),
|
||||
_check_fonts(profile, rules),
|
||||
_check_margins(profile, rules),
|
||||
_check_spacing(profile, rules),
|
||||
]
|
||||
if profile.get("source_kind") == "tex":
|
||||
checks.append(_check_font_packages(profile, rules))
|
||||
checks.append(_check_figures(profile, rules))
|
||||
checks.append(_check_refs(profile, rules))
|
||||
if profile.get("source_kind") == "tex" and rules.get("anonymous"):
|
||||
checks.append(_check_anonymous(profile, rules))
|
||||
return checks
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 入口
|
||||
|
||||
def check_document(data: bytes, kind: str, target_journal: str) -> dict[str, Any]:
|
||||
"""kind ∈ pdf|docx|tex。返回 {checks, summary, meta}（filename 由路由器补入 meta）。"""
|
||||
if kind == "pdf":
|
||||
profile = extract_profile_pdf(data)
|
||||
elif kind == "docx":
|
||||
profile = extract_profile_docx(data)
|
||||
elif kind == "tex":
|
||||
profile = extract_tex_profile(data)
|
||||
else:
|
||||
raise FormatCheckError(f"不支持的文件类型：{kind}")
|
||||
|
||||
rules, ruleset_meta = load_ruleset(target_journal)
|
||||
checks = build_checks(profile, rules)
|
||||
summary = {"pass": 0, "warn": 0, "fail": 0}
|
||||
for c in checks:
|
||||
summary[c["status"]] = summary.get(c["status"], 0) + 1
|
||||
|
||||
detected, _ = detect_refs_style(profile.get("refs_text")) if profile.get("refs_found") else ("unknown", "")
|
||||
meta = {
|
||||
**ruleset_meta,
|
||||
"file_type": kind,
|
||||
"pages": profile.get("pages"),
|
||||
"refs_style_expected": str(rules.get("refs_style", "apa")).lower(),
|
||||
"refs_style_detected": detected,
|
||||
}
|
||||
if kind == "tex":
|
||||
tex = profile.get("tex") or {}
|
||||
meta["tex"] = {
|
||||
"class": tex.get("class"),
|
||||
"fontsize_pt": tex.get("fontsize_pt"),
|
||||
"paper": tex.get("paper"),
|
||||
"packages": tex.get("packages"),
|
||||
"cites": tex.get("cites"),
|
||||
"macros": (tex.get("macros") or [])[:10],
|
||||
}
|
||||
return {"checks": checks, "summary": summary, "meta": meta}
|
||||
@@ -1,232 +0,0 @@
|
||||
"""services/format_extractor.py —— 投稿指南 → 格式规则 JSON（LLM 抽取，第一批）。
|
||||
|
||||
- 调 services/llm.py 的 complete_json（DeepSeek json_object + 修复重试，零新依赖）
|
||||
- Prompt = 「格式规则抽取器」模板：目标字段与 journals.json format_rules_default
|
||||
同构（max_pages / min_font_pt / min_margin_mm / min_line_spacing / min_figures /
|
||||
max_fonts / refs_style / anonymous），强制每字段带 source_quotes + confidence
|
||||
- 诚实原则：正文没写 → null，绝不编造（与 format_check 诚实原则一致）
|
||||
- 解析/校验失败 → {"ok": False, "error": ...}
|
||||
|
||||
返回：
|
||||
成功 {"ok": True, "rules": {...}, "source_quotes": {...},
|
||||
"confidence": {...}, "notes": str, "raw": LLM原始JSON, "error": None}
|
||||
失败 {"ok": False, "error": "...", 其余键为 None}
|
||||
source_url 由调用方（scripts/fetch_format_rules.py）在落盘时补入结果文件。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from .llm import LLMError, complete_json
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 「格式规则抽取器」模板（system prompt）
|
||||
_SYSTEM = """你是「格式规则抽取器」：从期刊官方投稿指南的 Markdown 正文中抽取格式规则，供投稿排版检查引擎使用。
|
||||
|
||||
# 目标字段（rules 对象，严格用以下字段名）
|
||||
- max_pages：正文页数上限（整数；原文无明确上限 → null）
|
||||
- min_font_pt：最小字号（数字，单位 pt；原文若用 cm/mm/inch 给出则换算为 pt）
|
||||
- min_margin_mm：最小页边距（数字，单位 mm；原文若用 inch 给出则 ×25.4 换算）
|
||||
- min_line_spacing：最小行距倍数（数字；如 "double-spaced" → 2.0，"1.5 spacing" → 1.5）
|
||||
- min_figures：研究论文最少图数（整数；无明确要求 → null）
|
||||
- max_fonts：全文最多字体种类（整数；无明确要求 → null）
|
||||
- refs_style：参考文献样式，枚举 apa | gb7714 | author_year | numbered | vancouver | mla | chicago | other（无法判断 → null）
|
||||
- anonymous：是否要求匿名/双盲评审（true / false / null）
|
||||
|
||||
# 凭证（强制）
|
||||
- 每个非 null 字段必须给 source_quotes[字段名]：从正文【逐字】摘录的 1-3 句原文，
|
||||
不改写、不翻译、不丢掉含关键数字的句子。
|
||||
- source_quotes["_general"]：整体最有凭证价值的 1-3 条原文摘录。
|
||||
- confidence：{"overall": 0-1, "per_field": {"字段名": 0-1}}。
|
||||
直接原文命中 ≥0.9；单位换算 0.6-0.9；靠上下文推断 <0.6。
|
||||
|
||||
# 诚实原则
|
||||
- 正文没写到的字段一律 null，绝不用常识或训练记忆补值。
|
||||
- 单位换算允许做，但该字段 confidence 不得超过 0.9。
|
||||
- 换算过程、拿不准的点、页面疑似不完整，写进 notes。
|
||||
|
||||
# 输出
|
||||
只输出一个 JSON 对象，不要代码块、不要解释文字：
|
||||
{"rules": {"max_pages": null, "min_font_pt": null, "min_margin_mm": null,
|
||||
"min_line_spacing": null, "min_figures": null, "max_fonts": null,
|
||||
"refs_style": null, "anonymous": null},
|
||||
"source_quotes": {"字段名": ["原文1", "..."], "_general": ["原文1", "..."]},
|
||||
"confidence": {"overall": 0.0, "per_field": {"字段名": 0.0}},
|
||||
"notes": "一句话说明抽取依据与不确定点"}"""
|
||||
|
||||
_USER_TMPL = """期刊：{journal_name}
|
||||
|
||||
=== 官方投稿指南正文（Markdown，已截断） ===
|
||||
{markdown}"""
|
||||
|
||||
_RULE_INT_KEYS = ("max_pages", "min_figures", "max_fonts")
|
||||
_REFS_ALLOWED = {"apa", "gb7714", "author_year", "numbered", "vancouver",
|
||||
"mla", "chicago", "other"}
|
||||
_REFS_ALIAS = {
|
||||
"apa-7": "apa", "apa7": "apa", "apa 7th": "apa", "apa_7th": "apa",
|
||||
"harvard": "author_year", "author-year": "author_year", "authoryear": "author_year",
|
||||
"gb/t 7714": "gb7714", "gb/t-7714": "gb7714", "gb7714-2015": "gb7714",
|
||||
"gb/t7714": "gb7714", "chicago author-date": "chicago",
|
||||
}
|
||||
|
||||
|
||||
# 字符串值带显式单位时的目标单位换算表（LLM 若回传原文单位的兜底；无单位=数值即目标单位）
|
||||
_UNIT_TO_MM = {"mm": 1.0, "cm": 10.0, "in": 25.4, "inch": 25.4, "inches": 25.4, "pt": 25.4 / 72}
|
||||
_UNIT_TO_PT = {"pt": 1.0, "mm": 72 / 25.4, "cm": 72 / 2.54, "in": 72.0, "inch": 72.0, "inches": 72.0}
|
||||
|
||||
|
||||
def _to_int(v: Any) -> int | None:
|
||||
if isinstance(v, bool) or v is None:
|
||||
return None
|
||||
if isinstance(v, int):
|
||||
return v
|
||||
if isinstance(v, float):
|
||||
return int(round(v))
|
||||
if isinstance(v, str):
|
||||
m = re.search(r"-?\d+", v)
|
||||
if m:
|
||||
return int(m.group(0))
|
||||
return None
|
||||
|
||||
|
||||
def _to_float(v: Any, units: dict[str, float] | None = None) -> float | None:
|
||||
if isinstance(v, bool) or v is None:
|
||||
return None
|
||||
if isinstance(v, (int, float)):
|
||||
return float(v)
|
||||
if isinstance(v, str):
|
||||
m = re.search(r"(-?\d+(?:\.\d+)?)\s*([a-zA-Z]+)?", v)
|
||||
if not m:
|
||||
return None
|
||||
x = float(m.group(1))
|
||||
unit = (m.group(2) or "").lower()
|
||||
if units and unit:
|
||||
factor = units.get(unit)
|
||||
if factor is None:
|
||||
return None # 未知单位不猜
|
||||
x *= factor
|
||||
return x
|
||||
return None
|
||||
|
||||
|
||||
def _to_bool(v: Any) -> bool | None:
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
if isinstance(v, (int, float)) and not isinstance(v, bool):
|
||||
if v in (0, 1):
|
||||
return bool(v)
|
||||
if isinstance(v, str):
|
||||
low = v.strip().lower()
|
||||
if low in ("true", "yes", "required"):
|
||||
return True
|
||||
if low in ("false", "no", "not required"):
|
||||
return False
|
||||
return None
|
||||
|
||||
|
||||
def _norm_refs(v: Any) -> str | None:
|
||||
if not isinstance(v, str) or not v.strip():
|
||||
return None
|
||||
low = v.strip().lower()
|
||||
low = _REFS_ALIAS.get(low, low)
|
||||
return low if low in _REFS_ALLOWED else "other"
|
||||
|
||||
|
||||
def _norm_conf(v: Any) -> float | None:
|
||||
if isinstance(v, bool) or not isinstance(v, (int, float)):
|
||||
return None
|
||||
return max(0.0, min(1.0, float(v)))
|
||||
|
||||
|
||||
def _normalize(raw: dict[str, Any]) -> dict[str, Any]:
|
||||
"""白名单归一：类型错的值置 null（宁缺勿错，raw 全量保留在结果里）。"""
|
||||
src = raw.get("rules")
|
||||
src = src if isinstance(src, dict) else {}
|
||||
rules: dict[str, Any] = {}
|
||||
for k in _RULE_INT_KEYS:
|
||||
rules[k] = _to_int(src.get(k))
|
||||
rules["min_font_pt"] = _to_float(src.get("min_font_pt"), _UNIT_TO_PT)
|
||||
rules["min_margin_mm"] = _to_float(src.get("min_margin_mm"), _UNIT_TO_MM)
|
||||
rules["min_line_spacing"] = _to_float(src.get("min_line_spacing"))
|
||||
rules["refs_style"] = _norm_refs(src.get("refs_style"))
|
||||
rules["anonymous"] = _to_bool(src.get("anonymous"))
|
||||
|
||||
sq_src = raw.get("source_quotes")
|
||||
sq_src = sq_src if isinstance(sq_src, dict) else {}
|
||||
quotes: dict[str, list[str]] = {}
|
||||
for key, val in sq_src.items():
|
||||
if key != "_general" and key not in rules:
|
||||
continue
|
||||
items = val if isinstance(val, list) else [val]
|
||||
cleaned = [str(x).strip() for x in items if isinstance(x, (str, int, float))
|
||||
and str(x).strip()]
|
||||
if cleaned:
|
||||
quotes[key] = cleaned
|
||||
|
||||
conf_src = raw.get("confidence")
|
||||
conf_src = conf_src if isinstance(conf_src, dict) else {}
|
||||
pf_src = conf_src.get("per_field")
|
||||
pf_src = pf_src if isinstance(pf_src, dict) else {}
|
||||
per_field: dict[str, float] = {}
|
||||
for k, v in pf_src.items():
|
||||
c = _norm_conf(v)
|
||||
if c is not None and (k in rules or k == "_general"):
|
||||
per_field[k] = c
|
||||
overall = _norm_conf(conf_src.get("overall"))
|
||||
if overall is None and per_field:
|
||||
overall = sum(per_field.values()) / len(per_field)
|
||||
confidence = {"overall": overall, "per_field": per_field}
|
||||
|
||||
notes = raw.get("notes")
|
||||
return {
|
||||
"rules": rules,
|
||||
"source_quotes": quotes,
|
||||
"confidence": confidence,
|
||||
"notes": str(notes) if notes else "",
|
||||
}
|
||||
|
||||
|
||||
def _validation_error(norm: dict[str, Any]) -> str | None:
|
||||
"""强制凭证：任一非 null 字段缺 source_quotes → 拒收（ok=False）。"""
|
||||
missing = [k for k, v in norm["rules"].items()
|
||||
if v is not None and k not in norm["source_quotes"]]
|
||||
if missing:
|
||||
return f"LLM 输出缺少非 null 字段的 source_quotes：{', '.join(missing)}"
|
||||
if not norm["source_quotes"].get("_general"):
|
||||
return "LLM 输出缺少 source_quotes._general 整体引证"
|
||||
if any(v is not None for v in norm["rules"].values()):
|
||||
return None
|
||||
return "LLM 未抽出任何非 null 字段（rules 全空）"
|
||||
|
||||
|
||||
async def extract_format_rules(markdown: str, journal_name: str) -> dict:
|
||||
"""markdown（≤30000 字符）+ 期刊名 → 抽取结果（schema 见模块 docstring）。"""
|
||||
if not markdown or not markdown.strip():
|
||||
return {"ok": False, "error": "markdown 为空", "rules": None,
|
||||
"source_quotes": None, "confidence": None, "notes": None, "raw": None}
|
||||
user = _USER_TMPL.format(journal_name=journal_name, markdown=markdown)
|
||||
try:
|
||||
raw = await complete_json(_SYSTEM, user, temperature=0.1, max_tokens=8000)
|
||||
except LLMError as exc:
|
||||
return {"ok": False, "error": f"LLM 调用失败：{exc}", "rules": None,
|
||||
"source_quotes": None, "confidence": None, "notes": None, "raw": None}
|
||||
if not isinstance(raw, dict):
|
||||
return {"ok": False, "error": f"LLM 输出非 JSON 对象：{type(raw).__name__}",
|
||||
"rules": None, "source_quotes": None, "confidence": None,
|
||||
"notes": None, "raw": None}
|
||||
|
||||
norm = _normalize(raw)
|
||||
err = _validation_error(norm)
|
||||
if err:
|
||||
logger.warning("[extract] %s 校验失败：%s", journal_name, err)
|
||||
return {"ok": False, "error": err, "rules": None,
|
||||
"source_quotes": norm["source_quotes"], "confidence": norm["confidence"],
|
||||
"notes": norm["notes"], "raw": raw}
|
||||
filled = sum(1 for v in norm["rules"].values() if v is not None)
|
||||
logger.info("[extract] %s → 8 字段中 %d 个非 null", journal_name, filled)
|
||||
return {"ok": True, "error": None, "rules": norm["rules"],
|
||||
"source_quotes": norm["source_quotes"],
|
||||
"confidence": norm["confidence"], "notes": norm["notes"], "raw": raw}
|
||||
@@ -1,356 +0,0 @@
|
||||
"""services/format_fetcher.py —— 投稿指南页面抓取（格式规则自动获取·第一批）。
|
||||
|
||||
约束（用户拍板 2026-10-10）：
|
||||
- 零新依赖：httpx + beautifulsoup4（venv 已装 4.15.0，用户明确指定使用；
|
||||
lxml 6.1.2 在 requirements.lock 内，作 bs4 解析器）
|
||||
- 同域限速 ≥ 2s（robots.txt 与正文请求共用计时器）
|
||||
- 抓正文前先 GET /robots.txt，命中 Disallow → ok=False 跳过
|
||||
- UA: "PlantAI-RAG-Bot/1.0 (+contact: admin@example.com)"
|
||||
- 只提取正文（去导航/页脚/脚本），截断至 30000 字符
|
||||
|
||||
返回固定四键：{"ok": bool, "markdown": str, "error": str | None, "fetched_at": str}
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup, NavigableString
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
USER_AGENT = "PlantAI-RAG-Bot/1.0 (+contact: admin@example.com)"
|
||||
TIMEOUT = 30.0
|
||||
MAX_CHARS = 30_000
|
||||
MIN_INTERVAL_PER_HOST = 2.0 # 同域请求间隔 ≥ 2s（robots 与正文共用）
|
||||
_MIN_TEXT_CHARS = 200 # 正文短于此 → 疑似 JS 渲染/反爬页
|
||||
_HEADERS = {
|
||||
"User-Agent": USER_AGENT,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
}
|
||||
|
||||
# 同域限速：host → 上次请求完成时刻（monotonic）
|
||||
_last_hit: dict[str, float] = {}
|
||||
_throttle_lock = asyncio.Lock()
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def _fail(error: str) -> dict:
|
||||
return {"ok": False, "markdown": "", "error": error, "fetched_at": _now_iso()}
|
||||
|
||||
|
||||
async def _throttle(host: str) -> None:
|
||||
"""同域串行 + ≥2s 间隔（全局锁：本批抓取量小，简单可靠优先）。"""
|
||||
async with _throttle_lock:
|
||||
now = time.monotonic()
|
||||
wait = _last_hit.get(host, now) + MIN_INTERVAL_PER_HOST - now
|
||||
if wait > 0:
|
||||
await asyncio.sleep(wait)
|
||||
_last_hit[host] = time.monotonic()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- robots.txt
|
||||
|
||||
def _parse_robots(text: str) -> list[tuple[list[str], list[tuple[str, str]]]]:
|
||||
"""→ [(agents, [(kind, pattern)...]), ...]（连续 UA 行同组；规则行后出现 UA=新组）。"""
|
||||
groups: list[tuple[list[str], list[tuple[str, str]]]] = []
|
||||
agents: list[str] = []
|
||||
rules: list[tuple[str, str]] = []
|
||||
for raw in text.splitlines():
|
||||
line = raw.split("#", 1)[0].strip()
|
||||
if not line or ":" not in line:
|
||||
continue
|
||||
key, _, value = line.partition(":")
|
||||
key, value = key.strip().lower(), value.strip()
|
||||
if key == "user-agent":
|
||||
if agents and rules: # 上一组已出规则 → 当前 UA 行开新组
|
||||
groups.append((agents, rules))
|
||||
agents, rules = [], []
|
||||
agents.append(value)
|
||||
elif key in ("allow", "disallow"):
|
||||
if not agents: # 无前置 UA 的规则按 * 组处理
|
||||
agents = ["*"]
|
||||
rules.append((key, value))
|
||||
if agents:
|
||||
groups.append((agents, rules))
|
||||
return groups
|
||||
|
||||
|
||||
def _robots_path_allowed(groups, path: str) -> tuple[bool, str]:
|
||||
"""最长匹配决胜（同长 Allow 优先；`$` 结尾=精确匹配；空 Disallow 跳过）。"""
|
||||
|
||||
def matched_by(pat: str) -> bool:
|
||||
if pat.endswith("$"):
|
||||
return path == pat[:-1]
|
||||
return path.startswith(pat)
|
||||
|
||||
token = "plantai-rag-bot"
|
||||
matched = [rules for agents, rules in groups
|
||||
if any(token in a.lower() for a in agents)]
|
||||
if not matched:
|
||||
matched = [rules for agents, rules in groups
|
||||
if any(a.lower() == "*" for a in agents)]
|
||||
if not matched:
|
||||
return True, "robots.txt 无匹配组（本 bot 不受限）"
|
||||
|
||||
best_len, best_allow, best_pat = -1, False, ""
|
||||
for rules in matched:
|
||||
for kind, pat in rules:
|
||||
if not pat: # "Disallow:" 空值 = 放行一切（该组无需处理）
|
||||
continue
|
||||
if not matched_by(pat):
|
||||
continue
|
||||
length = len(pat)
|
||||
allow = kind == "allow"
|
||||
if length > best_len or (length == best_len and allow and not best_allow):
|
||||
best_len, best_allow, best_pat = length, allow, pat
|
||||
if best_len < 0:
|
||||
return True, "robots.txt 允许（无路径匹配）"
|
||||
if best_allow:
|
||||
return True, f"robots.txt Allow 命中（{best_pat}）"
|
||||
return False, f"robots.txt Disallow 命中（{best_pat}）"
|
||||
|
||||
|
||||
async def _robots_allows(client: httpx.AsyncClient, url: str) -> tuple[bool, str]:
|
||||
p = urlparse(url)
|
||||
robots_url = f"{p.scheme}://{p.netloc}/robots.txt"
|
||||
await _throttle(p.netloc.lower())
|
||||
try:
|
||||
resp = await client.get(robots_url, headers=_HEADERS)
|
||||
except httpx.HTTPError as exc:
|
||||
return True, f"robots.txt 获取失败（{type(exc).__name__}），按允许处理"
|
||||
if resp.status_code != 200:
|
||||
return True, f"robots.txt HTTP {resp.status_code}，按允许处理"
|
||||
groups = _parse_robots(resp.text)
|
||||
return _robots_path_allowed(groups, p.path or "/")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 正文提取
|
||||
|
||||
_STRIP_TAGS = ("script", "style", "noscript", "iframe", "form", "nav",
|
||||
"footer", "aside", "button", "select", "textarea", "input", "svg")
|
||||
_STRIP_ROLES = ("navigation", "contentinfo", "complementary", "search")
|
||||
# 正文容器候选（按优先级；首个文本 ≥500 字符者胜出）
|
||||
_CONTENT_SELECTORS = (
|
||||
"main", "article", '[role="main"]', "#content", "#main", "#main-content",
|
||||
".article-body", "#article-body", ".article__body", ".hub-article-body",
|
||||
".page-content", ".main-content", ".content-area",
|
||||
)
|
||||
_CHALLENGE_MARKERS = ("just a moment", "attention required", "captcha",
|
||||
"access denied", "ddos protection", "verify you are human")
|
||||
|
||||
|
||||
def _decode_html(resp: httpx.Response) -> str:
|
||||
"""header charset → <meta charset> → utf-8（errors=replace），防 OUP/Wiley 无 header 乱码。"""
|
||||
enc = None
|
||||
m = re.search(r"charset=([\w.-]+)", resp.headers.get("content-type", ""), re.I)
|
||||
if m:
|
||||
enc = m.group(1)
|
||||
if not enc:
|
||||
m = re.search(rb"""charset\s*=\s*["']?([\w.-]+)""", resp.content[:4096], re.I)
|
||||
if m:
|
||||
enc = m.group(1).decode("ascii", "ignore")
|
||||
try:
|
||||
return resp.content.decode(enc or "utf-8", errors="replace")
|
||||
except LookupError:
|
||||
return resp.content.decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
def _squeeze(s: str) -> str:
|
||||
"""行内空白压缩 + 去空行（保留单换行）。"""
|
||||
lines = [re.sub(r"[ \t\r\f\v]+", " ", line).strip() for line in s.split("\n")]
|
||||
return "\n".join(line for line in lines if line)
|
||||
|
||||
|
||||
def _inline(el) -> str:
|
||||
"""行内渲染：a→md链接 / strong→** / em→* / code→` / br→换行；跳过列表与图片。"""
|
||||
parts: list[str] = []
|
||||
for child in el.children:
|
||||
name = getattr(child, "name", None)
|
||||
if name is None: # NavigableString
|
||||
parts.append(str(child))
|
||||
continue
|
||||
if name in ("ul", "ol", "img", "picture", "figure"):
|
||||
continue
|
||||
if name == "a":
|
||||
href = (child.get("href") or "").strip()
|
||||
txt = _inline(child).strip()
|
||||
if not txt:
|
||||
continue
|
||||
parts.append(f"[{txt}]({href})" if href.startswith(("http://", "https://")) else txt)
|
||||
elif name in ("strong", "b"):
|
||||
parts.append(f"**{_inline(child)}**")
|
||||
elif name in ("em", "i"):
|
||||
parts.append(f"*{_inline(child)}*")
|
||||
elif name == "code":
|
||||
parts.append(f"`{_inline(child).strip()}`")
|
||||
elif name == "br":
|
||||
parts.append("\n")
|
||||
else:
|
||||
parts.append(_inline(child))
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def _list_md(el, ordered: bool) -> str:
|
||||
lines: list[str] = []
|
||||
idx = 1
|
||||
for li in el.find_all("li", recursive=False):
|
||||
# li 文本不含嵌套列表（嵌套列表随后缩进输出）
|
||||
buf = []
|
||||
for child in li.children:
|
||||
if getattr(child, "name", None) in ("ul", "ol"):
|
||||
continue
|
||||
buf.append(_inline(child) if getattr(child, "name", None) else str(child))
|
||||
txt = " ".join("".join(buf).split())
|
||||
if txt:
|
||||
lines.append(f"{idx}. {txt}" if ordered else f"- {txt}")
|
||||
idx += 1
|
||||
for nested in li.find_all(("ul", "ol"), recursive=False):
|
||||
sub = _list_md(nested, nested.name == "ol")
|
||||
if sub:
|
||||
lines.extend(" " + line for line in sub.split("\n"))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _table_md(table) -> str | None:
|
||||
rows: list[list[str]] = []
|
||||
for tr in table.find_all("tr"):
|
||||
cells = [" ".join(_squeeze(_inline(c)).split())
|
||||
for c in tr.find_all(("td", "th"))]
|
||||
if cells:
|
||||
rows.append(cells)
|
||||
if not rows:
|
||||
return None
|
||||
out = ["| " + " | ".join(rows[0]) + " |"]
|
||||
if table.find("th"): # 首行含表头 → 补分隔行（合法 GFM 表）
|
||||
out.append("| " + " | ".join(["---"] * len(rows[0])) + " |")
|
||||
out += ["| " + " | ".join(r) + " |" for r in rows[1:]]
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def _blocks(el, out: list[str]) -> None:
|
||||
for child in el.children:
|
||||
name = getattr(child, "name", None)
|
||||
if name is None: # 裸文本
|
||||
txt = _squeeze(str(child))
|
||||
if txt:
|
||||
out.append(txt)
|
||||
continue
|
||||
if name in _STRIP_TAGS:
|
||||
continue
|
||||
if name in ("h1", "h2", "h3", "h4", "h5", "h6"):
|
||||
txt = " ".join(_squeeze(_inline(child)).split())
|
||||
if txt:
|
||||
out.append("#" * int(name[1]) + " " + txt)
|
||||
elif name == "p":
|
||||
txt = _squeeze(_inline(child))
|
||||
if txt:
|
||||
out.append(txt)
|
||||
elif name in ("ul", "ol"):
|
||||
md = _list_md(child, name == "ol")
|
||||
if md:
|
||||
out.append(md)
|
||||
elif name == "table":
|
||||
md = _table_md(child)
|
||||
if md:
|
||||
out.append(md)
|
||||
elif name == "blockquote":
|
||||
txt = _squeeze(_inline(child))
|
||||
if txt:
|
||||
out.append("\n".join("> " + line for line in txt.split("\n")))
|
||||
elif name == "pre":
|
||||
code = child.get_text().strip("\n")
|
||||
if code.strip():
|
||||
out.append("```\n" + code + "\n```")
|
||||
elif name == "hr":
|
||||
out.append("---")
|
||||
else: # div/section/article/main/figure/dl/details/… 容器递归
|
||||
_blocks(child, out)
|
||||
|
||||
|
||||
def _html_to_markdown(html: str) -> tuple[str, str | None]:
|
||||
"""→ (markdown, error)。选正文容器 → 去壳 → DOM 递归转 Markdown。"""
|
||||
soup = BeautifulSoup(html, "lxml") # lxml 在 lock 内；缺时 bs4 自动回退 html.parser
|
||||
|
||||
title = (soup.title.get_text() if soup.title else "") or ""
|
||||
for tag in soup.find_all(_STRIP_TAGS):
|
||||
tag.decompose()
|
||||
for tag in soup.find_all(attrs={"role": _STRIP_ROLES}):
|
||||
tag.decompose()
|
||||
|
||||
container = None
|
||||
fallback_hits = []
|
||||
for sel in _CONTENT_SELECTORS:
|
||||
for node in soup.select(sel):
|
||||
text_len = len(node.get_text(" ", strip=True))
|
||||
fallback_hits.append((text_len, node))
|
||||
if text_len >= 500:
|
||||
container = node
|
||||
break
|
||||
if container is not None:
|
||||
break
|
||||
if container is None:
|
||||
if fallback_hits:
|
||||
container = max(fallback_hits, key=lambda x: x[0])[1]
|
||||
else:
|
||||
container = soup.body or soup
|
||||
|
||||
blocks: list[str] = []
|
||||
_blocks(container, blocks)
|
||||
md = "\n\n".join(blocks)
|
||||
|
||||
probe = (title + " " + md[:500]).lower()
|
||||
marker = next((m for m in _CHALLENGE_MARKERS if m in probe), None)
|
||||
if marker:
|
||||
return "", f"疑似反爬/挑战页（命中 {marker!r}，title={title[:80]!r}）"
|
||||
if len(md) < _MIN_TEXT_CHARS:
|
||||
return "", f"正文提取过短（{len(md)} < {_MIN_TEXT_CHARS} 字符），疑似 JS 渲染或空壳页"
|
||||
if len(md) > MAX_CHARS:
|
||||
md = md[:MAX_CHARS - 1] + "…"
|
||||
return md, None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 入口
|
||||
|
||||
async def fetch_submission_guidelines(url: str) -> dict:
|
||||
"""抓投稿指南页面 → 正文 Markdown。
|
||||
|
||||
流程：robots.txt 检查（≥2s 限速）→ 禁止即跳过 → GET 正文（≥2s 限速）
|
||||
→ 状态/类型校验 → 正文提取 → 截断 30000 字符。
|
||||
"""
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in ("http", "https") or not parsed.netloc:
|
||||
return _fail(f"非法 URL：{url!r}")
|
||||
host = parsed.netloc.lower()
|
||||
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=TIMEOUT) as client:
|
||||
allowed, robots_reason = await _robots_allows(client, url)
|
||||
if not allowed:
|
||||
logger.info("[fetch] robots 禁止，跳过 %s（%s）", url, robots_reason)
|
||||
return _fail(robots_reason)
|
||||
|
||||
await _throttle(host)
|
||||
try:
|
||||
resp = await client.get(url, headers=_HEADERS)
|
||||
except httpx.HTTPError as exc:
|
||||
return _fail(f"请求失败（{type(exc).__name__}）：{exc}")
|
||||
if resp.status_code != 200:
|
||||
return _fail(f"HTTP {resp.status_code}")
|
||||
ctype = resp.headers.get("content-type", "")
|
||||
if "html" not in ctype and "xhtml" not in ctype:
|
||||
return _fail(f"非 HTML 响应：{ctype or '无 Content-Type'}")
|
||||
|
||||
markdown, err = _html_to_markdown(_decode_html(resp))
|
||||
if err:
|
||||
return _fail(f"{err}（robots: {robots_reason}）")
|
||||
logger.info("[fetch] %s → %d 字符（robots: %s）", url, len(markdown), robots_reason)
|
||||
return {"ok": True, "markdown": markdown, "error": None, "fetched_at": _now_iso()}
|
||||
@@ -1,223 +0,0 @@
|
||||
"""services/journal_kb.py —— 期刊 scope_desc → vstore 入库 + 向量匹配（只读消费 8094）。
|
||||
|
||||
链路（全部复用 vstore 既有 API，零改动）：
|
||||
1. ensure_scope_docs：各期刊 scope_desc 以 .txt 入库（category=journal_scope，
|
||||
name=期刊名），document_id 回写 journals.vstore_doc_id（幂等：已有 id 不重灌）。
|
||||
2. match：GET /api/vstore/search-chunks?q=<摘要截断>&doc_ids=<id 白名单>&top_k=N
|
||||
→ 按 doc_id 映射回期刊，取该期刊最高 chunk 分数为 similarity。
|
||||
3. 自愈：白名单命中 0 条 → GET /documents/{id} 探活 → 失效的重灌（vstore 去重
|
||||
语义下同内容重灌会换 doc_id，需回写）→ 重试一次 → 仍 0 视为语义低相似。
|
||||
|
||||
错误语义：连通性/上游异常抛 MatchError（路由降级为全量候选）；
|
||||
返回 ([], False) 表示检索成功但无有效匹配（路由同样降级）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from ..config import get_settings
|
||||
from ..db import set_vstore_doc_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SCOPE_CATEGORY = "journal_scope"
|
||||
|
||||
|
||||
class MatchError(Exception):
|
||||
"""vstore 向量匹配失败（不可达/响应异常）。"""
|
||||
|
||||
|
||||
def _timeout() -> httpx.Timeout:
|
||||
t = get_settings().vstore_match_timeout
|
||||
return httpx.Timeout(connect=5.0, read=t, write=t, pool=5.0)
|
||||
|
||||
|
||||
def _err_body(resp: httpx.Response) -> str:
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError:
|
||||
return (resp.text or "").strip()[:500] or "（空响应体）"
|
||||
if isinstance(payload, dict):
|
||||
err = payload.get("error")
|
||||
if isinstance(err, dict) and err.get("message"):
|
||||
return str(err["message"])
|
||||
if isinstance(err, str):
|
||||
return err
|
||||
return str(payload)[:500]
|
||||
|
||||
|
||||
async def _ingest_scope(client: httpx.AsyncClient, base: str, journal: dict[str, Any]) -> str:
|
||||
"""单本期刊 scope_desc 入库，返回 document_id。"""
|
||||
content = str(journal.get("scope_desc") or "").strip()
|
||||
if not content:
|
||||
raise MatchError(f"期刊 scope_desc 为空：{journal.get('id')}")
|
||||
jid = str(journal.get("id") or "")
|
||||
try:
|
||||
resp = await client.post(
|
||||
f"{base}/api/vstore/ingest",
|
||||
files={"file": (f"{jid}.txt", content.encode("utf-8"), "text/plain")},
|
||||
data={
|
||||
"name": str(journal.get("name") or jid),
|
||||
"category": SCOPE_CATEGORY,
|
||||
"description": "review_service 期刊范围种子（scope_desc 向量匹配）",
|
||||
},
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
raise MatchError(f"vstore 不可达（{base}）：{exc}") from exc
|
||||
if resp.status_code >= 400:
|
||||
raise MatchError(f"vstore ingest 失败（HTTP {resp.status_code}）：{_err_body(resp)}")
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError as exc:
|
||||
raise MatchError(f"vstore ingest 响应非 JSON：{exc}") from exc
|
||||
if not isinstance(payload, dict) or not payload.get("ok"):
|
||||
raise MatchError(
|
||||
str((payload.get("error") or {}).get("message") if isinstance(payload, dict) else "") or "vstore ingest 失败"
|
||||
)
|
||||
doc_id = str((payload.get("item") or {}).get("document_id") or "")
|
||||
if not doc_id:
|
||||
raise MatchError("vstore ingest 响应缺少 document_id")
|
||||
return doc_id
|
||||
|
||||
|
||||
async def ensure_scope_docs(journals: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""补齐缺失的 scope 入库映射（就地回写 vstore_doc_id 并持久到 DB）。"""
|
||||
settings = get_settings()
|
||||
missing = [j for j in journals if not j.get("vstore_doc_id")]
|
||||
if not missing:
|
||||
return journals
|
||||
async with httpx.AsyncClient(timeout=_timeout()) as client:
|
||||
for j in missing:
|
||||
try:
|
||||
doc_id = await _ingest_scope(client, settings.vstore_base, j)
|
||||
except MatchError as exc:
|
||||
# 单本失败不阻断其余（该期刊将不进白名单，由路由降级兜底）
|
||||
logger.warning("journal_kb: scope 入库失败（%s）：%s", j.get("id"), exc)
|
||||
continue
|
||||
j["vstore_doc_id"] = doc_id
|
||||
await set_vstore_doc_id(str(j["id"]), doc_id)
|
||||
return journals
|
||||
|
||||
|
||||
async def _search(
|
||||
client: httpx.AsyncClient, base: str, query: str, doc_ids: list[str]
|
||||
) -> list[dict[str, Any]]:
|
||||
"""chunk 级白名单检索。连通/上游错误抛 MatchError。"""
|
||||
if not doc_ids:
|
||||
# doc_ids 为空时 vstore 侧等于不过滤 → 会命中用户私有库，必须拦死
|
||||
raise MatchError("白名单为空，拒绝全库检索")
|
||||
try:
|
||||
resp = await client.get(
|
||||
f"{base}/api/vstore/search-chunks",
|
||||
params={
|
||||
"q": query,
|
||||
"top_k": get_settings().recommend_scope_top_k,
|
||||
"doc_ids": ",".join(doc_ids),
|
||||
},
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
raise MatchError(f"vstore search-chunks 不可达：{exc}") from exc
|
||||
if resp.status_code >= 400:
|
||||
raise MatchError(f"vstore search-chunks 失败（HTTP {resp.status_code}）：{_err_body(resp)}")
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError as exc:
|
||||
raise MatchError(f"vstore search-chunks 响应非 JSON：{exc}") from exc
|
||||
if not isinstance(payload, dict) or not payload.get("ok"):
|
||||
raise MatchError(
|
||||
str((payload.get("error") or {}).get("message") if isinstance(payload, dict) else "") or "vstore search-chunks 失败"
|
||||
)
|
||||
items = payload.get("items")
|
||||
return items if isinstance(items, list) else []
|
||||
|
||||
|
||||
async def _doc_alive(client: httpx.AsyncClient, base: str, doc_id: str) -> bool:
|
||||
try:
|
||||
resp = await client.get(f"{base}/api/vstore/documents/{doc_id}")
|
||||
except httpx.HTTPError:
|
||||
return False
|
||||
if resp.status_code != 200:
|
||||
return False
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError:
|
||||
return False
|
||||
return bool(isinstance(payload, dict) and payload.get("ok"))
|
||||
|
||||
|
||||
async def _refresh_dead_docs(
|
||||
client: httpx.AsyncClient, base: str, journals: list[dict[str, Any]]
|
||||
) -> None:
|
||||
"""白名单 0 命中时探活：失效文档重灌并回写新 id。"""
|
||||
for j in journals:
|
||||
doc_id = str(j.get("vstore_doc_id") or "")
|
||||
if not doc_id or await _doc_alive(client, base, doc_id):
|
||||
continue
|
||||
try:
|
||||
new_id = await _ingest_scope(client, base, j)
|
||||
except MatchError as exc:
|
||||
logger.warning("journal_kb: scope 重灌失败（%s）：%s", j.get("id"), exc)
|
||||
continue
|
||||
j["vstore_doc_id"] = new_id
|
||||
await set_vstore_doc_id(str(j["id"]), new_id)
|
||||
|
||||
|
||||
async def match(
|
||||
abstract: str, journals: list[dict[str, Any]]
|
||||
) -> tuple[list[dict[str, Any]], bool]:
|
||||
"""摘要 vs scope_desc 向量匹配。
|
||||
|
||||
返回 (candidates, vector_ok)；candidates 元素：
|
||||
{"journal": <journals 行>, "similarity": float}
|
||||
按 similarity 降序。MatchError → 路由降级；([], False) → 语义低相似。
|
||||
"""
|
||||
settings = get_settings()
|
||||
query = abstract.strip()[: settings.recommend_query_chars]
|
||||
if not query:
|
||||
return [], False
|
||||
|
||||
await ensure_scope_docs(journals)
|
||||
whitelisted = [j for j in journals if j.get("vstore_doc_id")]
|
||||
if not whitelisted:
|
||||
raise MatchError("无任何期刊 scope 入库成功（vstore_doc_id 为空）")
|
||||
|
||||
doc_map = {str(j["vstore_doc_id"]): j for j in whitelisted}
|
||||
timeout = _timeout()
|
||||
async with httpx.AsyncClient(timeout=timeout) as client:
|
||||
items = await _search(client, settings.vstore_base, query, list(doc_map))
|
||||
if not items:
|
||||
await _refresh_dead_docs(client, settings.vstore_base, whitelisted)
|
||||
doc_map = {str(j["vstore_doc_id"]): j for j in whitelisted if j.get("vstore_doc_id")}
|
||||
if not doc_map:
|
||||
return [], False
|
||||
items = await _search(client, settings.vstore_base, query, list(doc_map))
|
||||
|
||||
# 聚合：每本期刊取最高 chunk 分
|
||||
best: dict[str, float] = {}
|
||||
for item in items:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
j = doc_map.get(str(item.get("doc_id") or ""))
|
||||
if j is None:
|
||||
continue
|
||||
try:
|
||||
score = float(item.get("score"))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
jid = str(j["id"])
|
||||
if jid not in best or score > best[jid]:
|
||||
best[jid] = score
|
||||
if not best:
|
||||
return [], False
|
||||
|
||||
by_id = {str(j["id"]): j for j in journals}
|
||||
candidates = [
|
||||
{"journal": by_id[jid], "similarity": score}
|
||||
for jid, score in best.items()
|
||||
if jid in by_id
|
||||
]
|
||||
candidates.sort(key=lambda c: c["similarity"], reverse=True)
|
||||
return candidates, bool(candidates)
|
||||
@@ -1,210 +0,0 @@
|
||||
"""services/journal_rank.py —— LLM 重排 + 幻觉防护 + 三档归类。
|
||||
|
||||
职责：
|
||||
1. rerank：把带语义分的候选期刊（来自 journal_kb）+ 用户偏好交给 LLM，
|
||||
要求按 id 白名单返回 {rankings:[{id, tier, match_reason, risk_note}]}。
|
||||
2. validate：硬校验——白名单外 id 丢弃（幻觉拦截）；tier 非法/LLM 缺漏的
|
||||
期刊按规则补位（绝不因 LLM 输出丢失候选期刊）；期刊名称/指标一律不采信
|
||||
LLM，最终展示全部用 DB 字段（防止改名型幻觉）。
|
||||
3. fallback_rankings：LLM 不可用时的确定性规则分档（按 impact_factor）。
|
||||
|
||||
三档语义：challenge 挑战型（高影响/慢审稿）、fit 适配型（均衡）、
|
||||
safe 稳妥型（快速/门槛低）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from ..config import get_settings
|
||||
from . import llm
|
||||
|
||||
TIERS = ("challenge", "fit", "safe")
|
||||
TIER_LABELS = {
|
||||
"challenge": "挑战型（高影响因子/慢审稿）",
|
||||
"fit": "适配型（中等影响/周期适中）",
|
||||
"safe": "稳妥型（快速发表/门槛较低）",
|
||||
}
|
||||
|
||||
# 规则分档阈值（近似值，可后续配置化）
|
||||
_CHALLENGE_MIN_IF = 9.0
|
||||
_SAFE_MAX_IF = 4.0
|
||||
|
||||
# tier 同义词归一（LLM 偶发中文标签）
|
||||
_TIER_ALIASES = {
|
||||
"挑战": "challenge",
|
||||
"挑战型": "challenge",
|
||||
"高风险": "challenge",
|
||||
"适配": "fit",
|
||||
"适配型": "fit",
|
||||
"稳妥": "safe",
|
||||
"稳妥型": "safe",
|
||||
"低风险": "safe",
|
||||
}
|
||||
|
||||
SYSTEM_PROMPT = """你是学术投稿推荐专家（植物基因组学方向）。根据给定候选期刊、论文摘要与用户偏好，为**每一个**候选期刊输出推荐分档与理由。
|
||||
|
||||
硬规则（违反即视为无效输出）：
|
||||
1. id 白名单：只能使用输入 candidates 中出现的 id；绝对禁止添加、替换或臆造列表之外的任何期刊。
|
||||
2. 每个输入 id 恰好输出一次，不得遗漏、不得重复；期刊名称等元数据由系统填充，你只输出 id。
|
||||
3. tier 只能取 challenge | fit | safe：
|
||||
- challenge（挑战型）：高影响因子、审稿较慢或竞争激烈的期刊
|
||||
- fit（适配型）：影响力与发表周期均衡、与论文契合的期刊
|
||||
- safe（稳妥型）：审稿较快、门槛较低、命中概率较高的期刊
|
||||
4. match_reason：中文，1-2 句，结合摘要的研究主题与该刊 scope_desc 及指标（如有 similarity 一并参考），说明为何匹配/不匹配。
|
||||
5. risk_note：中文，1-2 句，指出投稿风险（超出用户偏好、IF 门槛偏高、范围匹配偏弱、审稿周期长等）；无明显风险写「无」。
|
||||
6. 用户 preferences 为软约束：不满足偏好的期刊仍必须输出，并在 risk_note 中明确说明。
|
||||
7. 不确定时宁可保守陈述，禁止编造统计数据（录用率、审稿天数等）。
|
||||
|
||||
输出：仅一个 JSON 对象，格式：
|
||||
{"rankings":[{"id":"...","tier":"challenge","match_reason":"...","risk_note":"..."}]}
|
||||
不要输出任何 JSON 之外的内容。"""
|
||||
|
||||
|
||||
class RankError(Exception):
|
||||
"""LLM 输出形态异常（无法按 JSON 解析/结构非法）。"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Ranking:
|
||||
journal_id: str
|
||||
tier: str
|
||||
match_reason: str
|
||||
risk_note: str
|
||||
|
||||
|
||||
def rule_tier(journal: dict[str, Any]) -> str:
|
||||
"""确定性分档规则（LLM 缺漏/不可用时兜底）。"""
|
||||
inf = journal.get("impact_factor")
|
||||
weeks = journal.get("review_time_weeks")
|
||||
if isinstance(inf, (int, float)):
|
||||
if inf >= _CHALLENGE_MIN_IF:
|
||||
return "challenge"
|
||||
if inf <= _SAFE_MAX_IF:
|
||||
return "safe"
|
||||
elif isinstance(weeks, (int, float)) and weeks <= 4:
|
||||
return "safe"
|
||||
return "fit"
|
||||
|
||||
|
||||
def build_user_prompt(
|
||||
abstract: str,
|
||||
candidates: list[dict[str, Any]],
|
||||
preferences: dict[str, Any],
|
||||
) -> str:
|
||||
"""候选 + 偏好 + 摘要 → 用户消息（JSON）。"""
|
||||
payload = {
|
||||
"abstract": abstract.strip()[:4000],
|
||||
"preferences": preferences or {},
|
||||
"candidates": [
|
||||
{
|
||||
"id": str(c["journal"]["id"]),
|
||||
"name": str(c["journal"].get("name") or ""),
|
||||
"impact_factor": c["journal"].get("impact_factor"),
|
||||
"review_time_weeks": c["journal"].get("review_time_weeks"),
|
||||
"acceptance_rate": c["journal"].get("acceptance_rate"),
|
||||
"similarity": c.get("similarity"),
|
||||
"scope_desc": str(c["journal"].get("scope_desc") or "")[:400],
|
||||
}
|
||||
for c in candidates
|
||||
],
|
||||
}
|
||||
return json.dumps(payload, ensure_ascii=False)
|
||||
|
||||
|
||||
def _normalize_tier(raw: Any) -> str | None:
|
||||
tier = str(raw or "").strip().lower()
|
||||
tier = _TIER_ALIASES.get(str(raw or "").strip(), tier)
|
||||
return tier if tier in TIERS else None
|
||||
|
||||
|
||||
def validate_rankings(
|
||||
payload: Any, candidates: list[dict[str, Any]]
|
||||
) -> tuple[list[Ranking], int, int]:
|
||||
"""校验 LLM 输出。
|
||||
|
||||
返回 (rankings, dropped_hallucinated, repaired)：
|
||||
- 白名单外 id / 重复 id → 丢弃并计数（repaired 中重复不计）；
|
||||
- tier 非法、LLM 缺漏的 id → 按规则补位（repaired 计数）。
|
||||
"""
|
||||
if not isinstance(payload, dict) or not isinstance(payload.get("rankings"), list):
|
||||
raise RankError("LLM 输出缺少 rankings 数组")
|
||||
|
||||
by_id = {str(c["journal"]["id"]): c["journal"] for c in candidates}
|
||||
input_ids = list(by_id)
|
||||
seen: set[str] = set()
|
||||
rankings: list[Ranking] = []
|
||||
dropped = 0
|
||||
repaired = 0
|
||||
|
||||
for entry in payload["rankings"]:
|
||||
if not isinstance(entry, dict):
|
||||
dropped += 1
|
||||
continue
|
||||
jid = str(entry.get("id") or "").strip()
|
||||
if jid in seen:
|
||||
continue
|
||||
if jid not in by_id:
|
||||
dropped += 1 # 幻觉期刊：绝不进输出
|
||||
continue
|
||||
seen.add(jid)
|
||||
tier = _normalize_tier(entry.get("tier"))
|
||||
if tier is None:
|
||||
tier = rule_tier(by_id[jid])
|
||||
repaired += 1
|
||||
reason = str(entry.get("match_reason") or "").strip() or "基于期刊范围与指标的匹配"
|
||||
risk = str(entry.get("risk_note") or "").strip() or "无"
|
||||
rankings.append(Ranking(journal_id=jid, tier=tier, match_reason=reason, risk_note=risk))
|
||||
|
||||
for jid in input_ids:
|
||||
if jid in seen:
|
||||
continue
|
||||
rankings.append(
|
||||
Ranking(
|
||||
journal_id=jid,
|
||||
tier=rule_tier(by_id[jid]),
|
||||
match_reason="LLM 未给出该期刊条目，已按影响因子/周期规则补位",
|
||||
risk_note="未提供（LLM 输出缺漏，规则兜底）",
|
||||
)
|
||||
)
|
||||
repaired += 1
|
||||
return rankings, dropped, repaired
|
||||
|
||||
|
||||
async def rerank(
|
||||
abstract: str,
|
||||
candidates: list[dict[str, Any]],
|
||||
preferences: dict[str, Any],
|
||||
) -> tuple[list[Ranking], int, int]:
|
||||
"""LLM 重排 + 校验。LLMError / RankError 上抛（路由降级为规则分档）。"""
|
||||
settings = get_settings()
|
||||
payload = await llm.complete_json(
|
||||
SYSTEM_PROMPT,
|
||||
build_user_prompt(abstract, candidates, preferences),
|
||||
temperature=0.2,
|
||||
max_tokens=3000,
|
||||
timeout=settings.recommend_read_timeout,
|
||||
)
|
||||
return validate_rankings(payload, candidates)
|
||||
|
||||
|
||||
def fallback_rankings(candidates: list[dict[str, Any]]) -> list[Ranking]:
|
||||
"""LLM 不可用时的确定性分档（不丢任何候选期刊）。"""
|
||||
rankings: list[Ranking] = []
|
||||
for c in candidates:
|
||||
j = c["journal"]
|
||||
inf = j.get("impact_factor")
|
||||
weeks = j.get("review_time_weeks")
|
||||
rankings.append(
|
||||
Ranking(
|
||||
journal_id=str(j["id"]),
|
||||
tier=rule_tier(j),
|
||||
match_reason=(
|
||||
f"规则分档：影响因子 {inf if inf is not None else '未收录'}、"
|
||||
f"审稿约 {weeks if weeks is not None else '未收录'} 周"
|
||||
),
|
||||
risk_note="LLM 不可用，本条按影响因子/周期规则估算",
|
||||
)
|
||||
)
|
||||
return rankings
|
||||
@@ -1,520 +0,0 @@
|
||||
"""app/services/journal_seed.py —— journal_directory 核心导入逻辑（CLI 与启动 seed 共享）。
|
||||
|
||||
数据流（schema 仍由本模块独家拥有，app 路由层只读不建表）：
|
||||
openpyxl 非 read_only 加载（读 cell.hyperlink.target 来源链接）
|
||||
→ 「国际期刊」「中文期刊」两 sheet 按【列名】映射
|
||||
→ 行级清洗（空→None 绝不填 0；分区数字→"N区"；JCR学科→JSON 数组）
|
||||
→ 按 id 幂等 upsert（ON CONFLICT DO UPDATE，重跑 count 不变）
|
||||
|
||||
两个入口：
|
||||
- scripts/import_journals.py（CLI）：extract_rows_from_xlsx + upsert_to_db
|
||||
+ 对账报告（exit code 语义见脚本 docstring）
|
||||
- ensure_journal_directory_seeded()：服务启动自动 seed（main.py lifespan），
|
||||
仅当表不存在或为空才导入；任何失败收敛为 dict 返回值，绝不抛异常。
|
||||
|
||||
v4 源（2026-10-10）：JIF=2025 统计年/2026 JCR 发布；中文 IF=2024 统计年/
|
||||
年报 2025 版。源 xlsx 路径走 settings.journal_directory_seed_abs
|
||||
（REVIEW_JOURNAL_DIRECTORY_SEED_PATH 覆盖）。
|
||||
|
||||
id 策略：有 ISSN → "{lang}:{ISSN}"; 无 ISSN → "{lang}:{slug(name)}"；
|
||||
同 lang 内冲突 → 抛 JournalSeedError（CLI 中止导入 / startup 返回 error）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
import unicodedata
|
||||
from collections import Counter
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import openpyxl
|
||||
|
||||
from ..config import get_settings
|
||||
|
||||
SHEET_INTL = "国际期刊"
|
||||
SHEET_ZH = "中文期刊"
|
||||
EXPECT_INTL = 779 # 口径 sheet 声明（v4 不变）
|
||||
EXPECT_ZH = 277
|
||||
|
||||
# 官网列（v4 实际列名「官网网址」在首位；其余为历史/备用别名）
|
||||
OFFICIAL_URL_HEADERS = ("官网网址", "官网", "官方网址", "期刊官网")
|
||||
# JCR 学科列别名：v4 改名「JCR学科（2024源表）」，v2 为「JCR学科（源表）」
|
||||
JCR_DISCIPLINE_HEADERS = ("JCR学科（2024源表）", "JCR学科（源表）")
|
||||
|
||||
# 既有表缺列时的幂等迁移（缺哪加哪，重复跑不报错）
|
||||
MIGRATE_COLS = (
|
||||
("official_url", "TEXT"),
|
||||
("official_check_status", "TEXT"),
|
||||
("jcr_release_year", "INTEGER"),
|
||||
)
|
||||
|
||||
SCHEMA_SQL = """
|
||||
CREATE TABLE IF NOT EXISTS journal_directory (
|
||||
id TEXT PRIMARY KEY,
|
||||
lang TEXT NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
research_direction TEXT,
|
||||
cas_partition TEXT,
|
||||
cas_partition_note TEXT,
|
||||
cas_partition_year INTEGER,
|
||||
jif REAL,
|
||||
jif_year INTEGER,
|
||||
jif_note TEXT,
|
||||
jif_check_status TEXT,
|
||||
composite_impact_factor REAL,
|
||||
comprehensive_impact_factor REAL,
|
||||
metric_year INTEGER,
|
||||
metric_report_year INTEGER,
|
||||
metric_caliber TEXT,
|
||||
annual_report_page INTEGER,
|
||||
top_flag TEXT,
|
||||
publisher_series TEXT,
|
||||
journal_mark TEXT,
|
||||
article_type TEXT,
|
||||
issn TEXT,
|
||||
eissn TEXT,
|
||||
jcr_disciplines TEXT,
|
||||
supplement_note TEXT,
|
||||
evidence_note TEXT,
|
||||
official_url TEXT,
|
||||
official_check_status TEXT,
|
||||
jcr_release_year INTEGER,
|
||||
partition_source TEXT,
|
||||
jif_source TEXT,
|
||||
jif_source_url TEXT,
|
||||
partition_source_url TEXT,
|
||||
data_source_label TEXT,
|
||||
data_source_url TEXT,
|
||||
imported_at TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_dir_lang_name ON journal_directory(lang, name);
|
||||
CREATE INDEX IF NOT EXISTS idx_dir_lang_jif ON journal_directory(lang, jif);
|
||||
CREATE INDEX IF NOT EXISTS idx_dir_lang_cif ON journal_directory(lang, composite_impact_factor);
|
||||
CREATE INDEX IF NOT EXISTS idx_dir_lang_part ON journal_directory(lang, cas_partition);
|
||||
CREATE INDEX IF NOT EXISTS idx_dir_lang_domain ON journal_directory(lang, research_direction);
|
||||
"""
|
||||
|
||||
# upsert 列序（除 id 外全部 ON CONFLICT DO UPDATE）
|
||||
COLS = (
|
||||
"lang", "name", "research_direction", "cas_partition", "cas_partition_note",
|
||||
"cas_partition_year", "jif", "jif_year", "jif_note", "jif_check_status",
|
||||
"composite_impact_factor", "comprehensive_impact_factor", "metric_year",
|
||||
"metric_report_year", "metric_caliber", "annual_report_page", "top_flag",
|
||||
"publisher_series", "journal_mark", "article_type", "issn", "eissn",
|
||||
"jcr_disciplines", "supplement_note", "evidence_note", "official_url",
|
||||
"official_check_status", "jcr_release_year",
|
||||
"partition_source", "jif_source", "jif_source_url", "partition_source_url",
|
||||
"data_source_label", "data_source_url", "imported_at",
|
||||
)
|
||||
UPSERT_SQL = (
|
||||
"INSERT INTO journal_directory (id, " + ", ".join(COLS) + ") VALUES ("
|
||||
+ ", ".join("?" for _ in range(len(COLS) + 1)) + ") "
|
||||
+ "ON CONFLICT(id) DO UPDATE SET "
|
||||
+ ", ".join(f"{c}=excluded.{c}" for c in COLS)
|
||||
)
|
||||
|
||||
|
||||
class JournalSeedError(Exception):
|
||||
"""xlsx 缺失/结构错误/口径不符/id 冲突（不写库）。
|
||||
|
||||
exit_code 对齐 CLI 语义：1=id 冲突中止；2=行数与口径不符；3=文件/结构错误。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
*,
|
||||
exit_code: int = 3,
|
||||
stage: str = "读取失败",
|
||||
conflicts: list[str] | None = None,
|
||||
) -> None:
|
||||
super().__init__(message)
|
||||
self.exit_code = exit_code
|
||||
self.stage = stage
|
||||
self.conflicts = conflicts or []
|
||||
|
||||
|
||||
def new_stats() -> dict[str, Any]:
|
||||
"""对账统计骨架（CLI 报告与解析共享）。"""
|
||||
return {
|
||||
"read_intl": 0, "read_zh": 0, "success": 0, "skipped": 0, "failed": 0,
|
||||
"jif_missing": [], "cif_missing": [], "part_blank": [], "part_notfound": [],
|
||||
"part_dist": Counter(), "hl_jif": 0, "hl_part": 0, "hl_zh": 0,
|
||||
"id_issn": 0, "id_slug": 0, "official_url_detected": False,
|
||||
"migrated_cols": [], "migrated_skipped": [],
|
||||
"official_total": 0, "official_filled": 0, "official_check_filled": 0,
|
||||
"jcr_filled": 0, "official_missing": [],
|
||||
}
|
||||
|
||||
|
||||
def migrate(conn: sqlite3.Connection) -> tuple[list[str], list[str]]:
|
||||
"""幂等加列：PRAGMA 探测现有列，缺哪个 ALTER ADD 哪个。"""
|
||||
existing = {r[1] for r in conn.execute("PRAGMA table_info(journal_directory)")}
|
||||
added: list[str] = []
|
||||
skipped: list[str] = []
|
||||
for col, decl in MIGRATE_COLS:
|
||||
if col in existing:
|
||||
skipped.append(col)
|
||||
else:
|
||||
conn.execute(f"ALTER TABLE journal_directory ADD COLUMN {col} {decl}")
|
||||
added.append(col)
|
||||
conn.commit()
|
||||
return added, skipped
|
||||
|
||||
|
||||
# ---- 清洗原语 ----
|
||||
|
||||
def text(v: Any) -> str | None:
|
||||
"""单元格 → strip 后的文本；空/纯空白 → None（绝不填 0/空串）。"""
|
||||
if v is None:
|
||||
return None
|
||||
s = str(v).strip()
|
||||
return s or None
|
||||
|
||||
|
||||
def to_float(v: Any) -> float | None:
|
||||
if v is None or (isinstance(v, str) and not v.strip()):
|
||||
return None
|
||||
try:
|
||||
return float(v)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def to_int(v: Any) -> int | None:
|
||||
f = to_float(v)
|
||||
if f is None:
|
||||
return None
|
||||
return int(f)
|
||||
|
||||
|
||||
def slugify(name: str) -> str:
|
||||
"""小写 + 非词字符折叠为连字符；\\w 含 CJK（中文期刊无 ISSN 时的 id 兜底）。"""
|
||||
s = unicodedata.normalize("NFC", name.strip().casefold())
|
||||
s = re.sub(r"[^\w]+", "-", s, flags=re.UNICODE)
|
||||
s = re.sub(r"-{2,}", "-", s)
|
||||
return s.strip("-")
|
||||
|
||||
|
||||
def make_id(lang: str, name: str, issn: str | None) -> str:
|
||||
if issn:
|
||||
return f"{lang}:{issn}"
|
||||
return f"{lang}:{slugify(name)}"
|
||||
|
||||
|
||||
def parse_partition(raw: Any) -> tuple[str | None, str | None]:
|
||||
"""分区 → (cas_partition, note)。数字 1-4 → 'N区'；'未检得' → note；空白 → 双 None。"""
|
||||
if raw is None:
|
||||
return None, None
|
||||
if isinstance(raw, (int, float)) and not isinstance(raw, bool):
|
||||
n = int(raw)
|
||||
if 1 <= n <= 4:
|
||||
return f"{n}区", None
|
||||
return None, None
|
||||
s = str(raw).strip()
|
||||
if not s:
|
||||
return None, None
|
||||
if "未检得" in s:
|
||||
return None, "未检得"
|
||||
m = re.fullmatch(r"([1-4])\s*区?", s)
|
||||
if m:
|
||||
return f"{m.group(1)}区", None
|
||||
return s, None # 未知文本：原样保留（不丢信息）
|
||||
|
||||
|
||||
def split_disciplines(raw: Any) -> str | None:
|
||||
"""'A(SCIE);B(SCIE)' → JSON 数组字符串。"""
|
||||
s = text(raw)
|
||||
if not s:
|
||||
return None
|
||||
parts = [p.strip() for p in s.split(";") if p.strip()]
|
||||
return json.dumps(parts, ensure_ascii=False) if parts else None
|
||||
|
||||
|
||||
def hyperlink_target(cell) -> str | None:
|
||||
"""超链接单元格 → target URL；无链接 → None（需要非 read_only 模式）。"""
|
||||
try:
|
||||
target = cell.hyperlink.target if cell.hyperlink is not None else None
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
return text(target)
|
||||
|
||||
|
||||
# ---- sheet 解析 ----
|
||||
|
||||
def header_index(ws) -> dict[str, int]:
|
||||
headers = [text(c.value) for c in ws[1]]
|
||||
return {h: i for i, h in enumerate(headers) if h is not None}
|
||||
|
||||
|
||||
def read_intl(ws, imported_at: str, stats: dict) -> list[tuple]:
|
||||
"""国际期刊 sheet → 行元组（顺序同 COLS 前置 id）。"""
|
||||
ix = header_index(ws)
|
||||
for req in ("期刊名称", "研究方向", "中科院分区", "JIF影响因子"):
|
||||
if req not in ix:
|
||||
raise JournalSeedError(
|
||||
f"{SHEET_INTL} 缺列：{req}（实际列：{list(ix)}）",
|
||||
exit_code=3, stage="结构错误",
|
||||
)
|
||||
url_col = next((ix[h] for h in OFFICIAL_URL_HEADERS if h in ix), None)
|
||||
jcr_hdr = next((h for h in JCR_DISCIPLINE_HEADERS if h in ix), None)
|
||||
stats["official_url_detected"] = url_col is not None
|
||||
if url_col is not None:
|
||||
stats["official_url_headers"] = next(h for h in OFFICIAL_URL_HEADERS if h in ix)
|
||||
|
||||
rows: list[tuple] = []
|
||||
for row in ws.iter_rows(min_row=2):
|
||||
if not any(c.value is not None for c in row):
|
||||
continue # 尾部幻影空行：不计入 skipped
|
||||
name = text(row[ix["期刊名称"]].value) if ix["期刊名称"] < len(row) else None
|
||||
if not name:
|
||||
stats["skipped"] += 1
|
||||
continue
|
||||
issn = text(row[ix["ISSN"]].value) if "ISSN" in ix else None
|
||||
jif = to_float(row[ix["JIF影响因子"]].value)
|
||||
check_status = text(row[ix["JIF核对状态"]].value) if "JIF核对状态" in ix else None
|
||||
official_check = text(row[ix["官网核对状态"]].value) if "官网核对状态" in ix else None
|
||||
jcr_year = to_int(row[ix["JCR发布年"]].value) if "JCR发布年" in ix else None
|
||||
part, part_note = parse_partition(row[ix["中科院分区"]].value)
|
||||
|
||||
# JIF 空 → note = 核对状态原文（"未检得"/"源表标N/A" 等说明）
|
||||
jif_note = check_status if jif is None else None
|
||||
|
||||
jif_url = hyperlink_target(row[ix["JIF数据来源"]]) if "JIF数据来源" in ix else None
|
||||
part_url = hyperlink_target(row[ix["分区数据来源"]]) if "分区数据来源" in ix else None
|
||||
official = None
|
||||
if url_col is not None and url_col < len(row):
|
||||
official = text(row[url_col].value) or hyperlink_target(row[url_col])
|
||||
|
||||
rows.append((
|
||||
make_id("en", name, issn), "en", name,
|
||||
text(row[ix["研究方向"]].value),
|
||||
part, part_note,
|
||||
to_int(row[ix["分区版本年"]].value) if "分区版本年" in ix else None,
|
||||
jif,
|
||||
to_int(row[ix["JIF统计年"]].value) if "JIF统计年" in ix else None,
|
||||
jif_note, check_status,
|
||||
None, None, None, None, None, None,
|
||||
text(row[ix["Top（源表）"]].value) if "Top（源表）" in ix else None,
|
||||
text(row[ix["出版系列"]].value) if "出版系列" in ix else None,
|
||||
text(row[ix["期刊标记"]].value) if "期刊标记" in ix else None,
|
||||
text(row[ix["文章类型"]].value) if "文章类型" in ix else None,
|
||||
issn,
|
||||
text(row[ix["eISSN"]].value) if "eISSN" in ix else None,
|
||||
split_disciplines(row[ix[jcr_hdr]].value) if jcr_hdr else None,
|
||||
text(row[ix["补充说明"]].value) if "补充说明" in ix else None,
|
||||
text(row[ix["补充证据"]].value) if "补充证据" in ix else None,
|
||||
official,
|
||||
official_check,
|
||||
jcr_year,
|
||||
"official", "official",
|
||||
jif_url, part_url,
|
||||
text(row[ix["JIF数据来源"]].value) if "JIF数据来源" in ix else None,
|
||||
None, # data_source_url（国际表来源即 jif/partition 两链接）
|
||||
imported_at,
|
||||
))
|
||||
# 统计
|
||||
stats["official_total"] += 1
|
||||
stats["official_filled"] += 1 if official else 0
|
||||
stats["official_check_filled"] += 1 if official_check else 0
|
||||
stats["jcr_filled"] += 1 if jcr_year else 0
|
||||
if not official:
|
||||
stats["official_missing"].append(("en", name))
|
||||
if jif is None:
|
||||
stats["jif_missing"].append((name, check_status or ""))
|
||||
if part is None and part_note == "未检得":
|
||||
stats["part_notfound"].append(name)
|
||||
elif part is None:
|
||||
stats["part_blank"].append(name)
|
||||
stats["hl_jif"] += 1 if jif_url else 0
|
||||
stats["hl_part"] += 1 if part_url else 0
|
||||
stats["id_issn" if issn else "id_slug"] += 1
|
||||
return rows
|
||||
|
||||
|
||||
def read_zh(ws, imported_at: str, stats: dict) -> list[tuple]:
|
||||
"""中文期刊 sheet → 行元组。"""
|
||||
ix = header_index(ws)
|
||||
for req in ("中文期刊名称", "复合影响因子", "综合影响因子"):
|
||||
if req not in ix:
|
||||
raise JournalSeedError(
|
||||
f"{SHEET_ZH} 缺列：{req}（实际列：{list(ix)}）",
|
||||
exit_code=3, stage="结构错误",
|
||||
)
|
||||
url_col = next((ix[h] for h in OFFICIAL_URL_HEADERS if h in ix), None)
|
||||
stats["official_url_detected_zh"] = url_col is not None
|
||||
|
||||
rows: list[tuple] = []
|
||||
for row in ws.iter_rows(min_row=2):
|
||||
if not any(c.value is not None for c in row):
|
||||
continue # 尾部幻影空行：不计入 skipped
|
||||
name = text(row[ix["中文期刊名称"]].value)
|
||||
if not name:
|
||||
stats["skipped"] += 1
|
||||
continue
|
||||
composite = to_float(row[ix["复合影响因子"]].value)
|
||||
comprehensive = to_float(row[ix["综合影响因子"]].value)
|
||||
check_status = text(row[ix["核对状态"]].value) if "核对状态" in ix else None
|
||||
|
||||
src_label = text(row[ix["数据来源"]].value) if "数据来源" in ix else None
|
||||
src_url = hyperlink_target(row[ix["数据来源"]]) if "数据来源" in ix else None
|
||||
official = None
|
||||
if url_col is not None and url_col < len(row):
|
||||
official = text(row[url_col].value) or hyperlink_target(row[url_col])
|
||||
official_check = text(row[ix["官网核对状态"]].value) if "官网核对状态" in ix else None
|
||||
|
||||
rows.append((
|
||||
make_id("zh", name, None), "zh", name,
|
||||
text(row[ix["研究方向"]].value),
|
||||
# 中文分区原样（常量"不按此口径列示"），不分区 → partition_source=None
|
||||
text(row[ix["中科院分区"]].value) if "中科院分区" in ix else None,
|
||||
None, # cas_partition_note：中文表无"未检得"语义（口径=不按此口径列示）
|
||||
None, # cas_partition_year
|
||||
None, None, None, check_status,
|
||||
composite, comprehensive,
|
||||
to_int(row[ix["指标统计年"]].value) if "指标统计年" in ix else None,
|
||||
to_int(row[ix["年报版本年"]].value) if "年报版本年" in ix else None,
|
||||
text(row[ix["指标口径"]].value) if "指标口径" in ix else None,
|
||||
to_int(row[ix["年报PDF页码"]].value) if "年报PDF页码" in ix else None,
|
||||
None, None, None, None, None, None, None,
|
||||
text(row[ix["备注"]].value) if "备注" in ix else None,
|
||||
None, # evidence_note（中文表无此列）
|
||||
official,
|
||||
official_check,
|
||||
None, # jcr_release_year：中文表无 JCR 发布年列（前端中文 Tab 不显示）
|
||||
None, # partition_source：中文不按中科院口径
|
||||
"official", # jif_source：国内年报（official）
|
||||
None, None,
|
||||
src_label, src_url,
|
||||
imported_at,
|
||||
))
|
||||
stats["official_total"] += 1
|
||||
stats["official_filled"] += 1 if official else 0
|
||||
stats["official_check_filled"] += 1 if official_check else 0
|
||||
if not official:
|
||||
stats["official_missing"].append(("zh", name))
|
||||
if composite is None:
|
||||
stats["cif_missing"].append((name, check_status or ""))
|
||||
stats["hl_zh"] += 1 if src_url else 0
|
||||
stats["id_slug"] += 1
|
||||
return rows
|
||||
|
||||
|
||||
# ---- 共享主流程 ----
|
||||
|
||||
def extract_rows_from_xlsx(
|
||||
source: Path,
|
||||
*,
|
||||
imported_at: str | None = None,
|
||||
stats: dict | None = None,
|
||||
) -> tuple[list[tuple], list[tuple], dict]:
|
||||
"""xlsx → (rows_en, rows_zh, stats)。口径不符/id 冲突抛 JournalSeedError（不写库）。"""
|
||||
stats = stats if stats is not None else new_stats()
|
||||
imported_at = imported_at or datetime.now().isoformat(timespec="seconds")
|
||||
|
||||
try:
|
||||
wb = openpyxl.load_workbook(source, data_only=True)
|
||||
except FileNotFoundError as exc:
|
||||
raise JournalSeedError(
|
||||
f"源文件不存在: {source}", exit_code=3, stage="读取失败"
|
||||
) from exc
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise JournalSeedError(
|
||||
f"无法打开 Excel: {exc}", exit_code=3, stage="读取失败"
|
||||
) from exc
|
||||
if SHEET_INTL not in wb.sheetnames or SHEET_ZH not in wb.sheetnames:
|
||||
raise JournalSeedError(
|
||||
f"缺少 sheet：需要 {SHEET_INTL}/{SHEET_ZH}，实际 {wb.sheetnames}",
|
||||
exit_code=3, stage="结构错误",
|
||||
)
|
||||
|
||||
rows_en = read_intl(wb[SHEET_INTL], imported_at, stats)
|
||||
rows_zh = read_zh(wb[SHEET_ZH], imported_at, stats)
|
||||
stats["read_intl"], stats["read_zh"] = len(rows_en), len(rows_zh)
|
||||
for r in rows_en:
|
||||
stats["part_dist"][r[4] or ("未检得" if r[5] else "空白")] += 1
|
||||
|
||||
# 口径行数校验（不符 → 抛错，不写库）
|
||||
if len(rows_en) != EXPECT_INTL or len(rows_zh) != EXPECT_ZH:
|
||||
raise JournalSeedError(
|
||||
f"行数与口径不符：国际 {len(rows_en)}/{EXPECT_INTL}、"
|
||||
f"中文 {len(rows_zh)}/{EXPECT_ZH}，中止导入",
|
||||
exit_code=2, stage="行数校验失败",
|
||||
)
|
||||
|
||||
# id 冲突检测（同 lang 内重复 → 抛错，不写库）
|
||||
ids = [r[0] for r in rows_en + rows_zh]
|
||||
dup = [i for i, c in Counter(ids).items() if c > 1]
|
||||
if dup:
|
||||
conflicts = [f"{i} × {Counter(ids)[i]}" for i in sorted(dup)]
|
||||
raise JournalSeedError(
|
||||
f"发现 {len(conflicts)} 个重复 id，中止导入（未写库）",
|
||||
exit_code=1, stage="id 冲突中止", conflicts=conflicts,
|
||||
)
|
||||
|
||||
return rows_en, rows_zh, stats
|
||||
|
||||
|
||||
def upsert_to_db(db_path: Path, rows_en: list[tuple], rows_zh: list[tuple]) -> dict[str, int]:
|
||||
"""建表(IF NOT EXISTS)+幂等加列+upsert → {"en","zh","total"}。"""
|
||||
db_path = Path(db_path)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
try:
|
||||
conn.executescript(SCHEMA_SQL)
|
||||
migrate(conn)
|
||||
conn.executemany(UPSERT_SQL, list(rows_en) + list(rows_zh))
|
||||
conn.commit()
|
||||
d = dict(conn.execute(
|
||||
"SELECT lang, COUNT(*) FROM journal_directory GROUP BY lang").fetchall())
|
||||
return {"en": d.get("en", 0), "zh": d.get("zh", 0), "total": sum(d.values())}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def ensure_journal_directory_seeded(
|
||||
db_path: Path | None = None,
|
||||
source_xlsx: Path | None = None,
|
||||
) -> dict:
|
||||
"""启动自动 seed：空表才导入；任何失败收敛为 dict，绝不抛异常。
|
||||
|
||||
返回统一四键：seeded/reason/error/rows（另有 counts 仅成功时给出 en/zh 明细）。
|
||||
- 已有数据（只认空/非空，不做补差）→ seeded=False, reason="already has N rows, skip"
|
||||
- 空表导入成功 → seeded=True, rows=1056
|
||||
- xlsx 缺失/结构错误/口径不符/id 冲突 → seeded=False, error="..."
|
||||
"""
|
||||
settings = get_settings()
|
||||
db = Path(db_path) if db_path is not None else settings.db_path_abs
|
||||
source = Path(source_xlsx) if source_xlsx is not None else settings.journal_directory_seed_abs
|
||||
|
||||
try:
|
||||
if db.exists():
|
||||
conn = sqlite3.connect(str(db))
|
||||
try:
|
||||
row = conn.execute("SELECT COUNT(*) FROM journal_directory").fetchone()
|
||||
except sqlite3.OperationalError:
|
||||
row = None # 表不存在 → 视为空
|
||||
finally:
|
||||
conn.close()
|
||||
else:
|
||||
row = None
|
||||
if row is not None and int(row[0]) > 0:
|
||||
n = int(row[0])
|
||||
return {
|
||||
"seeded": False, "reason": f"already has {n} rows, skip",
|
||||
"error": None, "rows": n,
|
||||
}
|
||||
|
||||
rows_en, rows_zh, _ = extract_rows_from_xlsx(source)
|
||||
counts = upsert_to_db(db, rows_en, rows_zh)
|
||||
return {
|
||||
"seeded": True, "reason": None, "error": None,
|
||||
"rows": counts["total"], "counts": counts,
|
||||
}
|
||||
except JournalSeedError as exc:
|
||||
return {"seeded": False, "reason": None, "error": str(exc), "rows": None}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"seeded": False, "reason": None, "error": str(exc), "rows": None}
|
||||
@@ -1,236 +0,0 @@
|
||||
"""services/latex_compile.py —— tectonic 编译 .tex → PDF（P3a）。
|
||||
|
||||
隔离与安全（对齐设计清单）：
|
||||
1. 一编译一目录：{reformat_dir}/{anchor_id}/compile/（input.tex/output.pdf/output.log/meta.json）
|
||||
2. 文件名永不落盘：用户文件名只做后缀校验，写入一律 input.tex
|
||||
3. tectonic --untrusted（官方旗标：禁用已知不安全特性，含 shell escape 路径）
|
||||
4. 子进程 start_new_session + 超时 killpg(SIGKILL)（整进程组）
|
||||
5. 单槽队列：asyncio.Lock + pending 计数，超出 compile_queue_limit → QueueFullError
|
||||
6. 产物 PDF > max_compile_pdf_bytes → 删文件 + OutputTooLargeError
|
||||
7. 启动/每次编译前机会性清扫 retention_days 天前的 job 目录（只删 12 位 hex 锚点目录）
|
||||
|
||||
tectonic 二进制缺失 → ToolUnavailableError（路由层回安装指引）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import signal
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from ..config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_ANCHOR_ID_RE = re.compile(r"^[0-9a-f]{12}$")
|
||||
_ENGINE = "tectonic-p3/0.17.0"
|
||||
_LOG_TAIL_LINES = 200
|
||||
|
||||
# 单槽并发控制（进程内；重启归零可接受——本服务无持久队列需求）
|
||||
_lock = asyncio.Lock()
|
||||
_pending = 0
|
||||
|
||||
|
||||
class CompileError(Exception):
|
||||
"""编译失败（含 log 尾）。code ∈ {COMPILE_FAILED, TIMEOUT, OUTPUT_TOO_LARGE}。"""
|
||||
|
||||
def __init__(self, code: str, message: str, log_tail: str = "") -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.log_tail = log_tail
|
||||
|
||||
|
||||
class QueueFullError(Exception):
|
||||
"""等待槽位超限。"""
|
||||
|
||||
|
||||
class ToolUnavailableError(Exception):
|
||||
"""tectonic 二进制缺失/不可执行。"""
|
||||
|
||||
|
||||
def _tectonic_bin() -> Path:
|
||||
return get_settings().tectonic_abs
|
||||
|
||||
|
||||
def _ensure_tool() -> Path:
|
||||
bin_path = _tectonic_bin()
|
||||
if not (bin_path.is_file() and os.access(bin_path, os.X_OK)):
|
||||
raise ToolUnavailableError(
|
||||
f"tectonic 未安装（{bin_path}）。"
|
||||
"请执行：cd review_service && bash tools/install_tectonic.sh"
|
||||
)
|
||||
return bin_path
|
||||
|
||||
|
||||
def _log_tail(path: Path, lines: int = _LOG_TAIL_LINES) -> str:
|
||||
try:
|
||||
text = path.read_text("utf-8", errors="replace")
|
||||
except OSError:
|
||||
return ""
|
||||
return "\n".join(text.splitlines()[-lines:])
|
||||
|
||||
|
||||
def compile_dir(anchor_id: str) -> Path:
|
||||
"""{reformat_dir}/{anchor_id}/compile/（存在性由调用方按需 mkdir）。"""
|
||||
return get_settings().reformat_dir_abs / anchor_id / "compile"
|
||||
|
||||
|
||||
def prepare_input(anchor_id: str, source: str) -> Path:
|
||||
"""写入隔离目录 input.tex；返回其路径。"""
|
||||
cdir = compile_dir(anchor_id)
|
||||
cdir.mkdir(parents=True, exist_ok=True)
|
||||
input_tex = cdir / "input.tex"
|
||||
input_tex.write_text(source, encoding="utf-8")
|
||||
return input_tex
|
||||
|
||||
|
||||
async def compile_tex(anchor_id: str) -> dict[str, Any]:
|
||||
"""同步编译 anchor 的 input.tex（须先 prepare_input）。
|
||||
|
||||
→ {compile_id: anchor_id, pdf_path, log_path, seconds, pdf_bytes}
|
||||
失败抛 CompileError / QueueFullError / ToolUnavailableError。
|
||||
"""
|
||||
global _pending
|
||||
settings = get_settings()
|
||||
_ensure_tool() # 快速失败：不占队列
|
||||
sweep_expired()
|
||||
|
||||
if _pending >= settings.compile_queue_limit:
|
||||
raise QueueFullError(
|
||||
f"编译队列已满（{_pending} 个任务排队），请稍后重试"
|
||||
)
|
||||
_pending += 1
|
||||
try:
|
||||
async with _lock:
|
||||
return await _run_once(anchor_id)
|
||||
finally:
|
||||
_pending -= 1
|
||||
|
||||
|
||||
async def _run_once(anchor_id: str) -> dict[str, Any]:
|
||||
settings = get_settings()
|
||||
cdir = compile_dir(anchor_id)
|
||||
input_tex = cdir / "input.tex"
|
||||
pdf_path = cdir / "output.pdf"
|
||||
log_path = cdir / "output.log"
|
||||
if not input_tex.is_file():
|
||||
raise CompileError("COMPILE_FAILED", "编译输入不存在，请重新上传源码")
|
||||
|
||||
env = dict(os.environ)
|
||||
env["TECTONIC_CACHE_DIR"] = str(settings.tectonic_cache_abs)
|
||||
cmd = [
|
||||
str(_tectonic_bin()), "-X", "compile", input_tex.name,
|
||||
"--untrusted", "--keep-logs", "--outdir", ".",
|
||||
]
|
||||
started = time.monotonic()
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd, cwd=cdir, env=env,
|
||||
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
|
||||
start_new_session=True,
|
||||
)
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
proc.communicate(), timeout=settings.compile_timeout_s
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
_kill_group(proc)
|
||||
try:
|
||||
await asyncio.wait_for(proc.wait(), timeout=5.0)
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
log_path.write_text(_log_tail(log_path) + "\n[timeout] 编译超时，进程已终止",
|
||||
encoding="utf-8")
|
||||
raise CompileError(
|
||||
"TIMEOUT",
|
||||
f"编译超过 {settings.compile_timeout_s:g} 秒未完成，已终止",
|
||||
_log_tail(log_path),
|
||||
) from None
|
||||
except FileNotFoundError as exc:
|
||||
raise ToolUnavailableError(f"tectonic 不可执行：{exc}") from exc
|
||||
finally:
|
||||
seconds = round(time.monotonic() - started, 2)
|
||||
|
||||
# tectonic --keep-logs 把日志写在 cwd 的 input.log
|
||||
produced_log = cdir / "input.log"
|
||||
if produced_log.is_file():
|
||||
shutil.move(str(produced_log), str(log_path))
|
||||
if not log_path.is_file():
|
||||
log_path.write_text("", encoding="utf-8")
|
||||
|
||||
produced_pdf = cdir / "input.pdf"
|
||||
if not produced_pdf.is_file():
|
||||
raise CompileError(
|
||||
"COMPILE_FAILED",
|
||||
"编译失败（未产出 PDF），请查看日志定位 LaTeX 错误",
|
||||
_log_tail(log_path),
|
||||
)
|
||||
pdf_bytes = produced_pdf.stat().st_size
|
||||
if pdf_bytes > settings.max_compile_pdf_bytes:
|
||||
produced_pdf.unlink(missing_ok=True)
|
||||
raise CompileError(
|
||||
"OUTPUT_TOO_LARGE",
|
||||
f"产出 PDF 超过 {settings.max_compile_pdf_bytes // (1024 * 1024)}MB 上限，已删除",
|
||||
_log_tail(log_path),
|
||||
)
|
||||
produced_pdf.rename(pdf_path)
|
||||
|
||||
meta = {
|
||||
"compile_id": anchor_id,
|
||||
"engine": _ENGINE,
|
||||
"seconds": seconds,
|
||||
"pdf_bytes": pdf_bytes,
|
||||
"returncode": 0,
|
||||
"compiled_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||
}
|
||||
(cdir / "meta.json").write_text(
|
||||
json.dumps(meta, ensure_ascii=False, indent=2), encoding="utf-8"
|
||||
)
|
||||
logger.info("compile: %s → PDF %d bytes, %.2fs", anchor_id, pdf_bytes, seconds)
|
||||
return {
|
||||
"compile_id": anchor_id,
|
||||
"pdf_path": pdf_path,
|
||||
"log_path": log_path,
|
||||
"seconds": seconds,
|
||||
"pdf_bytes": pdf_bytes,
|
||||
}
|
||||
|
||||
|
||||
def _kill_group(proc: asyncio.subprocess.Process) -> None:
|
||||
try:
|
||||
os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
|
||||
except (ProcessLookupError, PermissionError, OSError):
|
||||
try:
|
||||
proc.kill()
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
|
||||
|
||||
def sweep_expired() -> int:
|
||||
"""删除 mtime 超 retention_days 的 12-hex 锚点目录（编译产物+P1 落盘一并清）。"""
|
||||
settings = get_settings()
|
||||
base = settings.reformat_dir_abs
|
||||
if not base.is_dir():
|
||||
return 0
|
||||
cutoff = time.time() - settings.compile_retention_days * 86400
|
||||
removed = 0
|
||||
for child in base.iterdir():
|
||||
if not child.is_dir() or not _ANCHOR_ID_RE.fullmatch(child.name):
|
||||
continue
|
||||
try:
|
||||
if child.stat().st_mtime < cutoff:
|
||||
shutil.rmtree(child, ignore_errors=True)
|
||||
removed += 1
|
||||
except OSError:
|
||||
continue
|
||||
if removed:
|
||||
logger.info("compile sweep: 清理 %d 个过期任务目录（>%d 天）",
|
||||
removed, settings.compile_retention_days)
|
||||
return removed
|
||||
@@ -1,203 +0,0 @@
|
||||
"""services/llm.py —— LLM 调用（httpx 裸调 DeepSeek，模式同 paper_writer_service）。
|
||||
|
||||
- complete_text(system, user, *, temperature=0.3, max_tokens=8000, timeout=None) -> str
|
||||
- complete_json(...)（response_format json_object + 一次 JSON 修复重试）
|
||||
- 4 次尝试（1.5s / 3s / 6s 指数退避，429 / 5xx / 超时）
|
||||
- env 键（优先 REVIEW_LLM_*，回退 DOCGEN_LLM_*，再回退内置默认）：
|
||||
*_LLM_BASE_URL / *_LLM_API_KEY / *_LLM_MODEL / *_LLM_TIMEOUT
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_LLM_BASE_URL = "https://api.deepseek.com/v1"
|
||||
DEFAULT_LLM_MODEL = "deepseek-chat"
|
||||
DEFAULT_LLM_TIMEOUT = 300.0
|
||||
BACKOFF_SECONDS = (1.5, 3.0, 6.0)
|
||||
MAX_ATTEMPTS = 4
|
||||
|
||||
|
||||
class LLMError(Exception):
|
||||
"""LLM 调用失败（重试尽/响应异常）。"""
|
||||
|
||||
|
||||
def _env(keys: tuple[str, ...], default: str = "") -> str:
|
||||
for key in keys:
|
||||
val = os.environ.get(key)
|
||||
if val:
|
||||
return val
|
||||
return default
|
||||
|
||||
|
||||
def _llm_config() -> dict[str, Any]:
|
||||
return {
|
||||
"base_url": _env(
|
||||
("REVIEW_LLM_BASE_URL", "DOCGEN_LLM_BASE_URL"), DEFAULT_LLM_BASE_URL
|
||||
).rstrip("/"),
|
||||
"api_key": _env(("REVIEW_LLM_API_KEY", "DOCGEN_LLM_API_KEY"), ""),
|
||||
"model": _env(("REVIEW_LLM_MODEL", "DOCGEN_LLM_MODEL"), DEFAULT_LLM_MODEL),
|
||||
"timeout": float(
|
||||
_env(("REVIEW_LLM_TIMEOUT", "DOCGEN_LLM_TIMEOUT"), str(DEFAULT_LLM_TIMEOUT))
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _extract_text(payload: dict[str, Any]) -> str:
|
||||
try:
|
||||
choices = payload.get("choices") or []
|
||||
if choices:
|
||||
content = choices[0].get("message", {}).get("content")
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
return "".join(
|
||||
str(c.get("text", "")) if isinstance(c, dict) else str(c) for c in content
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
return ""
|
||||
|
||||
|
||||
async def _chat_once(
|
||||
client: httpx.AsyncClient,
|
||||
cfg: dict[str, Any],
|
||||
system: str,
|
||||
user: str,
|
||||
*,
|
||||
temperature: float,
|
||||
max_tokens: int,
|
||||
json_mode: bool,
|
||||
timeout: float | None,
|
||||
) -> str:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if cfg["api_key"]:
|
||||
headers["Authorization"] = f"Bearer {cfg['api_key']}"
|
||||
body: dict[str, Any] = {
|
||||
"model": cfg["model"],
|
||||
"messages": [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
],
|
||||
"temperature": temperature,
|
||||
"max_tokens": max_tokens,
|
||||
}
|
||||
if json_mode:
|
||||
body["response_format"] = {"type": "json_object"}
|
||||
resp = await client.post(
|
||||
f"{cfg['base_url']}/chat/completions",
|
||||
headers=headers,
|
||||
json=body,
|
||||
timeout=timeout if timeout is not None else cfg["timeout"],
|
||||
)
|
||||
resp.raise_for_status()
|
||||
payload = resp.json()
|
||||
if not isinstance(payload, dict):
|
||||
raise LLMError("LLM 响应形态异常（非 JSON 对象）")
|
||||
text = _extract_text(payload)
|
||||
if not text:
|
||||
raise LLMError("LLM 响应缺少 choices[0].message.content")
|
||||
return text
|
||||
|
||||
|
||||
async def _complete(
|
||||
system: str,
|
||||
user: str,
|
||||
*,
|
||||
temperature: float,
|
||||
max_tokens: int,
|
||||
json_mode: bool,
|
||||
timeout: float | None,
|
||||
) -> str:
|
||||
cfg = _llm_config()
|
||||
last_exc: BaseException | None = None
|
||||
async with httpx.AsyncClient() as client:
|
||||
for attempt in range(MAX_ATTEMPTS):
|
||||
try:
|
||||
return await _chat_once(
|
||||
client, cfg, system, user,
|
||||
temperature=temperature, max_tokens=max_tokens, json_mode=json_mode,
|
||||
timeout=timeout,
|
||||
)
|
||||
except LLMError:
|
||||
raise
|
||||
except (httpx.HTTPStatusError, httpx.HTTPError, httpx.TimeoutException) as exc:
|
||||
last_exc = exc
|
||||
status = getattr(getattr(exc, "response", None), "status_code", None)
|
||||
retryable = status is None or status == 429 or status >= 500
|
||||
if not retryable or attempt == MAX_ATTEMPTS - 1:
|
||||
raise LLMError(f"LLM 调用失败（HTTP {status or '网络'}）：{exc}") from exc
|
||||
await asyncio.sleep(BACKOFF_SECONDS[min(attempt, len(BACKOFF_SECONDS) - 1)])
|
||||
raise LLMError(f"LLM 调用失败：{last_exc}")
|
||||
|
||||
|
||||
async def complete_text(
|
||||
system: str,
|
||||
user: str,
|
||||
*,
|
||||
temperature: float = 0.3,
|
||||
max_tokens: int = 8000,
|
||||
timeout: float | None = None,
|
||||
) -> str:
|
||||
"""文本补全。失败抛 LLMError。"""
|
||||
return await _complete(
|
||||
system, user, temperature=temperature, max_tokens=max_tokens, json_mode=False,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def _repair_json(text: str) -> str:
|
||||
text = text.strip()
|
||||
if text.startswith("```"):
|
||||
text = text.strip("`")
|
||||
if text.lower().startswith("json"):
|
||||
text = text[4:]
|
||||
text = text.strip()
|
||||
if not text:
|
||||
raise LLMError("LLM 返回空内容")
|
||||
if text.startswith("{"):
|
||||
return text
|
||||
start = text.find("{")
|
||||
end = text.rfind("}")
|
||||
if start >= 0 and end > start:
|
||||
return text[start:end + 1]
|
||||
raise LLMError("LLM 输出中未找到 JSON 对象")
|
||||
|
||||
|
||||
async def complete_json(
|
||||
system: str,
|
||||
user: str,
|
||||
*,
|
||||
temperature: float = 0.3,
|
||||
max_tokens: int = 8000,
|
||||
timeout: float | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""JSON 补全（json_object + 一次修复重试）。"""
|
||||
text = await _complete(
|
||||
system, user, temperature=temperature, max_tokens=max_tokens, json_mode=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
try:
|
||||
return json.loads(text)
|
||||
except ValueError:
|
||||
pass
|
||||
text2 = await _complete(
|
||||
system + "\n（重要：你的回答必须是单个合法 JSON 对象，不要输出其他任何内容）",
|
||||
user,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
json_mode=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
try:
|
||||
return json.loads(text2)
|
||||
except ValueError:
|
||||
repaired = _repair_json(text2)
|
||||
return json.loads(repaired)
|
||||
@@ -1,139 +0,0 @@
|
||||
"""paper_parse.py —— PDF → 纯文本（经 vstore 8094 现有解析接口，零改动复用）。
|
||||
|
||||
链路（全部只读消费 vstore 既有 API，不修改 vstore 代码）：
|
||||
1. POST {vstore}/api/vstore/ingest multipart file（vstore 内部 pymupdf 解析）
|
||||
2. GET {vstore}/api/vstore/documents/{doc_id} → document.paragraphs 拼接纯文本
|
||||
3. DELETE {vstore}/api/vstore/documents/{doc_id} （临时解析件 best-effort 清理，
|
||||
REVIEW_PARSE_DELETE_AFTER=0 可关闭——保留文档便于人工排查）
|
||||
|
||||
为何不用 methodology 8092：其 /extract-upload 是 MinerU 分钟级异步 job
|
||||
（MinerU 子进程 + LLM 抽取两阶段），不适配同步自审稿；vstore 段落解析秒级
|
||||
返回，满足 presubmit 场景。methodology 地址在 config 中预留供后续扩展。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from ..config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ParseError(Exception):
|
||||
"""PDF 解析失败（上游不可达/响应异常/空文本）。"""
|
||||
|
||||
|
||||
async def parse_pdf(path: str) -> str:
|
||||
"""接收 PDF 文件路径，返回拼接后的纯文本。失败抛 ParseError。"""
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
file_bytes = fh.read()
|
||||
except OSError as exc:
|
||||
raise ParseError(f"读取 PDF 失败：{exc}") from exc
|
||||
return await parse_pdf_bytes(file_bytes, filename=str(path).rsplit("/", 1)[-1] or "paper.pdf")
|
||||
|
||||
|
||||
async def parse_pdf_bytes(file_bytes: bytes, filename: str = "paper.pdf") -> str:
|
||||
"""接收 PDF 字节流，返回拼接后的纯文本。失败抛 ParseError。"""
|
||||
if not file_bytes:
|
||||
raise ParseError("PDF 内容为空")
|
||||
|
||||
settings = get_settings()
|
||||
base = settings.vstore_base
|
||||
timeout = httpx.Timeout(
|
||||
connect=5.0, read=settings.parse_timeout, write=settings.parse_timeout, pool=5.0
|
||||
)
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout) as client:
|
||||
# ---- 1) ingest（vstore 内部 pymupdf 解析 + 嵌入 + 加密落盘）----
|
||||
try:
|
||||
resp = await client.post(
|
||||
f"{base}/api/vstore/ingest",
|
||||
files={"file": (filename, file_bytes, "application/pdf")},
|
||||
data={
|
||||
"name": filename,
|
||||
"category": "review_presubmit",
|
||||
"description": "review_service 临时解析件",
|
||||
},
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
raise ParseError(f"vstore 不可达（{base}）：{exc}") from exc
|
||||
|
||||
if resp.status_code >= 400:
|
||||
# 读上游错误体（信封 JSON 或裸文本），区分“不可达”与“上游报错”
|
||||
raise ParseError(f"vstore ingest 失败（HTTP {resp.status_code}）：{_err_body(resp)}")
|
||||
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError as exc:
|
||||
raise ParseError(f"vstore ingest 响应非 JSON：{exc}") from exc
|
||||
|
||||
if not isinstance(payload, dict) or not payload.get("ok"):
|
||||
message = _err_message(payload) or "vstore ingest 失败"
|
||||
raise ParseError(message)
|
||||
doc_id = str((payload.get("item") or {}).get("document_id") or "")
|
||||
if not doc_id:
|
||||
raise ParseError("vstore ingest 响应缺少 document_id")
|
||||
|
||||
# ---- 2) 取明文全文（paragraphs 拼接）+ 3) 清理临时件 ----
|
||||
try:
|
||||
text = await _fetch_text(client, base, doc_id)
|
||||
finally:
|
||||
if settings.parse_delete_after:
|
||||
await _best_effort_delete(client, base, doc_id)
|
||||
|
||||
text = text.strip()
|
||||
if not text:
|
||||
raise ParseError("PDF 解析结果为空（可能是扫描件/图片型 PDF，vstore 无 OCR）")
|
||||
return text
|
||||
|
||||
|
||||
async def _fetch_text(client: httpx.AsyncClient, base: str, doc_id: str) -> str:
|
||||
"""取 vstore 文档明文并按段落拼接纯文本。"""
|
||||
try:
|
||||
resp = await client.get(f"{base}/api/vstore/documents/{doc_id}")
|
||||
except httpx.HTTPError as exc:
|
||||
raise ParseError(f"vstore 取文档失败（{doc_id}）：{exc}") from exc
|
||||
if resp.status_code >= 400:
|
||||
raise ParseError(
|
||||
f"vstore 取文档失败（{doc_id}，HTTP {resp.status_code}）：{_err_body(resp)}"
|
||||
)
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError as exc:
|
||||
raise ParseError(f"vstore 取文档响应非 JSON（{doc_id}）：{exc}") from exc
|
||||
if not isinstance(payload, dict) or not payload.get("ok"):
|
||||
raise ParseError(_err_message(payload) or f"vstore 取文档失败（{doc_id}）")
|
||||
|
||||
document: dict[str, Any] = payload.get("document") or {}
|
||||
paragraphs = document.get("paragraphs") or []
|
||||
return "\n".join(str(p) for p in paragraphs if str(p).strip())
|
||||
|
||||
|
||||
async def _best_effort_delete(client: httpx.AsyncClient, base: str, doc_id: str) -> None:
|
||||
try:
|
||||
await client.delete(f"{base}/api/vstore/documents/{doc_id}")
|
||||
except Exception: # noqa: BLE001 - 清理失败不阻断（文档级 secure_delete 已尽力）
|
||||
logger.warning("paper_parse: vstore 临时文档清理失败（%s）", doc_id, exc_info=True)
|
||||
|
||||
|
||||
def _err_message(payload: Any) -> str:
|
||||
if isinstance(payload, dict):
|
||||
err = payload.get("error")
|
||||
if isinstance(err, dict):
|
||||
return str(err.get("message") or "")
|
||||
if isinstance(err, str):
|
||||
return err
|
||||
return ""
|
||||
|
||||
|
||||
def _err_body(resp: httpx.Response) -> str:
|
||||
"""上游错误体：优先 JSON 信封 message，回退裸文本（截断 500 字符）。"""
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError:
|
||||
return (resp.text or "").strip()[:500] or "（空响应体）"
|
||||
return _err_message(payload) or str(payload)[:500]
|
||||
@@ -1,226 +0,0 @@
|
||||
"""services/paper_scanner.py —— 关键词兜底扫描器（纯 stdlib，零 LLM）。
|
||||
|
||||
背景（2026-10-12）：hard_rule 标签由 LLM 输出，存在"正文明明有 accession/BUSCO/挂载率
|
||||
却报缺失"的漏检风险（真实案例 s41588 Nature Genetics：正文有 PRJCA038097 等 +
|
||||
BUSCO 98.57% + 94.18% anchored，仍被判 busco/accession/hic_missing → 大修）。
|
||||
本模块做**程序层证伪**：LLM 说缺某项、扫描器在文中找到该项 → 剥掉 hard_rule
|
||||
降级为非硬性 major，并在 rationale 记录冲突（前端黄色提示，请人工核对）。
|
||||
|
||||
边界（刻意保守，不引入新误判）：
|
||||
- 只做"证伪硬性缺项"，不做"证实"——扫不到不等于真缺（可能同义表述），不升级任何条目
|
||||
- accession 只在 Data Availability 段内扫（用户指令）：正文引用他人数据的
|
||||
accession 不得误认为"自己有"
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
# ---- accession（只在 DA 段扫）----
|
||||
ACCESSION_RE = re.compile(
|
||||
r"\b(PRJNA|PRJEB|PRJCA|SRP|ERP|DRP|CRA|SRR|ERR|DRR|GCA_|GCF_)\s?(\d[\d._]*)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# ---- DA 段定位：标题多变体（行首允许编号前缀）----
|
||||
_DA_START_RE = re.compile(
|
||||
r"^[ \t]*(?:\d+(?:\.\d+)*\.?\s+)?"
|
||||
r"(?:Data\s+(?:and\s+(?:code\s+)?)?[Aa]vailability"
|
||||
r"|Availability\s+of\s+(?:data|Data)"
|
||||
r"|Data\s+and\s+[Mm]aterials\s+[Aa]vailability"
|
||||
r"|Data\s+availability\s+statement)",
|
||||
re.MULTILINE,
|
||||
)
|
||||
# DA 段结束 = 下一个一级节标题
|
||||
_NEXT_HEADER_RE = re.compile(
|
||||
r"^[ \t]*(?:\d+(?:\.\d+)*\.?\s+)?"
|
||||
r"(?:Abstract|Summary|Introduction|Results|Discussion|Methods"
|
||||
r"|Materials\s+and\s+Methods|References|Acknowledg(?:e)?ments?"
|
||||
r"|Author\s+[Ii]nformation|Rights\s+and\s+permissions|Change\s+history"
|
||||
r"|Data\s+(?:and\s+(?:code\s+)?)?[Aa]vailability"
|
||||
r"|Availability\s+of\s+[Dd]ata|Data\s+and\s+[Mm]aterials\s+[Aa]vailability)"
|
||||
r"\s*$",
|
||||
re.MULTILINE,
|
||||
)
|
||||
# 回退窗：DA 无行首标题时，短语出现处 + 上下文
|
||||
_DA_PHRASE_RE = re.compile(r"data\s+availability", re.IGNORECASE)
|
||||
_DA_FALLBACK_WINDOW = 3000
|
||||
|
||||
# ---- BUSCO（全文）----
|
||||
_BUSCO_MENTION_RE = re.compile(r"\bBUSCO\b", re.IGNORECASE)
|
||||
_BUSCO_FULL_RE = re.compile(
|
||||
r"(?:\bS\s*:\s*\d|\bSingle[- ]copy)"
|
||||
r".{0,200}?"
|
||||
r"(?:\bD\s*:\s*\d|\bDuplicat)",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
|
||||
# ---- Hi-C 挂载率（全文）----
|
||||
HIC_RE = re.compile(
|
||||
r"(anchored\s+to\s+chromosomes?"
|
||||
r"|anchoring\s+rate"
|
||||
r"|pseudo[- ]?chromosomes?"
|
||||
r"|chromosome[- ]level"
|
||||
r"|%[^.\n]{0,40}anchored)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
_MAX_HITS = 20
|
||||
|
||||
|
||||
def _extract_da_section(text: str) -> str:
|
||||
"""定位 Data Availability 段正文（标题后 → 下一个一级节标题）。
|
||||
|
||||
无行首标题时回退：短语出现处向后 3000 字符窗。均未命中 → ""。
|
||||
"""
|
||||
m = _DA_START_RE.search(text)
|
||||
if m:
|
||||
nxt = _NEXT_HEADER_RE.search(text, m.end())
|
||||
end = nxt.start() if nxt else min(len(text), m.end() + _DA_FALLBACK_WINDOW)
|
||||
return text[m.start():end]
|
||||
m2 = _DA_PHRASE_RE.search(text)
|
||||
if m2:
|
||||
return text[m2.start():min(len(text), m2.start() + _DA_FALLBACK_WINDOW)]
|
||||
return ""
|
||||
|
||||
|
||||
def _scan_accession(da_text: str) -> dict[str, Any]:
|
||||
if not da_text:
|
||||
return {"found": False, "hits": [], "in_da_section": False}
|
||||
hits: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for m in ACCESSION_RE.finditer(da_text):
|
||||
token = (m.group(1) + m.group(2)).upper()
|
||||
if token not in seen:
|
||||
seen.add(token)
|
||||
hits.append(token)
|
||||
if len(hits) >= _MAX_HITS:
|
||||
break
|
||||
return {"found": bool(hits), "hits": hits, "in_da_section": True}
|
||||
|
||||
|
||||
def _scan_busco(text: str) -> dict[str, Any]:
|
||||
m = _BUSCO_MENTION_RE.search(text)
|
||||
if not m:
|
||||
return {"level": "none", "excerpt": ""}
|
||||
excerpt = text[m.start():m.start() + 200]
|
||||
if _BUSCO_FULL_RE.search(text):
|
||||
level = "full"
|
||||
else:
|
||||
level = "partial" # 只有百分比、无 S/D 分项 → prompt 已定为 minor
|
||||
return {"level": level, "excerpt": excerpt}
|
||||
|
||||
|
||||
def _scan_hic(text: str) -> dict[str, Any]:
|
||||
m = HIC_RE.search(text)
|
||||
if not m:
|
||||
return {"found": False, "excerpt": ""}
|
||||
start = max(0, m.start() - 40)
|
||||
return {"found": True, "excerpt": text[start:m.end() + 80]}
|
||||
|
||||
|
||||
def scan_paper_signals(text: str) -> dict[str, Any]:
|
||||
"""全文 → 结构化信号。accession 只在 DA 段；busco/hic 全文。"""
|
||||
if not text:
|
||||
return {
|
||||
"accession": {"found": False, "hits": [], "in_da_section": False},
|
||||
"busco": {"level": "none", "excerpt": ""},
|
||||
"hic": {"found": False, "excerpt": ""},
|
||||
}
|
||||
return {
|
||||
"accession": _scan_accession(_extract_da_section(text)),
|
||||
"busco": _scan_busco(text),
|
||||
"hic": _scan_hic(text),
|
||||
}
|
||||
|
||||
|
||||
def _get_hr(item: Any) -> Any:
|
||||
if isinstance(item, dict):
|
||||
return item.get("hard_rule")
|
||||
return getattr(item, "hard_rule", None)
|
||||
|
||||
|
||||
def _item_fatal(item: Any) -> bool:
|
||||
if isinstance(item, dict):
|
||||
return bool(item.get("fatal"))
|
||||
return bool(getattr(item, "fatal", False))
|
||||
|
||||
|
||||
def _set_downgraded(item: Any) -> None:
|
||||
if isinstance(item, dict):
|
||||
item["scanner_downgraded"] = True
|
||||
else:
|
||||
try:
|
||||
item.scanner_downgraded = True
|
||||
except Exception: # pydantic 严格模式等极端场景：吞掉，冲突记录仍保留
|
||||
pass
|
||||
|
||||
|
||||
def _set_hr_none(item: Any) -> None:
|
||||
if isinstance(item, dict):
|
||||
item["hard_rule"] = None
|
||||
else:
|
||||
try:
|
||||
item.hard_rule = None
|
||||
except Exception: # pydantic 严格模式等极端场景：吞掉，冲突记录仍保留
|
||||
pass
|
||||
|
||||
|
||||
def _item_title(item: Any) -> str:
|
||||
if isinstance(item, dict):
|
||||
return str(item.get("title") or "")
|
||||
return str(getattr(item, "title", "") or "")
|
||||
|
||||
|
||||
# 冲突映射：hard_rule 标签 → (信号检查, 冲突描述)
|
||||
def _conflict_for(tag: str, signals: dict[str, Any]) -> str | None:
|
||||
acc = signals.get("accession") or {}
|
||||
busco = signals.get("busco") or {}
|
||||
hic = signals.get("hic") or {}
|
||||
if tag == "accession_missing" and acc.get("found"):
|
||||
hits = ", ".join(acc.get("hits") or [])[:120]
|
||||
return f"Data Availability 段已检出 accession（{hits}）"
|
||||
if tag == "busco_missing" and busco.get("level") in ("partial", "full"):
|
||||
return f"正文已提及 BUSCO（{busco.get('level')} 格式）"
|
||||
if tag == "hic_missing" and hic.get("found"):
|
||||
excerpt = (hic.get("excerpt") or "").strip()[:80]
|
||||
return f"正文已检出挂载率描述（{excerpt}）"
|
||||
return None
|
||||
|
||||
|
||||
def reconcile_hard_rules(
|
||||
majors: list[Any], signals: dict[str, Any]
|
||||
) -> tuple[list[Any], list[Any], list[dict[str, Any]]]:
|
||||
"""对带 hard_rule 的 major 做扫描器证伪：命中 → 剥标签 + **移出 major 降级至
|
||||
minor**（A2 语义，2026-10-12）→ 记冲突。
|
||||
|
||||
返回 (kept_majors, demoted_minors, conflicts)：
|
||||
- kept_majors：未冲突/未处理的 major（原序）
|
||||
- demoted_minors：被冲突降级的条目（hard_rule 置 None、scanner_downgraded=True）
|
||||
- conflicts：{hard_rule, title, reason}——fatal 条目只记冲突不降级
|
||||
（拒稿级问题须人工过目，扫描器证伪不了伦理判断）。
|
||||
"""
|
||||
kept: list[Any] = []
|
||||
demoted: list[Any] = []
|
||||
conflicts: list[dict[str, Any]] = []
|
||||
for m in majors:
|
||||
tag = _get_hr(m)
|
||||
if not isinstance(tag, str):
|
||||
kept.append(m)
|
||||
continue
|
||||
reason = _conflict_for(tag, signals)
|
||||
if reason is None:
|
||||
kept.append(m)
|
||||
continue
|
||||
conflicts.append({
|
||||
"hard_rule": tag,
|
||||
"title": _item_title(m),
|
||||
"reason": reason,
|
||||
})
|
||||
if _item_fatal(m):
|
||||
kept.append(m) # fatal 保留原位（含其 hard_rule 供引擎 R1/R0 使用）
|
||||
continue
|
||||
_set_hr_none(m)
|
||||
_set_downgraded(m)
|
||||
demoted.append(m)
|
||||
return kept, demoted, conflicts
|
||||
@@ -1,153 +0,0 @@
|
||||
"""services/rebuttal_draft.py —— 5 类回复片段 Prompt + 反幻觉校验 + letter 拼装。
|
||||
|
||||
- part.type 枚举（决策 2，无 auto）：new_data | reanalysis | location | clarification | rebuttal。
|
||||
- 反幻觉校验（补充 2 收窄版，只做负向证伪，不做泛化打分）：
|
||||
* location：出现 "shows that / as shown" 但无 "added/新增" 迹证 → FAIL
|
||||
（"S3 shows that..." 红 FAIL；"we have added ... in S3" 过）。
|
||||
* rebuttal：命中负向词表 we agree / we accept / we concede / you are right /
|
||||
we were wrong → FAIL；无立场词（maintain/disagree/respectfully/坚持/不同意…）
|
||||
→ FAIL；"respectfully maintain" 过。
|
||||
* 其他三类：Phase 1 仅查非空。
|
||||
- 校验前置（决策：校验C前置）：draft 生成前先跑校验，FAIL 则不调 LLM。
|
||||
- letter 拼装：按 reviewer → comment 顺序，原文 + 草稿；无草稿留 "[待补充]"。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any, Callable
|
||||
|
||||
LLMCallable = Callable[..., Any]
|
||||
|
||||
PART_TYPES = ("new_data", "reanalysis", "location", "clarification", "rebuttal")
|
||||
|
||||
_TYPE_FRAGMENTS: dict[str, str] = {
|
||||
"new_data": (
|
||||
"你的回复属于【补充新数据】类：说明新增了哪些实验/数据（图/表编号、关键数值），"
|
||||
"并明确回应原审稿关切。不得虚构未提供的数据——只用 user_response 给出的事实。"
|
||||
),
|
||||
"reanalysis": (
|
||||
"你的回复属于【重新分析】类：说明重跑了哪些分析、方法/参数变化与新结论要点。"
|
||||
"不得虚构未提供的结果——只用 user_response 给出的事实。"
|
||||
),
|
||||
"location": (
|
||||
"你的回复属于【定位/指出位置】类：明确指出修改或证据在稿件中的位置"
|
||||
"（如 Section 3 / Fig. S2 / Methods 段落），与 user_response 给出的位置一致。"
|
||||
"不得虚构未提供的位置。"
|
||||
),
|
||||
"clarification": (
|
||||
"你的回复属于【澄清说明】类：针对误解给出准确、简洁的解释，"
|
||||
"只用 user_response 与原评论内容，不引入新主张。"
|
||||
),
|
||||
"rebuttal": (
|
||||
"你的回复属于【礼貌反驳】类：礼貌但明确地表达不同立场（maintain/disagree），"
|
||||
"给出理由（基于 user_response 与原文），不得虚构证据。"
|
||||
),
|
||||
}
|
||||
|
||||
_BASELINE = (
|
||||
"通用底线：只基于原评论与 user_response 起草；不得编造数据、图表编号、"
|
||||
"统计结果或已做修改；不确定处用占位（[XX]）而非猜测。"
|
||||
"输出 JSON：{\"draft_md\": \"英文正式回复正文（markdown，≤200词）\"}。只输出 JSON。"
|
||||
)
|
||||
|
||||
_STANCE = re.compile(
|
||||
r"\b(maintain|disagree|respectfully|cannot\s+agree|dispute|stand\s+by|"
|
||||
r"hold\s+to)\b|坚持|不同意|保留|异议",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_NEGATIVE = re.compile(
|
||||
r"\bwe\s+agree\b|\bwe\s+accept\b|\bwe\s+concede\b|\byou\s+are\s+right\b|"
|
||||
r"\bwe\s+were\s+wrong\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_LOC_SHOWS = re.compile(r"shows?\s+that|as\s+shown|表明", re.IGNORECASE)
|
||||
_LOC_ADDED = re.compile(r"\badded\b|新增|have\s+added|we\s+have\s+added", re.IGNORECASE)
|
||||
|
||||
|
||||
def validate_parts(parts: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""反幻觉校验 → {passed: bool, findings: [{part_index, code, message}]}。"""
|
||||
findings: list[dict[str, Any]] = []
|
||||
for i, p in enumerate(parts):
|
||||
ptype = str(p.get("type") or "")
|
||||
content = str(p.get("content") or "").strip()
|
||||
if not content:
|
||||
findings.append({"part_index": i, "code": "EMPTY_CONTENT",
|
||||
"message": "回复内容不能为空"})
|
||||
continue
|
||||
if ptype == "location":
|
||||
if _LOC_SHOWS.search(content) and not _LOC_ADDED.search(content):
|
||||
findings.append({
|
||||
"part_index": i, "code": "LOCATION_EVIDENCE_WEAK",
|
||||
"message": "location 类回复出现“shows that/如所示”但未给出新增位置；"
|
||||
"请写明“we have added … in S{n}”式定位",
|
||||
})
|
||||
elif ptype == "rebuttal":
|
||||
if _NEGATIVE.search(content):
|
||||
findings.append({
|
||||
"part_index": i, "code": "REBUTTAL_NEGATIVE",
|
||||
"message": "rebuttal 类回复出现让步词（we agree/accept/concede…）；"
|
||||
"该类应礼貌但明确地表达不同立场",
|
||||
})
|
||||
elif not _STANCE.search(content):
|
||||
findings.append({
|
||||
"part_index": i, "code": "REBUTTAL_NO_STANCE",
|
||||
"message": "rebuttal 类回复缺少立场词（maintain/disagree/respectfully…）",
|
||||
})
|
||||
# new_data / reanalysis / clarification：Phase 1 仅非空
|
||||
return {"passed": not findings, "findings": findings}
|
||||
|
||||
|
||||
def _draft_system(parts: list[dict[str, Any]]) -> str:
|
||||
frags = [_TYPE_FRAGMENTS.get(str(p.get("type") or ""), "") for p in parts]
|
||||
frags = [f for f in frags if f]
|
||||
return " ".join(frags) + " " + _BASELINE if frags else _BASELINE
|
||||
|
||||
|
||||
async def generate_draft(llm: LLMCallable, comment: dict[str, Any]) -> str:
|
||||
"""按 comment.response.parts 起草 draft_md（调用方须先过 validate_parts）。"""
|
||||
parts = (comment.get("response") or {}).get("parts") or []
|
||||
user_lines = []
|
||||
for p in parts:
|
||||
hint = str(p.get("location_hint") or "").strip()
|
||||
line = f"- [{p.get('type')}] {p.get('content')}"
|
||||
if hint:
|
||||
line += f" (location_hint: {hint})"
|
||||
user_lines.append(line)
|
||||
user_msg = (
|
||||
f"原审稿意见：\n{comment.get('body') or comment.get('title') or ''}\n\n"
|
||||
f"作者拟回复要点（user_response）：\n" + "\n".join(user_lines)
|
||||
)
|
||||
try:
|
||||
data = await llm(
|
||||
[{"role": "system", "content": _draft_system(parts)},
|
||||
{"role": "user", "content": user_msg}],
|
||||
temperature=0.2,
|
||||
)
|
||||
draft = str((data or {}).get("draft_md") or "").strip()
|
||||
return draft or "[draft generation returned empty]"
|
||||
except Exception: # noqa: BLE001
|
||||
return "[draft generation failed — 请重试]"
|
||||
|
||||
|
||||
def assemble_letter(job: dict[str, Any]) -> str:
|
||||
"""拼装整封 response letter markdown（Phase 1 纯 .md，docx 放 Phase 2）。"""
|
||||
lines: list[str] = ["# Response to Reviewers", ""]
|
||||
lines.append(f"_Job: {job.get('id')} · template: {job.get('template') or 'generic'}_")
|
||||
lines.append("")
|
||||
for r in job.get("reviewers") or []:
|
||||
lines.append(f"## {r.get('label') or 'Reviewer'}")
|
||||
lines.append("")
|
||||
for c in r.get("comments") or []:
|
||||
title = c.get("title") or c.get("id") or "Comment"
|
||||
lines.append(f"### {title}")
|
||||
lines.append("")
|
||||
lines.append("**Original comment:**")
|
||||
lines.append("")
|
||||
lines.append(str(c.get("body") or "").strip())
|
||||
lines.append("")
|
||||
lines.append("**Our response:**")
|
||||
lines.append("")
|
||||
draft = str(c.get("draft") or "").strip()
|
||||
lines.append(draft if draft else "[待补充]")
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
@@ -1,207 +0,0 @@
|
||||
"""services/rebuttal_parse.py —— 审稿意见文本 → 结构化 Comment 树。
|
||||
|
||||
流程（保守：正则先切骨架，LLM 只做分类/拆子条，行数交叉核对不匹配则回退正则）：
|
||||
1. 归一化换行；正则识别 Reviewer 段头（Reviewer 1 / 审稿人 2 / Review A 等），
|
||||
无段头 → 单 Reviewer（label=Reviewer）。
|
||||
2. 每个 Reviewer 段内按编号行切 comment 段（"1." "2)" "（3）" 等）；
|
||||
切不出编号 → 整段作 1 条。
|
||||
3. LLM complete_json 对每段做：{title, severity, sub_items:[{title,body,confidence}]}。
|
||||
- LLM 返回段数 != 提交段数 → 全量回退正则（title=首 60 字，severity 默认 major）。
|
||||
- sub_items ≥ 2 且最高 confidence ≥ 0.7 → 拆多条（共享 group_id）；否则单条。
|
||||
4. cross_refs：正文里 "see comment 3 / 参见第 2 条" → 解析目标；解析不到
|
||||
target_id=null 保留 raw（决策：不猜）。
|
||||
|
||||
不做：auto 分类（已砍）；severity 以外的任何 LLM 自由发挥字段。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any, Callable
|
||||
|
||||
# LLM 依赖注入（测试打桩）：async (messages, temperature=..) -> dict
|
||||
LLMCallable = Callable[..., Any]
|
||||
|
||||
_REVIEWER_HEADER = re.compile(
|
||||
r"^(?:reviewers?\s*#?\s*([0-9]+|[A-Z])|审稿人\s*#?\s*([0-9A-Z]+)|review\s+([A-Z]))\s*[:：.）)]?",
|
||||
re.IGNORECASE | re.MULTILINE,
|
||||
)
|
||||
_COMMENT_NUM = re.compile(
|
||||
r"(?:(?<=\n)|(?<=\.\s)|^)\s*(?:[#(（]?\s*(\d{1,2})\s*[)）.:、]|[（(]\s*([ivx]{1,4}|[a-z])\s*[)）])\s+",
|
||||
re.IGNORECASE | re.MULTILINE,
|
||||
)
|
||||
_CROSS_REF = re.compile(
|
||||
r"(?:see|noted\s+in|refer(?:ring)?\s+to|per)\s+comment\s*#?\s*(\d+)|"
|
||||
r"(?:please\s+)?refer\s+to\s+comment\s*#?\s*(\d+)|"
|
||||
r"参见\s*第?\s*(\d+)\s*条|见\s*comment\s*#?\s*(\d+)|同上(?:一条)?评论",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
_SPLIT_CONFIDENCE = 0.7 # 决策 3：confidence ≥ 0.7 才拆
|
||||
|
||||
|
||||
def _norm(text: str) -> str:
|
||||
return text.replace("\r\n", "\n").replace("\r", "\n").strip()
|
||||
|
||||
|
||||
def split_reviewers(text: str) -> list[tuple[str, str]]:
|
||||
"""→ [(label, block_text)]；无段头 → [(Reviewer, 全文)]。"""
|
||||
text = _norm(text)
|
||||
matches = list(_REVIEWER_HEADER.finditer(text))
|
||||
if not matches:
|
||||
return [("Reviewer", text)]
|
||||
out: list[tuple[str, str]] = []
|
||||
for i, m in enumerate(matches):
|
||||
raw = next(g for g in m.groups() if g is not None)
|
||||
label = raw.upper() if raw.isalpha() else f"Reviewer {raw}"
|
||||
start = m.end()
|
||||
end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
|
||||
block = text[start:end].strip()
|
||||
if block:
|
||||
out.append((label, block))
|
||||
if not out: # 段头全空块 → 兜底
|
||||
return [("Reviewer", text)]
|
||||
# 段头前的引言（无段头内容）丢弃——保守：不猜归属
|
||||
return out
|
||||
|
||||
|
||||
def split_comments(block: str) -> list[str]:
|
||||
"""段内按编号行切 comment 块；无编号 → [整段]。"""
|
||||
block = _norm(block)
|
||||
matches = list(_COMMENT_NUM.finditer(block))
|
||||
if len(matches) < 2: # 单个编号或无编号 → 整段一条
|
||||
return [block] if block else []
|
||||
out: list[str] = []
|
||||
for i, m in enumerate(matches):
|
||||
start = m.end()
|
||||
end = matches[i + 1].start() if i + 1 < len(matches) else len(block)
|
||||
chunk = block[start:end].strip()
|
||||
if chunk:
|
||||
out.append(chunk)
|
||||
return out or [block]
|
||||
|
||||
|
||||
def _fallback_segment(seg: str) -> dict[str, Any]:
|
||||
title = re.sub(r"\s+", " ", seg)[:60]
|
||||
return {
|
||||
"title": title,
|
||||
"severity": "major",
|
||||
"sub_items": [{"title": title, "body": seg, "confidence": 1.0}],
|
||||
}
|
||||
|
||||
|
||||
_SEG_SYSTEM = (
|
||||
"你是学术审稿意见解析助手。输入是一段审稿意见原文。输出 JSON："
|
||||
'{"title": "≤12词标题", "severity": "major|minor|question", '
|
||||
'"sub_items": [{"title": "...", "body": "该子问题完整原文", "confidence": 0.0-1.0}]}。'
|
||||
"severity：major=要求补实验/补数据/重分析；minor=格式/表述小问题；"
|
||||
"question=仅提问、无需修改稿件。"
|
||||
"若该段含多个可独立回复的子问题则拆成多个 sub_items（confidence 给拆分把握度），"
|
||||
"否则 1 个 sub_items（confidence=1.0）。只输出 JSON。"
|
||||
)
|
||||
|
||||
|
||||
async def _ask_llm_segment(llm: LLMCallable, seg: str) -> dict[str, Any] | None:
|
||||
try:
|
||||
data = await llm(
|
||||
[{"role": "system", "content": _SEG_SYSTEM},
|
||||
{"role": "user", "content": seg}],
|
||||
temperature=0.0,
|
||||
)
|
||||
if not isinstance(data, dict) or not isinstance(data.get("sub_items"), list):
|
||||
return None
|
||||
sev = str(data.get("severity", "")).lower()
|
||||
title = str(data.get("title") or "").strip() or seg[:60]
|
||||
subs: list[dict[str, Any]] = []
|
||||
for s in data["sub_items"][:8]:
|
||||
if not isinstance(s, dict):
|
||||
continue
|
||||
body = str(s.get("body") or "").strip()
|
||||
if not body:
|
||||
continue
|
||||
try:
|
||||
conf = float(s.get("confidence", 1.0))
|
||||
except (TypeError, ValueError):
|
||||
conf = 0.0
|
||||
subs.append({
|
||||
"title": str(s.get("title") or "").strip() or title,
|
||||
"body": body,
|
||||
"confidence": max(0.0, min(1.0, conf)),
|
||||
})
|
||||
if not subs:
|
||||
return None
|
||||
return {"title": title,
|
||||
"severity": sev if sev in ("major", "minor", "question") else "major",
|
||||
"sub_items": subs}
|
||||
except Exception: # noqa: BLE001 — LLM 失败即回退正则，不猜测
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_cross_refs(body: str, ordinal_map: dict[int, str]) -> list[dict[str, Any]]:
|
||||
refs: list[dict[str, Any]] = []
|
||||
for m in _CROSS_REF.finditer(body):
|
||||
raw = m.group(0).strip()
|
||||
num = next((int(g) for g in m.groups() if g and g.isdigit()), None)
|
||||
if num is None and "同上" in raw:
|
||||
num = _prev_ordinal(ordinal_map, m.start()) # 同上 → 上一条，保守仅前一条
|
||||
target = ordinal_map.get(num) if num is not None else None
|
||||
refs.append({"raw": raw, "target_id": target})
|
||||
return refs
|
||||
|
||||
|
||||
def _prev_ordinal(ordinal_map: dict[int, str], _pos: int) -> int | None:
|
||||
# 简化：同上 → 已知最大编号（即前一条）
|
||||
return max(ordinal_map) if ordinal_map else None
|
||||
|
||||
|
||||
async def parse_review_text(text: str, llm: LLMCallable) -> list[dict[str, Any]]:
|
||||
"""→ reviewers=[{id, label, ordinal, comments:[Comment]}]（不含 job 包装）。"""
|
||||
reviewers_out: list[dict[str, Any]] = []
|
||||
cid_seq = 0
|
||||
for r_ord, (label, block) in enumerate(split_reviewers(text), start=1):
|
||||
segs = split_comments(block)
|
||||
if not segs:
|
||||
continue
|
||||
comments: list[dict[str, Any]] = []
|
||||
ordinal_map: dict[int, str] = {}
|
||||
parsed_seg: list[dict[str, Any]] = []
|
||||
for seg in segs:
|
||||
llm_res = await _ask_llm_segment(llm, seg)
|
||||
# 交叉核对：LLM 无法给出结构 → 该段回退正则
|
||||
parsed_seg.append(llm_res if llm_res else _fallback_segment(seg))
|
||||
for seg_idx, (seg, p) in enumerate(zip(segs, parsed_seg), start=1):
|
||||
subs = p["sub_items"]
|
||||
split = len(subs) >= 2 and max(s["confidence"] for s in subs) >= _SPLIT_CONFIDENCE
|
||||
group_id = f"g{r_ord}-{seg_idx}"
|
||||
items = subs if split else [subs[0]]
|
||||
if not split and len(subs) > 1:
|
||||
# 未达阈值 → 多子问题并回一条（body 用整段原文）
|
||||
items = [{"title": p["title"], "body": seg, "confidence": 1.0}]
|
||||
for it in items:
|
||||
cid_seq += 1
|
||||
cid = f"c{cid_seq:04d}"
|
||||
# 保守：不拆时 body 保留该段原文（不采信 LLM 改写）；拆时用子项 body
|
||||
body = it["body"] if split else seg
|
||||
ordinal_map[seg_idx] = cid
|
||||
comments.append({
|
||||
"id": cid,
|
||||
"group_id": group_id if split else None,
|
||||
"severity": p["severity"],
|
||||
"title": it["title"],
|
||||
"body": body,
|
||||
"sub_items": [s["title"] for s in subs] if split else [],
|
||||
"cross_refs": _resolve_cross_refs(body, ordinal_map),
|
||||
"response": {"parts": []},
|
||||
"draft": None,
|
||||
"validation": None,
|
||||
"human_confirmed": False,
|
||||
"scanner_downgraded": False,
|
||||
})
|
||||
# cross_refs 需全量 comment id 后再扫一遍（前面扫时 map 未齐）
|
||||
for c in comments:
|
||||
c["cross_refs"] = _resolve_cross_refs(c["body"], ordinal_map)
|
||||
if comments:
|
||||
reviewers_out.append({
|
||||
"id": f"r{r_ord}", "label": label, "ordinal": r_ord,
|
||||
"comments": comments,
|
||||
})
|
||||
return reviewers_out
|
||||
@@ -1,162 +0,0 @@
|
||||
"""services/rebuttal_store.py —— Rebuttal Phase 1 任务持久化（SQLite，自含连接）。
|
||||
|
||||
设计（2026-10-12 定稿）：
|
||||
- 表 rebuttal_jobs 落在 review_service/review.db（与 journals 同库不同表）；
|
||||
本模块自开短连接（低频 CRUD，WAL 下足够），不改既有 app/db.py。
|
||||
- Job/Comment 全量以 JSON 存 reviewers_json 列（Phase 1 规模 ≤ 数十条，
|
||||
整读整写最简单且可测；并发写靠 updated_at + 单进程串行）。
|
||||
- human_ack 不信前端置位：confirm 端点改完 comment 后由本模块重算——
|
||||
所有 comment.human_confirmed 全 true 才置 1（决策 4）。
|
||||
- 不做 TTL 自动清理（决策 5）；DELETE 端点手动删。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from ..config import get_settings
|
||||
|
||||
_SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS rebuttal_jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
status TEXT NOT NULL,
|
||||
source_kind TEXT,
|
||||
filename TEXT,
|
||||
raw_text TEXT,
|
||||
template TEXT DEFAULT 'generic',
|
||||
reviewers_json TEXT NOT NULL,
|
||||
letter_md TEXT,
|
||||
validation_json TEXT,
|
||||
human_ack INTEGER DEFAULT 0,
|
||||
created_at TEXT,
|
||||
updated_at TEXT
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def new_id(prefix: str = "") -> str:
|
||||
return prefix + uuid.uuid4().hex[:12]
|
||||
|
||||
|
||||
def _connect() -> sqlite3.Connection:
|
||||
db_path: Path = get_settings().db_path_abs
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(str(db_path), timeout=5)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
return conn
|
||||
|
||||
|
||||
def ensure_table() -> None:
|
||||
with _connect() as conn:
|
||||
conn.executescript(_SCHEMA)
|
||||
|
||||
|
||||
def _row_to_job(row: sqlite3.Row) -> dict[str, Any]:
|
||||
return {
|
||||
"id": row["id"],
|
||||
"status": row["status"],
|
||||
"source": {
|
||||
"kind": row["source_kind"] or "text",
|
||||
"filename": row["filename"],
|
||||
"raw_text": row["raw_text"] or "",
|
||||
},
|
||||
"template": row["template"] or "generic",
|
||||
"reviewers": json.loads(row["reviewers_json"] or "[]"),
|
||||
"letter_md": row["letter_md"],
|
||||
"validation": json.loads(row["validation_json"]) if row["validation_json"] else None,
|
||||
"human_ack": bool(row["human_ack"]),
|
||||
"created_at": row["created_at"],
|
||||
"updated_at": row["updated_at"],
|
||||
}
|
||||
|
||||
|
||||
def insert_job(job: dict[str, Any]) -> None:
|
||||
"""写入完整 job（reviewers 等结构由调用方构造好）。"""
|
||||
now = job.get("created_at") or _now()
|
||||
with _connect() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO rebuttal_jobs (id, status, source_kind, filename, raw_text,"
|
||||
" template, reviewers_json, letter_md, validation_json, human_ack,"
|
||||
" created_at, updated_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(
|
||||
job["id"], job.get("status") or "ready",
|
||||
(job.get("source") or {}).get("kind"),
|
||||
(job.get("source") or {}).get("filename"),
|
||||
(job.get("source") or {}).get("raw_text"),
|
||||
job.get("template") or "generic",
|
||||
json.dumps(job.get("reviewers") or [], ensure_ascii=False),
|
||||
job.get("letter_md"),
|
||||
json.dumps(job.get("validation"), ensure_ascii=False)
|
||||
if job.get("validation") is not None else None,
|
||||
1 if job.get("human_ack") else 0,
|
||||
now, now,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def get_job(job_id: str) -> dict[str, Any] | None:
|
||||
with _connect() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM rebuttal_jobs WHERE id=?", (job_id,)
|
||||
).fetchone()
|
||||
return _row_to_job(row) if row else None
|
||||
|
||||
|
||||
def update_job(job_id: str, *, reviewers: list[dict[str, Any]] | None = None,
|
||||
letter_md: str | None = None, status: str | None = None) -> dict[str, Any] | None:
|
||||
"""局部更新；reviewers 变更时重算 human_ack（全勾才 true）。"""
|
||||
job = get_job(job_id)
|
||||
if job is None:
|
||||
return None
|
||||
if reviewers is not None:
|
||||
job["reviewers"] = reviewers
|
||||
if letter_md is not None:
|
||||
job["letter_md"] = letter_md
|
||||
if status is not None:
|
||||
job["status"] = status
|
||||
job["human_ack"] = all_comments_confirmed(job["reviewers"])
|
||||
job["updated_at"] = _now()
|
||||
with _connect() as conn:
|
||||
conn.execute(
|
||||
"UPDATE rebuttal_jobs SET reviewers_json=?, letter_md=?, status=?,"
|
||||
" human_ack=?, updated_at=? WHERE id=?",
|
||||
(
|
||||
json.dumps(job["reviewers"], ensure_ascii=False),
|
||||
job.get("letter_md"),
|
||||
job["status"],
|
||||
1 if job["human_ack"] else 0,
|
||||
job["updated_at"],
|
||||
job_id,
|
||||
),
|
||||
)
|
||||
return job
|
||||
|
||||
|
||||
def delete_job(job_id: str) -> bool:
|
||||
with _connect() as conn:
|
||||
cur = conn.execute("DELETE FROM rebuttal_jobs WHERE id=?", (job_id,))
|
||||
return cur.rowcount > 0
|
||||
|
||||
|
||||
def all_comments(reviewers: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
out: list[dict[str, Any]] = []
|
||||
for r in reviewers or []:
|
||||
out.extend(r.get("comments") or [])
|
||||
return out
|
||||
|
||||
|
||||
def all_comments_confirmed(reviewers: list[dict[str, Any]]) -> bool:
|
||||
comments = all_comments(reviewers)
|
||||
# 空 job（未解析出任何 comment）不置位 human_ack，避免"全勾"平凡真
|
||||
if not comments:
|
||||
return False
|
||||
return all(bool(c.get("human_confirmed")) for c in comments)
|
||||
@@ -1,232 +0,0 @@
|
||||
"""services/recommend_engine.py —— presubmit recommendation 规则引擎（纯函数，零 LLM）。
|
||||
|
||||
背景（2026-10-12 诊断）：旧链路由 LLM 直接输出 recommendation，且 SYSTEM_PROMPT 配额
|
||||
强制 "major_weaknesses 3-8 条"，导致 accept/minor_revision 几乎不可达（对照实验中
|
||||
强稿——BUSCO 96.8%/Hi-C 98.7%/accession 齐/UFBoot 1000——仍被凑出 6 条 major）。
|
||||
新链路：LLM 只如实列 weakness（major 带 hard_rule/fatal 标签），本引擎按可测规则
|
||||
计算 recommendation + rationale（模板拼装，可复现，不经 LLM）。
|
||||
|
||||
判定链（自上而下首条命中）：
|
||||
R0 任一 major 带 fatal=true → reject（须人工复核）
|
||||
R1 H = 硬性 major 数 ≥ 1 → major_revision（与 tier 无关，不可协商）
|
||||
R2' tier==3 且 1≤S≤2 且 H==0 → minor_revision（入门刊仅 soft 可降）
|
||||
R2 S = soft major 数 ≥ 1 → major_revision（S≥5 标 severe，仍非 reject）
|
||||
R4 S == 0 且 minor ≤ accept_max_minor[tier] → accept
|
||||
R5 S == 0 → minor_revision
|
||||
其中 S = 总 major 数 − 硬性标签数（标签漏打时 S 变大，只会更严不会漏放）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
from ..config import get_settings
|
||||
|
||||
# 硬性规则标签（SYSTEM_PROMPT 与 coerce 共用白名单；未知标签一律置 None）
|
||||
HARD_RULE_TAGS = (
|
||||
"busco_missing", # 组装类论文缺 BUSCO 完整格式
|
||||
"hic_missing", # 宣称 chromosome-level/Hi-C 但缺挂载率
|
||||
"accession_missing", # 缺数据 Accession（"will be deposited" 视为缺）
|
||||
"voucher_missing", # 野生材料缺凭证标本
|
||||
"abs_missing", # 缺 ABS/名古屋合规声明
|
||||
)
|
||||
|
||||
HARD_RULE_LABELS = {
|
||||
"busco_missing": "BUSCO 完整度缺失",
|
||||
"hic_missing": "Hi-C 挂载率缺失",
|
||||
"accession_missing": "数据 Accession 缺失",
|
||||
"voucher_missing": "凭证标本缺失",
|
||||
"abs_missing": "ABS/名古屋合规声明缺失",
|
||||
}
|
||||
|
||||
TIER_LABELS = {1: "顶刊", 2: "主流期刊", 3: "入门期刊"}
|
||||
|
||||
# tier 参数：accept 门槛（minor 上限）；tier3 额外允许 ≤2 条 soft major 降为小修
|
||||
TIER_ACCEPT_MAX_MINOR = {1: 1, 2: 2, 3: 3}
|
||||
TIER3_SOFT_MAJOR_AS_MINOR_MAX = 2
|
||||
|
||||
# ---- tier 查找优先级（2026-10-12）：override → featured → jif → 默认 ----
|
||||
# ① 用户高频投稿顶刊硬编码（精确名匹配，不受 jif/综述刊启发式影响；
|
||||
# "The " 前缀归一化，"Plant Cell"/"The Plant Cell" 互认）
|
||||
TIER1_OVERRIDES = frozenset(
|
||||
re.sub(r"^the\s+", "", name.casefold()) for name in (
|
||||
"Nature", "Nature Genetics", "Nature Plants", "Nature Biotechnology",
|
||||
"Nature Methods", "Cell", "Science", "Genome Biology", "Molecular Plant",
|
||||
"The Plant Cell", "Nature Communications", "Science Advances", "Cell Research",
|
||||
)
|
||||
)
|
||||
# 综述刊不因高 jif 升 tier1（Annual Review of… / Trends in… / …Reviews）
|
||||
_REVIEW_JOURNAL_RE = re.compile(r"reviews?\b|\btrends\b|annual\s+review", re.IGNORECASE)
|
||||
_JIF_TIER1_MIN = 15.0 # jif >= 15 → tier1；5 ≤ jif < 15 → tier2；<5/NULL → 默认 tier2
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _tier_by_name() -> dict[str, int]:
|
||||
"""journals.json 精选期刊 name(casefold) → review_tier（读文件一次，缓存）。"""
|
||||
path = get_settings().journals_seed_abs
|
||||
try:
|
||||
payload = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
out: dict[str, int] = {}
|
||||
for j in payload.get("journals") or []:
|
||||
if not isinstance(j, dict):
|
||||
continue
|
||||
name = str(j.get("name") or "").strip().casefold()
|
||||
tier = j.get("review_tier")
|
||||
if name and isinstance(tier, int) and tier in (1, 2, 3):
|
||||
out[name] = tier
|
||||
return out
|
||||
|
||||
|
||||
@lru_cache(maxsize=256)
|
||||
def _jif_by_name(name_bare_cf: str) -> float | None:
|
||||
"""journal_directory 按名查 jif（只读一列；表缺失/查不到 → None）。"""
|
||||
db = get_settings().db_path_abs
|
||||
if not Path(db).exists():
|
||||
return None
|
||||
try:
|
||||
conn = sqlite3.connect(f"file:{db}?mode=ro", uri=True, timeout=2)
|
||||
try:
|
||||
row = conn.execute(
|
||||
"SELECT jif FROM journal_directory WHERE lang='en' "
|
||||
"AND (lower(name)=? OR lower(name)=?) "
|
||||
"ORDER BY jif DESC LIMIT 1",
|
||||
(name_bare_cf, "the " + name_bare_cf),
|
||||
).fetchone()
|
||||
finally:
|
||||
conn.close()
|
||||
except sqlite3.Error:
|
||||
return None
|
||||
if not row or row[0] is None:
|
||||
return None
|
||||
try:
|
||||
return float(row[0])
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def resolve_tier(target_journal: str) -> tuple[int, str]:
|
||||
"""目标期刊 → (tier, source)。source ∈ override/featured/jif/默认。
|
||||
|
||||
查找优先级：① 硬编码 13 本顶刊 ② journals.json review_tier
|
||||
③ journal_directory jif 启发式（综述刊除外）④ 默认 tier 2。
|
||||
"""
|
||||
raw = (target_journal or "").strip()
|
||||
if not raw:
|
||||
return 2, "默认"
|
||||
key = raw.casefold()
|
||||
bare = re.sub(r"^the\s+", "", key)
|
||||
if bare in TIER1_OVERRIDES:
|
||||
return 1, "override"
|
||||
featured = _tier_by_name()
|
||||
tier = featured.get(key, featured.get(bare))
|
||||
if isinstance(tier, int) and tier in (1, 2, 3):
|
||||
return tier, "featured"
|
||||
if not _REVIEW_JOURNAL_RE.search(bare):
|
||||
jif = _jif_by_name(bare)
|
||||
if jif is not None:
|
||||
return (1 if jif >= _JIF_TIER1_MIN else 2), "jif"
|
||||
return 2, "默认"
|
||||
|
||||
|
||||
def _item_field(item: Any, key: str) -> Any:
|
||||
if isinstance(item, dict):
|
||||
return item.get(key)
|
||||
return getattr(item, key, None)
|
||||
|
||||
|
||||
def decide_recommendation(
|
||||
majors: Iterable[Any],
|
||||
minors: Iterable[Any] = (),
|
||||
presentations: Iterable[Any] = (),
|
||||
tier: int = 2,
|
||||
) -> dict[str, Any]:
|
||||
"""规则引擎入口。majors 项可为 dict 或带 hard_rule/fatal 属性的对象。
|
||||
|
||||
返回 {"recommendation": ..., "rationale": {...}}；
|
||||
rationale 含 rule/hard_hits/计数/tier/tier_label/explanation（模板拼装）。
|
||||
"""
|
||||
del presentations # 呈现类不参与判定（仅归档展示）
|
||||
majors = list(majors)
|
||||
minors = list(minors)
|
||||
if tier not in (1, 2, 3):
|
||||
tier = 2
|
||||
|
||||
hard_hits: list[str] = []
|
||||
fatal = False
|
||||
for m in majors:
|
||||
if bool(_item_field(m, "fatal")):
|
||||
fatal = True
|
||||
hr = _item_field(m, "hard_rule")
|
||||
if hr in HARD_RULE_LABELS:
|
||||
hard_hits.append(hr)
|
||||
|
||||
m_count = len(majors)
|
||||
h_count = len(hard_hits)
|
||||
s_count = m_count - h_count # soft major（标签漏打 → 计入 S，只会更严）
|
||||
minor_count = len(minors)
|
||||
tier_label = TIER_LABELS[tier]
|
||||
|
||||
# tier3 降级通道提示（对作者可见：同稿投不同档结果不同）
|
||||
tier_note = ""
|
||||
if h_count == 0 and 1 <= s_count <= TIER3_SOFT_MAJOR_AS_MINOR_MAX and tier != 3:
|
||||
tier_note = f"当前判定基于 tier {tier} 档；tier 3 入门刊此稿可能判 minor_revision。"
|
||||
|
||||
def _result(rec: str, rule: str, explanation: str) -> dict[str, Any]:
|
||||
return {
|
||||
"recommendation": rec,
|
||||
"rationale": {
|
||||
"rule": rule,
|
||||
"hard_hits": hard_hits,
|
||||
"major_count": m_count,
|
||||
"hard_count": h_count,
|
||||
"soft_count": s_count,
|
||||
"minor_count": minor_count,
|
||||
"tier": tier,
|
||||
"tier_label": tier_label,
|
||||
"explanation": explanation + tier_note,
|
||||
},
|
||||
}
|
||||
|
||||
# R0：fatal → reject（前端须配"建议人工复核"）
|
||||
if fatal:
|
||||
return _result(
|
||||
"reject", "R0",
|
||||
"存在 fatal 级问题（数据可靠性/伦理，须人工复核）→ 建议拒稿。")
|
||||
|
||||
# R1：硬性 major 不可协商
|
||||
if h_count >= 1:
|
||||
labels = "、".join(HARD_RULE_LABELS.get(t, t) for t in dict.fromkeys(hard_hits))
|
||||
return _result(
|
||||
"major_revision", "R1",
|
||||
f"命中硬性缺陷（{labels}）→ 大修（不可协商）；"
|
||||
f"另有大修级 {s_count} 条、小修级 {minor_count} 条。")
|
||||
|
||||
# R2'：入门刊 1-2 条 soft major 可降为小修
|
||||
if tier == 3 and 1 <= s_count <= TIER3_SOFT_MAJOR_AS_MINOR_MAX:
|
||||
return _result(
|
||||
"minor_revision", "R2'",
|
||||
f"无硬性缺陷，大修级（soft）{s_count} 条；"
|
||||
f"入门刊（tier 3）可将 ≤{TIER3_SOFT_MAJOR_AS_MINOR_MAX} 条 soft 降为小修 → 建议小修。")
|
||||
|
||||
# R2/R3：soft major ≥ 1（S≥5 标 severe）
|
||||
if s_count >= 1:
|
||||
sev = "，severe" if s_count >= 5 else ""
|
||||
return _result(
|
||||
"major_revision", "R2",
|
||||
f"大修级问题 {s_count} 条（均非硬性{sev}）→ 建议大修。")
|
||||
|
||||
# R4/R5：无 major
|
||||
accept_max = TIER_ACCEPT_MAX_MINOR[tier]
|
||||
if minor_count <= accept_max:
|
||||
return _result(
|
||||
"accept", "R4",
|
||||
f"无大修级问题，小修级 {minor_count} 条（≤{accept_max}）→ 建议接收。")
|
||||
return _result(
|
||||
"minor_revision", "R5",
|
||||
f"无大修级问题，小修级 {minor_count} 条 → 建议小修。")
|
||||
@@ -1,553 +0,0 @@
|
||||
r"""services/reformat.py —— P1 论文重排版（.tex 源码级规则变换，不编译）。
|
||||
|
||||
链路：resolve_rules（三级规则解析）→ reformat_tex（4 变换器 + 结构守卫）
|
||||
→ 路由层落盘 data/reformat_jobs/{job_id}.tex/.json。
|
||||
纯规则、零 LLM、零执行用户内容、不碰 docx/pdf（那两个只在 FormatPanel 审核）。
|
||||
|
||||
拍板（用户 4 点确认，2026-10-10）：
|
||||
1. changes[] 每条带 rule_key ∈ {font_pt, margin, line_spacing, bib_style}、
|
||||
action ∈ {modified, inserted, warning}（P1 只产生 modified/warning，
|
||||
inserted 留给后续阶段）；前端按 rule_key 着色、按 action 计数。
|
||||
2. 保守插入策略：命令/选项缺失一律【不改源码】→ changes 里 action=warning，
|
||||
reason 附"建议手动添加"的代码片段；warning 不计入"已修改"计数。
|
||||
（warnings[] 顶层数组只放结构级提示，如多处 \documentclass。）
|
||||
3. target_journal 必填；"default"= 通用规则（min_font_pt=11, min_margin_mm=20,
|
||||
min_line_spacing=1.5, refs_style=apa），前端下拉首项即通用。
|
||||
4. P1 仅 .tex；结构畸形宁拒不猜（PARSER_FAILED）。
|
||||
|
||||
铁律：
|
||||
① 只在违反规则时改（合规零触碰 → 重跑幂等，changes=[]）
|
||||
② 只收紧不放松：字号/行距/边距只调大、绝不调小
|
||||
③ 每条修改必须能回答"为什么"：reason 带原值/目标值/规则
|
||||
|
||||
规则解析链（resolve_rules）：
|
||||
"default" → 内置通用规则
|
||||
journals.json id/name 命中 → data/format_cache/{id}.json ok → fetched
|
||||
→ 否则 format_rules 非 placeholder → official
|
||||
→ 否则通用规则 + 占位说明
|
||||
形如 "en:xxx"/"zh:xxx"（journal_directory）→ 只读查目录拿 name → 回链上一步
|
||||
其他字符串（期刊名） → 回链 journals.json → 未中 → 通用 + 未匹配说明
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
# review_service/app/services/ → parents[2] = review_service/
|
||||
_ROOT = Path(__file__).resolve().parents[2]
|
||||
_SEED_PATH = _ROOT / "data" / "journals" / "journals.json"
|
||||
_CACHE_DIR = _ROOT / "data" / "format_cache"
|
||||
|
||||
_EPS = 1e-9
|
||||
|
||||
# 通用规则（拍板 3：与 journals.json format_rules_default 的 9/15/1.15 不同，
|
||||
# 这是重排版自己的"通用规则"口径，勿混用）
|
||||
RULES_DEFAULT: dict[str, Any] = {
|
||||
"rules_source": "default",
|
||||
"max_pages": 10,
|
||||
"min_font_pt": 11,
|
||||
"min_margin_mm": 20.0,
|
||||
"min_line_spacing": 1.5,
|
||||
"min_figures": 1,
|
||||
"max_fonts": 6,
|
||||
"refs_style": "apa",
|
||||
"anonymous": None,
|
||||
}
|
||||
|
||||
# 参考文献样式 → 经典 .bst（均为发行版基础件）。
|
||||
# 修正（2026-10-10）：author_year 旧映射 plain 是错的——plain.bst 是编号制 [1]；
|
||||
# author-year 制应映射 apalike.bst（Author, 2020）。apa 同理。
|
||||
REFS_BST_MAP: dict[str, str] = {
|
||||
"apa": "apalike",
|
||||
"author_year": "apalike",
|
||||
"numbered": "unsrt",
|
||||
"vancouver": "unsrt",
|
||||
"gb7714": "gb7714",
|
||||
}
|
||||
# biblatex 工作流的 style= 建议名（与 .bst 是两套体系）
|
||||
BIBLATEX_STYLE_MAP: dict[str, str] = {
|
||||
"apa": "apa",
|
||||
"author_year": "authoryear",
|
||||
"numbered": "numeric",
|
||||
"vancouver": "numeric",
|
||||
"gb7714": "gb7714",
|
||||
}
|
||||
|
||||
_DOCCLASS_RE = re.compile(r"(\\documentclass)\s*(\[[^\]]*\])?\s*\{([^}]*)\}")
|
||||
_PKG_RE = re.compile(r"\\(?:usepackage|RequirePackage)\s*(\[[^\]]*\])?\s*\{([^}]*)\}")
|
||||
_SPACING_RE = re.compile(r"\\(?:linespread|setstretch)\s*\{\s*(\d+(?:\.\d+)?)\s*\}")
|
||||
_BIBSTYLE_RE = re.compile(r"\\bibliographystyle\s*\{\s*([^}]+?)\s*\}")
|
||||
_PT_RE = re.compile(r"(\d+(?:\.\d+)?)\s*pt\b")
|
||||
_BEGIN_DOC_RE = re.compile(r"\\begin\s*\{\s*document\s*\}")
|
||||
_BIBLATEX_RE = re.compile(r"\\(?:printbibliography|addbibresource)")
|
||||
|
||||
_MM_FACTORS = {"mm": 1.0, "cm": 10.0, "in": 25.4, "inch": 25.4, "pt": 25.4 / 72.0}
|
||||
_SIDE_KEYS = ("left", "right", "top", "bottom")
|
||||
_MARGIN_ATTRS = ("margin", "hmargin", "vmargin", "left", "right", "top", "bottom",
|
||||
"inner", "outer")
|
||||
|
||||
|
||||
class ReformatError(Exception):
|
||||
"""结构畸形无法安全处理（code: PARSER_FAILED）。"""
|
||||
|
||||
def __init__(self, code: str, message: str) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
|
||||
|
||||
# ================================================================ 规则解析链
|
||||
|
||||
def _seed_entries() -> list[dict[str, Any]]:
|
||||
try:
|
||||
data = json.loads(_SEED_PATH.read_text("utf-8"))
|
||||
entries = data.get("journals")
|
||||
return [e for e in entries if isinstance(e, dict)] if isinstance(entries, list) else []
|
||||
except (OSError, ValueError):
|
||||
return []
|
||||
|
||||
|
||||
def _find_seed(text: str) -> dict[str, Any] | None:
|
||||
low = text.strip().casefold()
|
||||
if not low:
|
||||
return None
|
||||
for e in _seed_entries():
|
||||
if str(e.get("id", "")).casefold() == low or str(e.get("name", "")).casefold() == low:
|
||||
return e
|
||||
return None
|
||||
|
||||
|
||||
def _cache_rules(journal_id: str) -> dict[str, Any] | None:
|
||||
"""data/format_cache/{id}.json（抓取+LLM 抽取产物）；ok=false/缺失 → None。"""
|
||||
try:
|
||||
payload = json.loads((_CACHE_DIR / f"{journal_id}.json").read_text("utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
if not isinstance(payload, dict) or not payload.get("ok"):
|
||||
return None
|
||||
rules = payload.get("format_rules")
|
||||
return rules if isinstance(rules, dict) and rules.get("rules_source") else None
|
||||
|
||||
|
||||
def _merged(rules: dict[str, Any]) -> dict[str, Any]:
|
||||
"""专属规则叠加在通用规则上（缺失键/显式 null 保留——fetched 的 null 是诚实未知）。"""
|
||||
return {**RULES_DEFAULT, **rules}
|
||||
|
||||
|
||||
async def _directory_name(journal_id: str) -> str | None:
|
||||
"""journal_directory id → name（只读；表未连/不存在 → None，绝不抛错）。"""
|
||||
try:
|
||||
from ..db import db # 惰性导入：resolver 可在无 lifespan 的测试中工作
|
||||
row = await db.fetchone(
|
||||
"SELECT name FROM journal_directory WHERE id = ?", (journal_id,)
|
||||
)
|
||||
return str(row["name"]) if row and row.get("name") else None
|
||||
except Exception: # noqa: BLE001 —— 查不到/未连接都按未匹配处理
|
||||
return None
|
||||
|
||||
|
||||
async def resolve_rules(target: str) -> dict[str, Any]:
|
||||
"""→ {journal_id, journal_name, rules, rules_source, rules_note}（永不失败）。"""
|
||||
target = (target or "").strip()
|
||||
|
||||
if target == "default":
|
||||
return {
|
||||
"journal_id": "default",
|
||||
"journal_name": "通用规则",
|
||||
"rules": dict(RULES_DEFAULT),
|
||||
"rules_source": "default",
|
||||
"rules_note": "通用规则（APA + 通用论文规范）",
|
||||
}
|
||||
|
||||
display_name = target
|
||||
entry = _find_seed(target)
|
||||
if entry is None and ":" in target:
|
||||
# journal_directory id（en:0028-0836 / zh:slug）→ 目录查 name → 再回链种子
|
||||
name = await _directory_name(target)
|
||||
if name:
|
||||
display_name = name
|
||||
entry = _find_seed(name)
|
||||
elif entry is None:
|
||||
display_name = target # 直接当期刊名回链
|
||||
|
||||
if entry is not None:
|
||||
jid = str(entry.get("id") or "")
|
||||
jname = str(entry.get("name") or display_name)
|
||||
cached = _cache_rules(jid)
|
||||
if cached:
|
||||
return {
|
||||
"journal_id": jid, "journal_name": jname, "rules": _merged(cached),
|
||||
"rules_source": "fetched",
|
||||
"rules_note": "该刊规则来自官网抓取 + LLM 抽取（data/format_cache，待人工核验）",
|
||||
}
|
||||
fr = entry.get("format_rules") if isinstance(entry.get("format_rules"), dict) else {}
|
||||
if fr and fr.get("rules_source") not in (None, "placeholder", "default"):
|
||||
return {
|
||||
"journal_id": jid, "journal_name": jname, "rules": _merged(fr),
|
||||
"rules_source": str(fr.get("rules_source")),
|
||||
"rules_note": "journals.json 期刊专属规则",
|
||||
}
|
||||
return {
|
||||
"journal_id": jid, "journal_name": jname, "rules": dict(RULES_DEFAULT),
|
||||
"rules_source": "default",
|
||||
"rules_note": f"「{jname}」在 journals.json 中仅有占位规则（placeholder），已用通用规则",
|
||||
}
|
||||
|
||||
return {
|
||||
"journal_id": None,
|
||||
"journal_name": display_name or target,
|
||||
"rules": dict(RULES_DEFAULT),
|
||||
"rules_source": "default",
|
||||
"rules_note": f"未匹配到「{display_name or target}」的专属格式规则，已用通用规则",
|
||||
}
|
||||
|
||||
|
||||
# ================================================================ 变换器工具
|
||||
|
||||
def _change(line_no: int, original: str, modified: str, reason: str,
|
||||
rule_key: str, action: str) -> dict[str, Any]:
|
||||
return {
|
||||
"line": line_no,
|
||||
"original": original.rstrip("\n"),
|
||||
"modified": modified.rstrip("\n"),
|
||||
"reason": reason,
|
||||
"rule_key": rule_key,
|
||||
"action": action,
|
||||
}
|
||||
|
||||
|
||||
def _warning_change(line_no: int, original: str, reason: str, rule_key: str) -> dict[str, Any]:
|
||||
return _change(line_no, original, "", reason, rule_key, "warning")
|
||||
|
||||
|
||||
def _parse_mm(raw: str) -> float | None:
|
||||
"""geometry 长度值 → mm；裸数字按 TeX 惯例 pt（同 format_check 先例）；未知 → None。"""
|
||||
m = re.fullmatch(r"\s*(\d+(?:\.\d+)?)\s*([A-Za-z]*)\s*", raw or "")
|
||||
if not m:
|
||||
return None
|
||||
unit = (m.group(2) or "pt").lower()
|
||||
factor = _MM_FACTORS.get(unit)
|
||||
if factor is None:
|
||||
return None
|
||||
return float(m.group(1)) * factor
|
||||
|
||||
|
||||
def _fmt(v: float) -> str:
|
||||
return f"{v:g}"
|
||||
|
||||
|
||||
# ================================================================ ① 字号 font_pt
|
||||
|
||||
def _transform_font(lines: list[str], orig: list[str], rules: dict[str, Any],
|
||||
doc_idx: int) -> list[dict[str, Any]]:
|
||||
min_pt = rules.get("min_font_pt")
|
||||
if min_pt is None:
|
||||
return []
|
||||
min_pt = float(min_pt)
|
||||
line = lines[doc_idx]
|
||||
m = _DOCCLASS_RE.search(line)
|
||||
if not m: # 守卫已保证，理论不可达
|
||||
return []
|
||||
opts = m.group(2) # "[...]" 或 None
|
||||
cls = m.group(3).strip()
|
||||
opts_inner = opts[1:-1] if opts else ""
|
||||
pt_m = _PT_RE.search(opts_inner)
|
||||
|
||||
if pt_m is None:
|
||||
# 拍板 2：选项缺失不自动加 → warning + 建议片段
|
||||
if opts_inner:
|
||||
suggestion = f"\\documentclass[{opts_inner}, {_fmt(min_pt)}pt]{{{cls}}}"
|
||||
else:
|
||||
suggestion = f"\\documentclass[{_fmt(min_pt)}pt]{{{cls}}}"
|
||||
return [_warning_change(
|
||||
doc_idx + 1, orig[doc_idx],
|
||||
f"未检测到字号选项，无法核验正文字号（目标下限 {_fmt(min_pt)}pt），"
|
||||
f"建议手动声明：{suggestion}",
|
||||
"font_pt",
|
||||
)]
|
||||
|
||||
cur = float(pt_m.group(1))
|
||||
if cur >= min_pt - _EPS:
|
||||
return []
|
||||
target = next((s for s in (10, 11, 12) if s >= min_pt - _EPS), min_pt)
|
||||
new_inner = opts_inner[:pt_m.start(1)] + _fmt(target) + opts_inner[pt_m.end(1):]
|
||||
new_line = line[:m.start(2) + 1] + new_inner + line[m.start(2) + 1 + len(opts_inner):]
|
||||
lines[doc_idx] = new_line
|
||||
return [_change(
|
||||
doc_idx + 1, orig[doc_idx], new_line,
|
||||
f"正文字号 {_fmt(cur)}pt 低于目标下限 {_fmt(min_pt)}pt，改为 {_fmt(target)}pt",
|
||||
"font_pt", "modified",
|
||||
)]
|
||||
|
||||
|
||||
# ================================================================ ② 页边距 margin
|
||||
|
||||
def _find_geometry(lines: list[str]) -> tuple[int, re.Match] | None:
|
||||
for i, line in enumerate(lines):
|
||||
for m in _PKG_RE.finditer(line):
|
||||
pkgs = [p.strip().lower() for p in m.group(2).split(",")]
|
||||
if "geometry" in pkgs:
|
||||
return i, m
|
||||
return None
|
||||
|
||||
|
||||
def _patch_attr(line: str, key: str, new_val: str) -> str:
|
||||
"""geometry 选项里 key=old → key=new（首个匹配）。
|
||||
|
||||
值止于 `,` 或 `]`——绝不能吞掉 `]{geometry}` 尾巴（否则下一轮就找不到
|
||||
geometry 宏包，破坏幂等）。key 带 \\b 防 hmargin/margin 互撞。
|
||||
"""
|
||||
return re.sub(rf"(\b{key}\s*=\s*)([^,\]]+)", lambda m: m.group(1) + new_val,
|
||||
line, count=1)
|
||||
|
||||
|
||||
def _transform_margin(lines: list[str], orig: list[str], rules: dict[str, Any],
|
||||
doc_idx: int) -> list[dict[str, Any]]:
|
||||
min_mm = rules.get("min_margin_mm")
|
||||
if min_mm is None:
|
||||
return []
|
||||
min_mm = float(min_mm)
|
||||
out: list[dict[str, Any]] = []
|
||||
found = _find_geometry(lines)
|
||||
|
||||
if found is None:
|
||||
return [_warning_change(
|
||||
doc_idx + 1, orig[doc_idx],
|
||||
f"未检测到 geometry 宏包，无法核验页边距（目标下限 {_fmt(min_mm)}mm），"
|
||||
f"建议手动添加：\\usepackage[margin={_fmt(min_mm)}mm]{{geometry}}"
|
||||
f"（置于 \\documentclass 之后）",
|
||||
"margin",
|
||||
)]
|
||||
|
||||
gi, gm = found
|
||||
opts = gm.group(1)
|
||||
if opts is None:
|
||||
# \usepackage{geometry} 裸用 → 建议加选项（拍板 2 不自动改）
|
||||
return [_warning_change(
|
||||
gi + 1, orig[gi],
|
||||
f"geometry 未设置边距选项，无法核验页边距（目标下限 {_fmt(min_mm)}mm），"
|
||||
f"建议手动改为：\\usepackage[margin={_fmt(min_mm)}mm]{{geometry}}",
|
||||
"margin",
|
||||
)]
|
||||
opts_inner = opts[1:-1]
|
||||
attrs: dict[str, str] = {}
|
||||
for part in opts_inner.split(","):
|
||||
if "=" in part:
|
||||
k, _, v = part.partition("=")
|
||||
attrs[k.strip().lower()] = v.strip()
|
||||
|
||||
present = [k for k in _MARGIN_ATTRS if k in attrs]
|
||||
if not present:
|
||||
return [_warning_change(
|
||||
gi + 1, orig[gi],
|
||||
f"geometry 未设置边距相关选项（目标下限 {_fmt(min_mm)}mm），"
|
||||
f"建议在现有选项中添加：margin={_fmt(min_mm)}mm",
|
||||
"margin",
|
||||
)]
|
||||
|
||||
# 逐边解析提供者（side > h/vmargin > margin；inner/outer 单独按自身长度评）
|
||||
def provider_for(side: str) -> str | None:
|
||||
if side in attrs:
|
||||
return side
|
||||
if side in ("left", "right") and "hmargin" in attrs:
|
||||
return "hmargin"
|
||||
if side in ("top", "bottom") and "vmargin" in attrs:
|
||||
return "vmargin"
|
||||
if "margin" in attrs:
|
||||
return "margin"
|
||||
if side in ("left", "right") and "inner" in attrs:
|
||||
return "inner" # 见下：inner/outer 单独评估，此处不按边处理
|
||||
if side in ("left", "right") and "outer" in attrs:
|
||||
return "outer"
|
||||
return None
|
||||
|
||||
patch_keys: dict[str, str] = {} # attr → 新值（同键只抬一次）
|
||||
reasons: list[str] = []
|
||||
unresolved: list[str] = []
|
||||
|
||||
for side in _SIDE_KEYS:
|
||||
pkey = provider_for(side)
|
||||
if pkey in ("inner", "outer"):
|
||||
continue # 由下方 inner/outer 评估负责
|
||||
if pkey is None:
|
||||
unresolved.append(side)
|
||||
continue
|
||||
mm = _parse_mm(attrs.get(pkey, ""))
|
||||
if mm is None:
|
||||
unresolved.append(side)
|
||||
continue
|
||||
if mm < min_mm - _EPS and pkey not in patch_keys:
|
||||
patch_keys[pkey] = f"{_fmt(min_mm)}mm"
|
||||
reasons.append(f"页边距 {side}({_fmt(mm)}mm) 低于下限 {_fmt(min_mm)}mm → {_fmt(min_mm)}mm")
|
||||
|
||||
for key in ("inner", "outer"):
|
||||
if key in attrs:
|
||||
mm = _parse_mm(attrs[key])
|
||||
if mm is None:
|
||||
unresolved.append(key)
|
||||
elif mm < min_mm - _EPS and key not in patch_keys:
|
||||
patch_keys[key] = f"{_fmt(min_mm)}mm"
|
||||
reasons.append(f"页边距 {key}({_fmt(mm)}mm) 低于下限 {_fmt(min_mm)}mm → {_fmt(min_mm)}mm")
|
||||
|
||||
out: list[dict[str, Any]] = []
|
||||
if patch_keys:
|
||||
new_line = lines[gi]
|
||||
for key, val in patch_keys.items():
|
||||
new_line = _patch_attr(new_line, key, val)
|
||||
lines[gi] = new_line
|
||||
out.append(_change(
|
||||
gi + 1, orig[gi], new_line, "；".join(reasons), "margin", "modified",
|
||||
))
|
||||
|
||||
if unresolved:
|
||||
uniq = "、".join(dict.fromkeys(unresolved))
|
||||
out.append(_warning_change(
|
||||
gi + 1, orig[gi],
|
||||
f"geometry 未设置/无法解析 {uniq} 边（沿用类默认，无法核验），"
|
||||
f"建议显式声明：{uniq}={_fmt(min_mm)}mm",
|
||||
"margin",
|
||||
))
|
||||
return out
|
||||
|
||||
|
||||
# ================================================================ ③ 行距 line_spacing
|
||||
|
||||
def _transform_spacing(lines: list[str], orig: list[str], rules: dict[str, Any],
|
||||
doc_idx: int) -> list[dict[str, Any]]:
|
||||
min_ls = rules.get("min_line_spacing")
|
||||
if min_ls is None:
|
||||
return []
|
||||
min_ls = float(min_ls)
|
||||
|
||||
# 只看导言区（\begin{document} 之前）；正文中的 \setstretch 是局部覆盖，不碰
|
||||
end = len(lines)
|
||||
for i, line in enumerate(lines):
|
||||
if _BEGIN_DOC_RE.search(line):
|
||||
end = i
|
||||
break
|
||||
records: list[tuple[int, re.Match]] = []
|
||||
for i in range(end):
|
||||
for m in _SPACING_RE.finditer(lines[i]):
|
||||
records.append((i, m))
|
||||
|
||||
if not records:
|
||||
return [_warning_change(
|
||||
doc_idx + 1, orig[doc_idx],
|
||||
f"未在导言区检测到 \\linespread 或 \\setstretch，无法核验行距"
|
||||
f"（目标下限 {_fmt(min_ls)}），建议手动添加到导言区："
|
||||
f"\\linespread{{{_fmt(min_ls)}}}",
|
||||
"line_spacing",
|
||||
)]
|
||||
|
||||
i, m = records[-1] # 最后一次出现者生效（LaTeX 后设覆盖前设）
|
||||
cur = float(m.group(1))
|
||||
if cur >= min_ls - _EPS:
|
||||
return []
|
||||
new_line = lines[i][:m.start(1)] + _fmt(min_ls) + lines[i][m.end(1):]
|
||||
lines[i] = new_line
|
||||
return [_change(
|
||||
i + 1, orig[i], new_line,
|
||||
f"行距 {_fmt(cur)} 低于目标下限 {_fmt(min_ls)}，改为 {_fmt(min_ls)}",
|
||||
"line_spacing", "modified",
|
||||
)]
|
||||
|
||||
|
||||
# ================================================================ ④ 参考文献 bib_style
|
||||
|
||||
def _transform_bib(lines: list[str], orig: list[str], rules: dict[str, Any],
|
||||
doc_idx: int) -> list[dict[str, Any]]:
|
||||
style = str(rules.get("refs_style") or "").strip().lower()
|
||||
if not style:
|
||||
# 规则未提供已确认的 refs_style（如 Frontiers 两种制式并存待核实）：
|
||||
# 绝不猜测改写用户样式——保持原样 + warning 提示（拍板 2026-10-10）
|
||||
return [_warning_change(
|
||||
doc_idx + 1, orig[doc_idx],
|
||||
"该刊引用风格未确认，未改动 \\bibliographystyle；"
|
||||
"建议人工核实期刊投稿指南后再调整",
|
||||
"bib_style",
|
||||
)]
|
||||
|
||||
m = next((mm for line in lines if (mm := _BIBSTYLE_RE.search(line))), None)
|
||||
if m is not None:
|
||||
i = next(idx for idx, line in enumerate(lines) if _BIBSTYLE_RE.search(line))
|
||||
cur = m.group(1).strip()
|
||||
if style not in REFS_BST_MAP:
|
||||
return [_warning_change(
|
||||
i + 1, orig[i],
|
||||
f"目标参考文献样式 {style} 无对应标准 .bst，未替换"
|
||||
f"（当前 \\bibliographystyle{{{cur}}}），请按目标期刊模板手动选择样式",
|
||||
"bib_style",
|
||||
)]
|
||||
target = REFS_BST_MAP[style]
|
||||
if cur.casefold() == target.casefold():
|
||||
return []
|
||||
new_line = lines[i][:m.start(1)] + target + lines[i][m.end(1):]
|
||||
lines[i] = new_line
|
||||
hint = "（需 gbt7714 宏包）" if target == "gb7714" else ""
|
||||
return [_change(
|
||||
i + 1, orig[i], new_line,
|
||||
f"参考文献样式要求 {style}：\\bibliographystyle{{{cur}}} → {{{target}}}{hint}",
|
||||
"bib_style", "modified",
|
||||
)]
|
||||
|
||||
has_biblatex = any(_BIBLATEX_RE.search(line) for line in lines)
|
||||
if has_biblatex:
|
||||
suggest = BIBLATEX_STYLE_MAP.get(style, style)
|
||||
return [_warning_change(
|
||||
doc_idx + 1, orig[doc_idx],
|
||||
f"检测到 biblatex 工作流（未找到 \\bibliographystyle），"
|
||||
f"请在 \\printbibliography 选项中手动配置：style={suggest}（对应 {style}）",
|
||||
"bib_style",
|
||||
)]
|
||||
|
||||
if style in REFS_BST_MAP:
|
||||
target = REFS_BST_MAP[style]
|
||||
return [_warning_change(
|
||||
doc_idx + 1, orig[doc_idx],
|
||||
f"未检测到 \\bibliographystyle，建议手动添加：\\bibliographystyle{{{target}}}"
|
||||
f"（对应 {style}）",
|
||||
"bib_style",
|
||||
)]
|
||||
return [_warning_change(
|
||||
doc_idx + 1, orig[doc_idx],
|
||||
f"未检测到 \\bibliographystyle，目标样式 {style} 无标准 .bst 映射，"
|
||||
f"请按期刊模板手动配置",
|
||||
"bib_style",
|
||||
)]
|
||||
|
||||
|
||||
# ================================================================ 入口
|
||||
|
||||
def reformat_tex(source: str, rules: dict[str, Any]) -> dict[str, Any]:
|
||||
"""源码 + 规则 → {modified_tex, changes[], warnings[]}。
|
||||
|
||||
结构守卫：0 处 \\documentclass → ReformatError(PARSER_FAILED)；
|
||||
>1 处 → 只处理第一处 + warnings 补 MULTI_DOCUMENTCLASS。
|
||||
"""
|
||||
lines = source.splitlines(keepends=True)
|
||||
doc_idxs = [i for i, line in enumerate(lines) if _DOCCLASS_RE.search(line)]
|
||||
if not doc_idxs:
|
||||
raise ReformatError(
|
||||
"PARSER_FAILED",
|
||||
"未找到 \\documentclass{...}：无法确定主文档"
|
||||
"（P1 不支持由 \\input 拆分的多文件工程，请上传完整单文件源码）",
|
||||
)
|
||||
warnings: list[dict[str, str]] = []
|
||||
if len(doc_idxs) > 1:
|
||||
warnings.append({
|
||||
"code": "MULTI_DOCUMENTCLASS",
|
||||
"message": f"检测到 {len(doc_idxs)} 处 \\documentclass，"
|
||||
f"仅处理第一处（第 {doc_idxs[0] + 1} 行）",
|
||||
})
|
||||
doc_idx = doc_idxs[0]
|
||||
orig = list(lines) # 纯净快照：所有 change.original 取自这里
|
||||
|
||||
changes: list[dict[str, Any]] = []
|
||||
changes += _transform_font(lines, orig, rules, doc_idx)
|
||||
changes += _transform_margin(lines, orig, rules, doc_idx)
|
||||
changes += _transform_spacing(lines, orig, rules, doc_idx)
|
||||
changes += _transform_bib(lines, orig, rules, doc_idx)
|
||||
changes.sort(key=lambda c: c["line"])
|
||||
|
||||
return {"modified_tex": "".join(lines), "changes": changes, "warnings": warnings}
|
||||
@@ -1,223 +0,0 @@
|
||||
"""services/self_review.py —— 植物基因组学专属自审稿（System Prompt + 报告模型）。
|
||||
|
||||
六大领域维度检查清单硬编码在 SYSTEM_PROMPT（发给 DeepSeek）：
|
||||
1. 基因组组装质量（N50/L50/BUSCO 完整格式/Hi-C 挂载率；硬性缺项打 hard_rule 标签）
|
||||
2. 基因注释可靠性（MAKER/AUGUSTUS 流程、RNA-seq 证据、基因数与已知范围一致性）
|
||||
3. 比较基因组学与进化（Bootstrap/后验概率、OrthoFinder 等直系同源策略）
|
||||
4. 数据可用性合规（SRA/BioProject + 组装/GFF3 Accession，缺失即硬性 major）
|
||||
5. 统计与实验设计（GWAS/QTL 样本量·多重校正·群体结构；转录组生物学重复）
|
||||
6. 伦理与合规·植物特异性（凭证标本 voucher、ABS 惠益分享声明——均为硬性 major）
|
||||
|
||||
2026-10-12 判定链重构：LLM 不再输出 recommendation（旧配额制 "major 3-8 条" 导致
|
||||
accept/minor 几乎不可达，见 recommend_engine.py 模块 docstring）；只如实列 weakness
|
||||
并打 hard_rule/fatal 标签，recommendation 由规则引擎计算。Major/Minor 每条必须给
|
||||
Evidence（论文章节位置或具体指标数值），由 coerce 宽容归一化（缺失补空串，不丢条目）。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from .recommend_engine import HARD_RULE_TAGS
|
||||
|
||||
SYSTEM_PROMPT = """你是一位植物基因组学领域资深审稿人，长期为 Plant Cell、Nature Plants、
|
||||
Genome Biology、Molecular Plant、Plant Journal 等期刊审稿。请对用户提供的论文正文做
|
||||
投稿前自审（pre-submission self review），严格按以下六个植物基因组学专属维度逐项检查，
|
||||
输出结构化审稿意见。所有条目使用中文撰写（专业术语、指标名、工具名保留英文）。
|
||||
|
||||
【维度一：基因组组装质量】必查指标：
|
||||
- contig N50 / scaffold N50 / L50（报了哪个要写明哪个）
|
||||
- BUSCO 完整度：必须按完整格式报告——例如
|
||||
"Complete 95.3% [S:93.1%, D:2.2%], Missing 4.7%, n=1614 (embryophyta_odb10)"
|
||||
（含 Complete/Single-copy/Duplicated/Missing 四项与数据集版本、n 值）
|
||||
- Hi-C 挂载率（anchoring rate，chromosome-level 比例）及挂载到染色体的 scaffold 比例
|
||||
BUSCO 完整度三级判定：
|
||||
- 完全未提 BUSCO → Major Weakness，hard_rule="busco_missing"
|
||||
- 提了 BUSCO 但只有总百分比（如 98.57%），缺 Single-copy/Duplicated/Missing
|
||||
分项或数据集版本（如 n=1614, embryophyta_odb10）→ 列 Minor（不打 hard_rule）
|
||||
- 完整格式（如 Complete 95.3% [S:93.1%, D:2.2%], Missing 4.7%, n=1614）→ 通过，不列
|
||||
Hi-C/挂载率报告识别（命中任一且给出数值即视为已报告，不得判缺）：
|
||||
anchoring rate / anchored to chromosome(s) / % of sequences anchored /
|
||||
pseudo-chromosome(s) / chromosome-level 挂载比例。
|
||||
判定规则：若组装类论文只报 N50 而未报 BUSCO 且未报 Hi-C 挂载率，列为
|
||||
Major Weakness 并打对应 hard_rule 标签（BUSCO 缺失→"busco_missing"；宣称
|
||||
chromosome-level/Hi-C 挂载但未报挂载率→"hic_missing"）。同时检查组装是否为
|
||||
pseudo-chromosome 级别、污染评估（如 k-mer/coverage 检查）与 gap 比例。
|
||||
|
||||
【维度二：基因注释可靠性】检查：
|
||||
- 是否使用标准注释流程（MAKER、AUGUSTUS、BRAKER、GeMoMa、EVidenceModeler 等），参数与版本是否交代
|
||||
- 是否有转录组证据支持：RNA-seq（组织/时期/物种内或近缘种）比对支撑比例，或蛋白同源证据
|
||||
- 基因数量是否与该物种已知基因数范围一致（如水稻 ~37-40k、拟南芥 ~27k、玉米 ~39-45k、
|
||||
小麦 ~107k、大豆 ~46k；近缘种偏差过大须说明）；有无转座元件基因误注释的排查
|
||||
- 注释质量评估：BRAKER/AED 分值、BUSCO 在蛋白集上的复核
|
||||
|
||||
【维度三：比较基因组学与进化】检查：
|
||||
- 系统发育树：每个节点建议标注 Bootstrap 支持率（建议 ≥1000 次重复）或贝叶斯后验概率。
|
||||
分诊：
|
||||
* 完全未做节点支持率分析（方法与正文均无任何声明）且支撑核心进化结论 → Major
|
||||
* 方法/正文声明已做 bootstrap 或后验概率，仅图中未标注数值 → presentation_format
|
||||
（图注补标即可，不需重跑分析）
|
||||
* 附带性质的树无支持率 → Minor
|
||||
- 直系同源基因鉴定策略是否说明清楚：OrthoFinder/OrthoMCL/RBX 等工具、版本、参数、
|
||||
聚类阈值（inflation/coverage）、single-copy 标记基因的筛选方式
|
||||
- 分子进化分析是否规范：dN/dS 检验方法与多重校正、共线性（synteny）分析参数、
|
||||
分化时间估算是否校准（化石/二次校准点）
|
||||
|
||||
【维度四：数据可用性合规——极其重要】必查（在 Data Availability 节或
|
||||
Methods 数据声明处核对，不因正文引用他人数据的 accession 而误判）：
|
||||
- 原始测序数据是否上传公共数据库并给出 Accession 号。前缀清单：
|
||||
PRJNA/PRJEB（NCBI/ENA BioProject）、PRJCA/CRA（中国 NGDC/GSA）、
|
||||
SRP/ERP/DRP（SRA 项目）、SRR/ERR/DRR（run 级号）——命中任一即视为已有
|
||||
原始数据 accession
|
||||
- 基因组组装是否有 GenBank/ENA accession（GCA_/GCF_），GFF3 注释文件、
|
||||
蛋白/CDS 文件是否有可公开获取的位置（仓库 accession 或补充数据链接）
|
||||
- 转录组数据是否同样有 SRA accession
|
||||
判定规则：组装/注释类论文若缺失上述任何一环（含 "will be deposited" 等未给号的
|
||||
承诺式表述），列为 Major Weakness 并打 hard_rule 标签 "accession_missing"。
|
||||
|
||||
【维度五：统计与实验设计】检查：
|
||||
- GWAS/QTL 定位：样本量/群体大小是否充分；多重检验校正是否明确（Bonferroni/FDR/
|
||||
permutation 阈值/PCA 显著性水平）；群体结构控制（PCA、亲缘关系矩阵 K、分层分析）
|
||||
是否交代；表型测量描述是否充分
|
||||
- 转录组分析：生物学重复数量（≥3 为底线）、差异分析工具（DESeq2/edgeR 等）与
|
||||
校正方法、批次效应处理、|log2FC| 与 padj 阈值
|
||||
- 其他统计：t 检验/ANOVA 是否满足前提、重复类型（生物学 vs 技术重复）写明与否
|
||||
|
||||
【维度六：伦理与合规（植物特异性）】检查：
|
||||
- 野生植物材料：是否提供凭证标本（voucher specimen）信息——标本馆编号、存放机构、
|
||||
采集地点（GPS）与采集许可；无凭证标本的野生资源取样 → Major Weakness 并打
|
||||
hard_rule 标签 "voucher_missing"
|
||||
- 遗传资源获取与惠益分享（ABS）合规性：是否声明符合《名古屋议定书》/本国遗传资源
|
||||
获取条例（如中国《生物安全法》/《种子法》相关条款）、是否说明材料来源与授权；
|
||||
涉及遗传资源但缺 ABS/名古屋合规声明 → Major Weakness 并打 hard_rule 标签
|
||||
"abs_missing"
|
||||
- 农业转基因/基因编辑材料的合规声明（若涉及）
|
||||
- fatal 标签：仅当你发现明确的数据造假或伦理违规证据时，将该条 "fatal" 置 true
|
||||
（数据内部矛盾、明显编造迹象等；无明确证据绝不可打标——该标签会触发拒稿建议）。
|
||||
|
||||
输出要求（必须输出单个 JSON 对象，不要输出其他任何内容；不要输出 recommendation
|
||||
字段——推荐结论由程序规则引擎根据你列出的问题与标签计算）：
|
||||
{
|
||||
"summary": "总体评价（3-5 句，须点明论文类型与所适用的维度）",
|
||||
"major_weaknesses": [{"title": "...", "detail": "...", "evidence": "...", "suggestion": "...",
|
||||
"hard_rule": "busco_missing|hic_missing|accession_missing|voucher_missing|abs_missing|null",
|
||||
"fatal": false}],
|
||||
"minor_weaknesses": [{"title": "...", "detail": "...", "evidence": "...", "suggestion": "..."}],
|
||||
"presentation_format": [{"title": "...", "detail": "...", "evidence": "...", "suggestion": "..."}],
|
||||
"questions_for_authors": ["..."]
|
||||
}
|
||||
|
||||
【严重程度分级——据此分类，不设条数配额】
|
||||
- major_weaknesses：问题的解决需要补充实验/补充数据/补充分析流程/大幅改写论证结构，
|
||||
或影响论文核心结论的可信度与数据合规性。列在这里意味着"不解决就不能发表"。
|
||||
- minor_weaknesses：在现有数据与实验范围内即可解决（补一段说明、补一个基于已有数据的
|
||||
统计、澄清方法描述、局部重写）。列在这里意味着"改了更好，不改也能审"。
|
||||
- presentation_format：仅呈现层面的问题（图表样式、图注格式、参考文献体例、语法）。
|
||||
严禁把呈现问题升格为 major。
|
||||
|
||||
【负向清单——以下情况严禁列为 major_weaknesses，按对应类别归档】
|
||||
- 图表未标注统计量（bootstrap 数值/误差棒/显著性标记/P 值），但方法或正文已声明
|
||||
做过该分析 → presentation_format 或 minor_weaknesses
|
||||
- 方法描述不够详细，但所用方法是领域标准做法且论证逻辑完整 → minor_weaknesses
|
||||
- 阈值/参数未给出具体数值，但逻辑完整且结论不依赖精确数值 → minor_weaknesses
|
||||
- 引用格式/排版/图注体例/语法 → presentation_format
|
||||
- 标准流程的合理省略（如 IQ-TREE/DESeq2 等工具默认参数未逐一列出）→ 不列
|
||||
总纲：只有"需要补做实验/补数据/重跑分析/改核心论证"的问题才允许列为 major；
|
||||
其余一律 minor 或 presentation。与上文各维度冲突时以本节为准。
|
||||
|
||||
分级与条数规则：
|
||||
1. major_weaknesses、minor_weaknesses、presentation_format 均【不设下限，可以为 0 条】。
|
||||
论文确实完善时输出空数组是正确答案；严禁为凑数而降低分级标准或制造问题。
|
||||
2. **每条 major_weaknesses 的 evidence 字段必填**：必须引用论文的具体章节位置
|
||||
（如 "Methods 2.1"、"Table 2"、"Results 第 3 段"、"Data Availability 一句话未见"）
|
||||
或具体指标数值（如 "仅报 scaffold N50=42.6 Mb，未提供 BUSCO 与 Hi-C 挂载率"）。
|
||||
evidence 严禁编造：引用不到具体位置时，写明"正文未见相关报告"。只列有 evidence
|
||||
的问题：列不出具体章节位置或指标数值的问题，不要列。
|
||||
3. minor_weaknesses/presentation_format 的 evidence 尽量给出，无法定位时可留空字符串。
|
||||
4. 论文不涉及的维度（如纯 RNA-seq 论文不谈组装）在 summary 中说明并跳过，不硬凑条目。
|
||||
5. 宁可把拿不准严重程度的问题放 minor_weaknesses（由作者判断），不要放 major。
|
||||
|
||||
硬性规则（命中必须列 major 并打 hard_rule 标签，不论其他方面多完善；规则不可协商）：
|
||||
A. 组装类论文缺 BUSCO 完整格式报告 → hard_rule="busco_missing"
|
||||
B. 宣称 chromosome-level/Hi-C 挂载但未报挂载率 → hard_rule="hic_missing"
|
||||
C. 缺数据 Accession（SRA/GenBank/组装/GFF3；"will be deposited" 等承诺式表述视为缺）
|
||||
→ hard_rule="accession_missing"
|
||||
D. 野生植物材料缺凭证标本（voucher）→ hard_rule="voucher_missing"
|
||||
E. 涉及遗传资源但缺 ABS/名古屋合规声明 → hard_rule="abs_missing"
|
||||
（核心进化结论所依赖的系统发育树若**完全未做**节点支持率分析 → 列 major（非硬性，
|
||||
不打标签）；若方法/正文已声明分析但图中未标数值 → presentation/minor；附带性质
|
||||
的树无支持率 → minor。GWAS 缺多重校正等统计问题按同样原则判级：阈值/方法完全
|
||||
缺失且影响结论 → major；已声明校正逻辑仅缺细节 → minor。）
|
||||
|
||||
presentation_format 覆盖：图表规范（图注含单位/误差棒说明）、参考文献格式、
|
||||
字数与结构、英文语法（若投英文刊）。
|
||||
"""
|
||||
|
||||
|
||||
class WeaknessItem(BaseModel):
|
||||
title: str
|
||||
detail: str = ""
|
||||
evidence: str = ""
|
||||
suggestion: str = ""
|
||||
hard_rule: str | None = None # 仅 major 可带；白名单外置 None（见 coerce）
|
||||
fatal: bool = False # 明确造假/伦理证据才可 true（触发 R0 拒稿）
|
||||
# 扫描器冲突降级标记（程序层置位，LLM 输出不读/coerce 不透传 → 不可伪造）：
|
||||
# True 表示该条原为 major，因关键词扫描器证伪其 hard_rule 缺项声明被降级至 minor
|
||||
scanner_downgraded: bool = False
|
||||
|
||||
|
||||
class SelfReviewReport(BaseModel):
|
||||
summary: str = ""
|
||||
major_weaknesses: list[WeaknessItem] = Field(default_factory=list)
|
||||
minor_weaknesses: list[WeaknessItem] = Field(default_factory=list)
|
||||
presentation_format: list[WeaknessItem] = Field(default_factory=list)
|
||||
questions_for_authors: list[str] = Field(default_factory=list)
|
||||
# recommendation 由 recommend_engine 计算后写入；LLM 输出的同名字段一律忽略
|
||||
recommendation: str = ""
|
||||
recommendation_rationale: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
def build_user_prompt(text: str, target_journal: str) -> str:
|
||||
header = f"目标期刊：{target_journal}\n\n" if target_journal else ""
|
||||
return f"{header}论文正文：\n\n{text}"
|
||||
|
||||
|
||||
def _coerce_items(raw: object) -> list[WeaknessItem]:
|
||||
items: list[WeaknessItem] = []
|
||||
if not isinstance(raw, list):
|
||||
return items
|
||||
for entry in raw:
|
||||
if isinstance(entry, str) and entry.strip():
|
||||
items.append(WeaknessItem(title=entry.strip()))
|
||||
elif isinstance(entry, dict):
|
||||
title = str(entry.get("title") or "").strip()
|
||||
detail = str(entry.get("detail") or "").strip()
|
||||
if not title and not detail:
|
||||
continue
|
||||
hard_rule = str(entry.get("hard_rule") or "").strip() or None
|
||||
if hard_rule not in HARD_RULE_TAGS:
|
||||
hard_rule = None # 白名单外的标签丢弃（防 LLM 编造标签操纵引擎）
|
||||
items.append(WeaknessItem(
|
||||
title=title or "（未命名）",
|
||||
detail=detail,
|
||||
evidence=str(entry.get("evidence") or "").strip(),
|
||||
suggestion=str(entry.get("suggestion") or "").strip(),
|
||||
hard_rule=hard_rule,
|
||||
fatal=bool(entry.get("fatal")),
|
||||
))
|
||||
return items
|
||||
|
||||
|
||||
def coerce_report(payload: dict) -> SelfReviewReport:
|
||||
"""宽容归一化 LLM 输出。不读 recommendation 字段（引擎唯一来源，防注入操纵）。"""
|
||||
questions = payload.get("questions_for_authors") or []
|
||||
if not isinstance(questions, list):
|
||||
questions = []
|
||||
return SelfReviewReport(
|
||||
summary=str(payload.get("summary") or "").strip(),
|
||||
major_weaknesses=_coerce_items(payload.get("major_weaknesses")),
|
||||
minor_weaknesses=_coerce_items(payload.get("minor_weaknesses")),
|
||||
presentation_format=_coerce_items(payload.get("presentation_format")),
|
||||
questions_for_authors=[str(q).strip() for q in questions if str(q).strip()],
|
||||
)
|
||||
@@ -1,493 +0,0 @@
|
||||
{
|
||||
"version": 2,
|
||||
"updated": "2026-10-10",
|
||||
"notes": "会议种子数据 v2。deadline_source=placeholder 表示日期为首版占位数据（按各会议公开周期窗口/历年规律填写，未经官方渠道逐条核验），接入官方数据源前须人工核验；严禁以 AI 推测日期替代核验。deadline_confidence 标注日期置信度：high=官方已公布或历年极稳定；medium=按历年规律区间推算；low=新一届尚未公告或历届波动大，待核验。ccf_rank 按 CCF 推荐目录（第七版，2026-03 发布）标注，非目录内会议为 null（含全部生物信息/植物类会议）。timezone_note 默认 AoE (UTC-12)。首版不做自动同步。",
|
||||
"conferences": [
|
||||
{
|
||||
"id": "naacl-2027",
|
||||
"name": "NAACL 2027",
|
||||
"full_name": "North American Chapter of the Association for Computational Linguistics",
|
||||
"domain_tags": ["AI/ML", "自然语言处理"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "B",
|
||||
"deadline_abstract": null,
|
||||
"deadline_full_paper": "2026-10-12T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-06-01 ~ 2027-06-05",
|
||||
"location": "San Francisco, California, USA",
|
||||
"official_url": "https://www.naacl.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "high"
|
||||
},
|
||||
{
|
||||
"id": "coling-2027",
|
||||
"name": "COLING 2027",
|
||||
"full_name": "International Conference on Computational Linguistics",
|
||||
"domain_tags": ["AI/ML", "自然语言处理"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "B",
|
||||
"deadline_abstract": null,
|
||||
"deadline_full_paper": "2026-10-12T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-05-09 ~ 2027-05-14",
|
||||
"location": "Macau SAR, China",
|
||||
"official_url": null,
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "www-2027",
|
||||
"name": "WWW 2027",
|
||||
"full_name": "The Web Conference (ACM WWW)",
|
||||
"domain_tags": ["AI/ML", "数据挖掘"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "A",
|
||||
"deadline_abstract": "2026-10-18T23:59:59+00:00",
|
||||
"deadline_full_paper": "2026-10-25T23:59:59+00:00",
|
||||
"notification_date": "2027-01-04",
|
||||
"conference_date": "2027-05-10 ~ 2027-05-14",
|
||||
"location": "Dublin, Ireland",
|
||||
"official_url": "https://www2027.thewebconf.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "high"
|
||||
},
|
||||
{
|
||||
"id": "recomb-2027",
|
||||
"name": "RECOMB 2027",
|
||||
"full_name": "International Conference on Research in Computational Molecular Biology",
|
||||
"domain_tags": ["生物信息学"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": "2026-11-06T23:59:59+00:00",
|
||||
"deadline_full_paper": "2026-11-13T23:59:59+00:00",
|
||||
"notification_date": "2027-02-01",
|
||||
"conference_date": "2027-04-25 ~ 2027-04-28",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": "https://recomb.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "cvpr-2027",
|
||||
"name": "CVPR 2027",
|
||||
"full_name": "IEEE/CVF Conference on Computer Vision and Pattern Recognition",
|
||||
"domain_tags": ["AI/ML", "计算机视觉"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "A",
|
||||
"deadline_abstract": "2026-11-10T23:59:59+00:00",
|
||||
"deadline_full_paper": "2026-11-16T23:59:59+00:00",
|
||||
"notification_date": "2027-02-25",
|
||||
"conference_date": "2027-06-20 ~ 2027-06-25",
|
||||
"location": "Seattle, USA",
|
||||
"official_url": "https://cvpr.thecvf.com/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "high"
|
||||
},
|
||||
{
|
||||
"id": "wsdm-2027",
|
||||
"name": "WSDM 2027",
|
||||
"full_name": "International Conference on Web Search and Data Mining",
|
||||
"domain_tags": ["AI/ML", "数据挖掘"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "B",
|
||||
"deadline_abstract": null,
|
||||
"deadline_full_paper": "2026-11-17T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-02-15 ~ 2027-02-19",
|
||||
"location": "Hong Kong SAR, China",
|
||||
"official_url": "https://www.wsdm-conference.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "high"
|
||||
},
|
||||
{
|
||||
"id": "pag-2027",
|
||||
"name": "PAG 2027",
|
||||
"full_name": "Plant & Animal Genome Conference",
|
||||
"domain_tags": ["植物/农学", "基因组学"],
|
||||
"application_tags": ["种业", "分子育种"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": "2026-11-30T23:59:59+00:00",
|
||||
"deadline_full_paper": "2026-12-15T23:59:59+00:00",
|
||||
"notification_date": "2027-01-05",
|
||||
"conference_date": "2027-01-08 ~ 2027-01-12",
|
||||
"location": "San Diego, USA",
|
||||
"official_url": "https://www.intlpag.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "acl-2027",
|
||||
"name": "ACL 2027",
|
||||
"full_name": "Annual Meeting of the Association for Computational Linguistics",
|
||||
"domain_tags": ["AI/ML", "自然语言处理"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "A",
|
||||
"deadline_abstract": null,
|
||||
"deadline_full_paper": "2027-01-04T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-08-20 ~ 2027-08-22",
|
||||
"location": "Kyoto, Japan",
|
||||
"official_url": "https://2027.aclweb.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "high"
|
||||
},
|
||||
{
|
||||
"id": "plantbio-2027",
|
||||
"name": "Plant Biology 2027",
|
||||
"full_name": "ASPB Annual Meeting — Plant Biology",
|
||||
"domain_tags": ["植物/农学", "分子生物学"],
|
||||
"application_tags": ["分子育种"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": "2027-01-08T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-01-08T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-07（待官方公布）",
|
||||
"location": "Albuquerque, New Mexico, USA",
|
||||
"official_url": "https://plantbiology.aspb.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "high"
|
||||
},
|
||||
{
|
||||
"id": "ijcai-2027",
|
||||
"name": "IJCAI 2027",
|
||||
"full_name": "International Joint Conference on Artificial Intelligence",
|
||||
"domain_tags": ["AI/ML"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "B",
|
||||
"deadline_abstract": "2027-01-11T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-01-18T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-08-07 ~ 2027-08-17",
|
||||
"location": "Kyoto, Japan / Hengqin, China",
|
||||
"official_url": "https://www.ijcai.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "sigir-2027",
|
||||
"name": "SIGIR 2027",
|
||||
"full_name": "ACM SIGIR Conference on Research and Development in Information Retrieval",
|
||||
"domain_tags": ["AI/ML", "信息检索"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "A",
|
||||
"deadline_abstract": "2027-01-14T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-01-21T23:59:59+00:00",
|
||||
"notification_date": "2027-04-05",
|
||||
"conference_date": "2027-07-20 ~ 2027-07-24",
|
||||
"location": "San Jose, California, USA",
|
||||
"official_url": "https://sigir2027.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "high"
|
||||
},
|
||||
{
|
||||
"id": "icml-2027",
|
||||
"name": "ICML 2027",
|
||||
"full_name": "International Conference on Machine Learning",
|
||||
"domain_tags": ["AI/ML", "深度学习"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "A",
|
||||
"deadline_abstract": "2027-01-15T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-01-22T23:59:59+00:00",
|
||||
"notification_date": "2027-05-01",
|
||||
"conference_date": "2027-07-19 ~ 2027-07-25",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": "https://icml.cc/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "chil-2027",
|
||||
"name": "CHIL 2027",
|
||||
"full_name": "ACM Conference on Health, Inference, and Learning",
|
||||
"domain_tags": ["AI/ML", "健康医疗"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": "2027-01-15T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-01-29T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-06（待官方公布）",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": "https://dl.acm.org/conference/chil",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "low"
|
||||
},
|
||||
{
|
||||
"id": "kdd-2027",
|
||||
"name": "KDD 2027 (Cycle 2)",
|
||||
"full_name": "ACM SIGKDD Conference on Knowledge Discovery and Data Mining",
|
||||
"domain_tags": ["AI/ML", "数据挖掘"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "A",
|
||||
"deadline_abstract": "2027-01-31T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-02-07T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-08-01 ~ 2027-08-05",
|
||||
"location": "San Jose, California, USA",
|
||||
"official_url": "https://kdd2027.kdd.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "ismb-eccb-2027",
|
||||
"name": "ISMB/ECCB 2027",
|
||||
"full_name": "Intelligent Systems for Molecular Biology / European Conference on Computational Biology",
|
||||
"domain_tags": ["生物信息学"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": "2027-02-12T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-02-19T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-07-18 ~ 2027-07-22",
|
||||
"location": "Copenhagen, Denmark",
|
||||
"official_url": "https://www.iscb.org/ismbeccb2027/home",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "ipmbc-2027",
|
||||
"name": "IPMB 2027",
|
||||
"full_name": "International Congress on Plant Molecular Biology (14th)",
|
||||
"domain_tags": ["植物/农学", "分子生物学"],
|
||||
"application_tags": ["分子育种"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": "2027-02-15T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-02-22T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-07（待官方公布）",
|
||||
"location": "Wuhan, China",
|
||||
"official_url": "https://www.ipmb2027.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "low"
|
||||
},
|
||||
{
|
||||
"id": "uai-2027",
|
||||
"name": "UAI 2027",
|
||||
"full_name": "Conference on Uncertainty in Artificial Intelligence",
|
||||
"domain_tags": ["AI/ML"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "B",
|
||||
"deadline_abstract": null,
|
||||
"deadline_full_paper": "2027-02-24T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-08-16 ~ 2027-08-20",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": "https://www.auai.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "miccai-2027",
|
||||
"name": "MICCAI 2027",
|
||||
"full_name": "International Conference on Medical Image Computing and Computer-Assisted Intervention",
|
||||
"domain_tags": ["AI/ML", "医学影像"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": "2027-02-11T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-02-25T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-09-26 ~ 2027-10-01",
|
||||
"location": "Auckland, New Zealand",
|
||||
"official_url": "https://miccai.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "iccv-2027",
|
||||
"name": "ICCV 2027",
|
||||
"full_name": "IEEE/CVF International Conference on Computer Vision",
|
||||
"domain_tags": ["AI/ML", "计算机视觉"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "A",
|
||||
"deadline_abstract": "2027-02-26T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-03-05T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-10-02 ~ 2027-10-08",
|
||||
"location": "Hong Kong SAR, China",
|
||||
"official_url": null,
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "bosc-2027",
|
||||
"name": "BOSC 2027",
|
||||
"full_name": "Bioinformatics Open Source Conference",
|
||||
"domain_tags": ["生物信息学"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": null,
|
||||
"deadline_full_paper": "2027-03-26T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-07-18 ~ 2027-07-22",
|
||||
"location": "Copenhagen, Denmark",
|
||||
"official_url": "https://openbioinformatics.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "low"
|
||||
},
|
||||
{
|
||||
"id": "acm-bcb-2027",
|
||||
"name": "ACM-BCB 2027",
|
||||
"full_name": "ACM Conference on Bioinformatics, Computational Biology and Health Informatics",
|
||||
"domain_tags": ["生物信息学", "AI/ML"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": "2027-04-16T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-04-23T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-08（待官方公布）",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": "https://acm-bcb.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "ieee-bibm-2027",
|
||||
"name": "IEEE BIBM 2027",
|
||||
"full_name": "IEEE International Conference on Bioinformatics and Bioengineering",
|
||||
"domain_tags": ["生物信息学"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": "2027-05-07T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-05-14T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-11（待官方公布）",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": null,
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "low"
|
||||
},
|
||||
{
|
||||
"id": "emnlp-2027",
|
||||
"name": "EMNLP 2027",
|
||||
"full_name": "Conference on Empirical Methods in Natural Language Processing",
|
||||
"domain_tags": ["AI/ML", "自然语言处理"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "B",
|
||||
"deadline_abstract": "2027-05-14T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-05-21T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-11（待官方公布）",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": null,
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "low"
|
||||
},
|
||||
{
|
||||
"id": "neurips-2027",
|
||||
"name": "NeurIPS 2027",
|
||||
"full_name": "Conference on Neural Information Processing Systems",
|
||||
"domain_tags": ["AI/ML", "深度学习"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "A",
|
||||
"deadline_abstract": "2027-05-14T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-05-21T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-12（待官方公布）",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": "https://neurips.cc/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
},
|
||||
{
|
||||
"id": "psb-2028",
|
||||
"name": "PSB 2028",
|
||||
"full_name": "Pacific Symposium on Biocomputing",
|
||||
"domain_tags": ["生物信息学"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": null,
|
||||
"deadline_full_paper": "2027-07-01T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2028-01（待官方公布）",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": "https://psb.stanford.edu/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "low"
|
||||
},
|
||||
{
|
||||
"id": "pag-europe-2027",
|
||||
"name": "PAG Europe 2027",
|
||||
"full_name": "Plant & Animal Genome Conference Europe",
|
||||
"domain_tags": ["植物/农学", "基因组学"],
|
||||
"application_tags": ["种业", "分子育种"],
|
||||
"ccf_rank": null,
|
||||
"deadline_abstract": null,
|
||||
"deadline_full_paper": "2027-07-09T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2027-10（待官方公布）",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": null,
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "low"
|
||||
},
|
||||
{
|
||||
"id": "aaai-2028",
|
||||
"name": "AAAI 2028",
|
||||
"full_name": "AAAI Conference on Artificial Intelligence",
|
||||
"domain_tags": ["AI/ML"],
|
||||
"application_tags": ["交叉学科"],
|
||||
"ccf_rank": "A",
|
||||
"deadline_abstract": "2027-07-20T23:59:59+00:00",
|
||||
"deadline_full_paper": "2027-07-27T23:59:59+00:00",
|
||||
"notification_date": null,
|
||||
"conference_date": "2028-02（待官方公布）",
|
||||
"location": "TBD（待官方公布）",
|
||||
"official_url": "https://aaai.org/",
|
||||
"timezone_note": "AoE (UTC-12)",
|
||||
"deadline_source": "placeholder",
|
||||
"last_verified": "2026-10-10",
|
||||
"deadline_confidence": "medium"
|
||||
}
|
||||
]
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user