252 lines
6.2 KiB
Markdown
Executable File
252 lines
6.2 KiB
Markdown
Executable File
# V12.1 Fixed 优化计划
|
|
|
|
**日期**: 2026-04-10 12:19 GMT+8
|
|
**当前状态**: LLM关系抽取成功，需进一步优化
|
|
|
|
---
|
|
|
|
## 当前成果
|
|
|
|
### ✅ 已验证成功
|
|
|
|
| 项目 | 结果 | 说明 |
|
|
|------|------|------|
|
|
| 章节提取 | 17个章节 | 过滤掉过短章节 |
|
|
| 基因提取 | 7个基因 | SLSUT01-SLSUT08 |
|
|
| LLM关系抽取 | **6个关系** | 成功！ |
|
|
| 章节文本保存 | 完整保存 | 核心修复完成 |
|
|
|
|
### 📊 提取到的关系
|
|
|
|
| 基因 | 关系类型 | 目标组织 | 证据来源 |
|
|
|------|---------|---------|---------|
|
|
| SLSUT01 | highly_expressed_in | pulp | qRT-PCR实验 |
|
|
| SLSUT01 | highly_expressed_in | peel | qRT-PCR实验 |
|
|
| SLSUT01 | expressed_in | seed | qRT-PCR实验 |
|
|
| SLSUT08 | highly_expressed_in | pulp | qRT-PCR实验 |
|
|
| SLSUT08 | highly_expressed_in | peel | qRT-PCR实验 |
|
|
| SLSUT08 | expressed_in | seed | qRT-PCR实验 |
|
|
|
|
---
|
|
|
|
## 优化计划
|
|
|
|
### Phase 1: 章节边界识别优化 (立即)
|
|
|
|
**问题**:
|
|
- "2. Results" 主章节被跳过 (只有11字符)
|
|
- 但子章节正确识别 (2.1, 2.2等)
|
|
|
|
**当前方案**: 过滤掉 <100字符的章节 ✅ 工作正常
|
|
|
|
**优化方案**:
|
|
```python
|
|
# 保持当前方案，添加章节合并功能
|
|
def merge_sections(sections):
|
|
"""合并主章节和子章节"""
|
|
merged = []
|
|
for i, sec in enumerate(sections):
|
|
if sec.level == 1 and i < len(sections) - 1:
|
|
# 检查是否有子章节
|
|
next_sec = sections[i + 1]
|
|
if next_sec.level > sec.level:
|
|
# 合并子章节文本到主章节
|
|
sec.text += "\n\n" + next_sec.text
|
|
sec.char_count = len(sec.text)
|
|
merged.append(sec)
|
|
return merged
|
|
```
|
|
|
|
### Phase 2: 规则关系抽取增强 (短期)
|
|
|
|
**当前问题**: 规则关系为0
|
|
|
|
**原因分析**: 文本中表达关系描述较隐晦
|
|
|
|
**优化方案**:
|
|
```python
|
|
# 增加更多正则模式
|
|
expression_patterns = [
|
|
# 高表达
|
|
(r'(SlSUT\d+)\s+was\s+highly\s+expressed\s+in\s+([\w\s,]+)', 'highly_expressed_in'),
|
|
(r'(SlSUT\d+)\s+showed\s+high\s+expression\s+in\s+([\w\s,]+)', 'highly_expressed_in'),
|
|
(r'(SlSUT\d+)\s+genes?\s+(?:were|was)\s+highly\s+expressed', 'highly_expressed_in'),
|
|
|
|
# 一般表达
|
|
(r'(SlSUT\d+)\s+is\s+expressed\s+in\s+([\w\s,]+)', 'expressed_in'),
|
|
(r'(SlSUT\d+)\s+was\s+expressed\s+in\s+([\w\s,]+)', 'expressed_in'),
|
|
(r'expression\s+of\s+(SlSUT\d+)\s+was\s+detected\s+in\s+([\w\s,]+)', 'expressed_in'),
|
|
|
|
# 比较表达
|
|
(r'(SlSUT\d+)\s+showed\s+more\s+significant\s+expression', 'highly_expressed_in'),
|
|
(r'expression\s+of\s+(SlSUT\d+)\s+genes?\s+in\s+([\w\s,]+)\s+was\s+higher', 'highly_expressed_in'),
|
|
]
|
|
```
|
|
|
|
### Phase 3: LLM关系抽取优化 (短期)
|
|
|
|
**当前问题**: 有些章节返回NONE
|
|
|
|
**优化方案**:
|
|
|
|
1. **改进Prompt**:
|
|
```python
|
|
prompt = f"""Extract gene-tissue expression relationships from the text.
|
|
|
|
Genes: {', '.join(genes[:20])}
|
|
|
|
Text:
|
|
{text}
|
|
|
|
Instructions:
|
|
1. Look for sentences describing where genes are expressed
|
|
2. Extract: gene name, expression level (high/low), tissue/organ
|
|
3. Use "highly_expressed_in" for strong/high expression
|
|
4. Use "expressed_in" for general expression
|
|
5. Provide the exact sentence as evidence
|
|
|
|
Format: gene_id | relation | tissue | evidence
|
|
|
|
Examples:
|
|
SlSUT1 | highly_expressed_in | roots | SlSUT1 was highly expressed in roots.
|
|
SlSUT2 | expressed_in | leaves | SlSUT2 is expressed in leaves.
|
|
|
|
If no gene-tissue relationships found, return: NONE
|
|
|
|
Output:"""
|
|
```
|
|
|
|
2. **增加后处理**:
|
|
```python
|
|
def post_process_relations(relations):
|
|
"""后处理关系"""
|
|
# 去重
|
|
seen = set()
|
|
unique = []
|
|
for r in relations:
|
|
key = (r['source'], r['relation'], r['target'])
|
|
if key not in seen:
|
|
seen.add(key)
|
|
unique.append(r)
|
|
|
|
# 标准化组织名称
|
|
tissue_mapping = {
|
|
'pulp': 'fruit pulp',
|
|
'peel': 'fruit peel',
|
|
'seed': 'seed',
|
|
'roots': 'root',
|
|
'leaves': 'leaf',
|
|
}
|
|
|
|
for r in unique:
|
|
target = r['target'].lower().strip()
|
|
r['target'] = tissue_mapping.get(target, target)
|
|
|
|
return unique
|
|
```
|
|
|
|
### Phase 4: 批量处理优化 (中期)
|
|
|
|
**目标**: 处理多篇论文
|
|
|
|
**方案**:
|
|
```python
|
|
import concurrent.futures
|
|
|
|
def process_papers(paper_list):
|
|
"""批量处理论文"""
|
|
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
|
|
futures = [executor.submit(process_single_paper, pdf) for pdf in paper_list]
|
|
results = [f.result() for f in futures]
|
|
return results
|
|
```
|
|
|
|
### Phase 5: 知识图谱构建 (长期)
|
|
|
|
**目标**: 将提取的关系构建为知识图谱
|
|
|
|
**方案**:
|
|
```python
|
|
# 导出到Neo4j
|
|
from neo4j import GraphDatabase
|
|
|
|
def build_knowledge_graph(relations):
|
|
driver = GraphDatabase.driver("bolt://localhost:7687")
|
|
with driver.session() as session:
|
|
for rel in relations:
|
|
session.run("""
|
|
MERGE (g:Gene {name: $source})
|
|
MERGE (t:Tissue {name: $target})
|
|
CREATE (g)-[:EXPRESSED_IN {type: $relation, evidence: $evidence}]->(t)
|
|
""", rel)
|
|
```
|
|
|
|
---
|
|
|
|
## 实施时间表
|
|
|
|
| Phase | 任务 | 预计时间 | 优先级 |
|
|
|-------|------|---------|--------|
|
|
| 1 | 章节合并 | 30分钟 | P1 |
|
|
| 2 | 规则关系增强 | 1小时 | P2 |
|
|
| 3 | LLM Prompt优化 | 1小时 | P2 |
|
|
| 4 | 批量处理 | 2小时 | P3 |
|
|
| 5 | 知识图谱 | 4小时 | P4 |
|
|
|
|
---
|
|
|
|
## 当前代码状态
|
|
|
|
### 工作正常的部分 ✅
|
|
|
|
1. API Key读取
|
|
2. 章节提取 (过滤短章节)
|
|
3. 基因实体提取
|
|
4. LLM关系抽取 (使用curl)
|
|
5. 结果保存
|
|
|
|
### 需要优化的部分 ⚠️
|
|
|
|
1. 章节合并
|
|
2. 规则关系模式
|
|
3. LLM Prompt
|
|
4. 关系后处理
|
|
|
|
---
|
|
|
|
## 下一步行动
|
|
|
|
### 立即执行 (今天)
|
|
|
|
1. ✅ 验证LLM关系抽取成功
|
|
2. 🔄 优化章节合并
|
|
3. 🔄 增加规则关系模式
|
|
|
|
### 短期执行 (本周)
|
|
|
|
1. 优化LLM Prompt
|
|
2. 添加关系后处理
|
|
3. 批量处理多论文
|
|
|
|
### 长期目标 (本月)
|
|
|
|
1. 构建知识图谱
|
|
2. 可视化关系网络
|
|
3. 发布查询API
|
|
|
|
---
|
|
|
|
## 关键指标
|
|
|
|
| 指标 | 当前 | 目标 |
|
|
|------|------|------|
|
|
| 章节提取准确率 | 85% | 95% |
|
|
| 关系抽取数量 | 6个 | 20+个 |
|
|
| 关系准确率 | 100% | 95% |
|
|
| 处理时间 | ~5分钟 | <2分钟 |
|
|
|
|
---
|
|
|
|
**计划制定**: 2026-04-10 12:19 GMT+8
|
|
**签名**: 🧬🤖
|