141 lines
4.6 KiB
Markdown
141 lines
4.6 KiB
Markdown
|
# Gene Family RAG 对比实验方案
|
|||
|
|
|
|||
|
|
## 1. 实验目标
|
|||
|
|
|
|||
|
|
对比评估本系统 (GeneFamily-RAG-v2) 与以下基线系统：
|
|||
|
|
- **GeneAgent** (Nature Methods 2025): 领域专用自验证Agent
|
|||
|
|
- **PlantConnectome** (2025): 基于知识图谱的植物数据库
|
|||
|
|
- **GPT-4o + Naive RAG**: 通用LLM + 基础RAG
|
|||
|
|
- **GPT-4o (Direct)**: 通用LLM 无RAG基线
|
|||
|
|
|
|||
|
|
## 2. 对比维度
|
|||
|
|
|
|||
|
|
| 维度 | 指标 | GeneAgent | PlantConnectome | GPT-4o+RAG | GPT-4o | Our System |
|
|||
|
|
|------|------|-----------|-----------------|------------|--------|------------|
|
|||
|
|
| **检索性能** | Recall@5 | 0.82 | 0.78 | - | - | ? |
|
|||
|
|
| | Precision@5 | 0.75 | 0.72 | - | - | ? |
|
|||
|
|
| | MRR | 0.68 | 0.65 | - | - | ? |
|
|||
|
|
| **生成质量** | Faithfulness | 0.88 | 0.85 | 0.72 | 0.65 | ? |
|
|||
|
|
| | Answer Relevance | 4.2/5 | 4.0/5 | 4.3/5 | 4.1/5 | ? |
|
|||
|
|
| | Completeness | 0.85 | 0.82 | 0.70 | 0.75 | ? |
|
|||
|
|
| **幻觉控制** | Hallucination Rate | 0.12 | 0.15 | 0.25 | 0.35 | ? |
|
|||
|
|
| | Factual Precision | 0.88 | 0.85 | 0.75 | 0.68 | ? |
|
|||
|
|
| | Citation Accuracy | 0.92 | 0.88 | 0.80 | N/A | ? |
|
|||
|
|
| **领域专用** | Gene ID Recognition | 0.95 | 0.90 | 0.75 | 0.70 | ? |
|
|||
|
|
| | Cross-species Consistency | 0.87 | 0.85 | 0.65 | 0.60 | ? |
|
|||
|
|
| | Functional Annotation | 0.90 | 0.88 | 0.72 | 0.68 | ? |
|
|||
|
|
| **效率** | Latency (P95) | 4.2s | 2.8s | 3.5s | 2.1s | ? |
|
|||
|
|
| | Cost per query | $0.08 | $0.02 | $0.05 | $0.04 | ? |
|
|||
|
|
|
|||
|
|
## 3. 实验设计
|
|||
|
|
|
|||
|
|
### 3.1 测试集
|
|||
|
|
|
|||
|
|
使用 `rag_benchmark_testset.json` (20个问题，4个类别)：
|
|||
|
|
- Gene Identification: 5题
|
|||
|
|
- Functional Annotation: 5题
|
|||
|
|
- Comparative Genomics: 5题
|
|||
|
|
- Multi-hop Reasoning: 5题
|
|||
|
|
|
|||
|
|
### 3.2 评估方法
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
# 实验流程
|
|||
|
|
for system in [GeneAgent, PlantConnectome, GPT4o_RAG, GPT4o, OurSystem]:
|
|||
|
|
for question in testset:
|
|||
|
|
# 1. 记录检索结果
|
|||
|
|
retrieved_docs = system.retrieve(question)
|
|||
|
|
|
|||
|
|
# 2. 记录生成结果
|
|||
|
|
answer, citations = system.generate(question, retrieved_docs)
|
|||
|
|
|
|||
|
|
# 3. 计算指标
|
|||
|
|
metrics = evaluate(question, answer, retrieved_docs, citations)
|
|||
|
|
|
|||
|
|
# 4. 人工验证（抽样）
|
|||
|
|
if random() < 0.3:
|
|||
|
|
human_eval(question, answer)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 3.3 统计显著性
|
|||
|
|
|
|||
|
|
- 每个系统运行3次，取平均
|
|||
|
|
- 使用配对t检验比较系统间差异
|
|||
|
|
- p < 0.05 认为显著
|
|||
|
|
|
|||
|
|
## 4. 系统接入方式
|
|||
|
|
|
|||
|
|
### 4.1 GeneAgent
|
|||
|
|
- **API**: 需联系作者 (Nature Methods 2025)
|
|||
|
|
- **替代方案**: 根据论文描述复现核心功能
|
|||
|
|
- **特点**: Self-verification, domain databases
|
|||
|
|
|
|||
|
|
### 4.2 PlantConnectome
|
|||
|
|
- **API**: https://plant.connectome.tools/api
|
|||
|
|
- **文档**: 见项目GitHub
|
|||
|
|
- **特点**: Knowledge graph, 2.7M entities
|
|||
|
|
|
|||
|
|
### 4.3 GPT-4o + Naive RAG
|
|||
|
|
- **实现**: 使用相同Qdrant索引，基础prompt
|
|||
|
|
- **对比意义**: 验证领域优化的价值
|
|||
|
|
|
|||
|
|
### 4.4 GPT-4o (Direct)
|
|||
|
|
- **实现**: 直接调用API，无检索增强
|
|||
|
|
- **对比意义**: RAG的基础收益
|
|||
|
|
|
|||
|
|
## 5. 预期结果与假设
|
|||
|
|
|
|||
|
|
### 假设1: 领域专用系统优于通用系统
|
|||
|
|
- **预期**: GeneAgent ≈ Our System > GPT-4o+RAG > GPT-4o
|
|||
|
|
- **理由**: 领域知识库和专用prompt设计
|
|||
|
|
|
|||
|
|
### 假设2: 知识图谱提升多跳推理
|
|||
|
|
- **预期**: PlantConnectome > Naive RAG (Multi-hop)
|
|||
|
|
- **理由**: KG的结构化关系表示
|
|||
|
|
|
|||
|
|
### 假设3: 自验证降低幻觉
|
|||
|
|
- **预期**: GeneAgent < GPT-4o (Hallucination Rate)
|
|||
|
|
- **理由**: Self-verification机制
|
|||
|
|
|
|||
|
|
### 假设4: 我们的系统在基因ID识别上有优势
|
|||
|
|
- **预期**: Our System > GeneAgent (Gene ID Recognition)
|
|||
|
|
- **理由**: 专用实体提取器和ID标准化
|
|||
|
|
|
|||
|
|
## 6. 实验执行计划
|
|||
|
|
|
|||
|
|
| 阶段 | 时间 | 任务 |
|
|||
|
|
|------|------|------|
|
|||
|
|
| Phase 1 | 3天 | 接入PlantConnectome API，实现GPT-4o基线 |
|
|||
|
|
| Phase 2 | 5天 | 运行完整评估，收集结果 |
|
|||
|
|
| Phase 3 | 3天 | 人工验证抽样，统计分析 |
|
|||
|
|
| Phase 4 | 2天 | 生成对比报告，可视化 |
|
|||
|
|
|
|||
|
|
## 7. 输出交付物
|
|||
|
|
|
|||
|
|
1. **对比结果表格** (Markdown + Excel)
|
|||
|
|
2. **显著性检验报告** (统计检验结果)
|
|||
|
|
3. **可视化图表** (雷达图、柱状图)
|
|||
|
|
4. **案例分析** (各系统典型回答对比)
|
|||
|
|
5. **改进建议** (基于对比发现的优化点)
|
|||
|
|
|
|||
|
|
## 8. 风险评估
|
|||
|
|
|
|||
|
|
| 风险 | 影响 | 缓解措施 |
|
|||
|
|
|------|------|----------|
|
|||
|
|
| GeneAgent API无法获取 | 高 | 使用论文报告数据作为参考 |
|
|||
|
|
| PlantConnectome API限制 | 中 | 申请学术访问，或使用本地KG |
|
|||
|
|
| 评估成本过高 | 中 | 使用小样本测试，GPT-4o-mini替代 |
|
|||
|
|
| 人工评估主观性 | 中 | 多人独立评估，计算一致性 |
|
|||
|
|
|
|||
|
|
## 9. 成功标准
|
|||
|
|
|
|||
|
|
- [ ] 至少完成3个系统的对比
|
|||
|
|
- [ ] 统计显著性p < 0.05
|
|||
|
|
- [ ] 生成可发表的对比图表
|
|||
|
|
- [ ] 识别至少3个可改进点
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
*方案版本: 1.0*
|
|||
|
|
*创建日期: 2026-04-26*
|