279 lines
8.4 KiB
Markdown
Executable File
279 lines
8.4 KiB
Markdown
Executable File
# 提取器依赖关系分析
|
|
|
|
## 📊 分析说明
|
|
|
|
本文档分析各提取器之间的依赖关系，为归档操作提供参考。
|
|
|
|
---
|
|
|
|
## 🎯 核心提取器依赖图
|
|
|
|
```
|
|
根目录提取器 (独立运行)
|
|
├── extractor_fusion_complete.py [独立，无外部依赖]
|
|
├── extractor_fusion_v1.py [独立]
|
|
├── extractor_pc_based.py [依赖: rule_base_full.json]
|
|
├── llm_extractor_dashscope.py [依赖: dashscope API]
|
|
├── llm_relation_extractor.py [依赖: dashscope API]
|
|
└── universal_extractor_v2.py [待分析]
|
|
|
|
src/rag/extractors/ (模块间有依赖)
|
|
├── extract_v12_final.py [依赖: supplementary_downloader, multi_db_extractor, data_availability_extractor]
|
|
├── extract_v12_1_fixed.py [依赖: 内置规则]
|
|
├── supplementary_downloader.py [独立工具]
|
|
├── gene_entity_filter.py [独立工具]
|
|
├── gene_centric_chunker.py [独立工具]
|
|
├── section_supplementer.py [独立工具]
|
|
├── geo_extractor.py [独立，NCBI API]
|
|
├── multi_db_extractor.py [独立工具]
|
|
├── data_availability_extractor.py [依赖: multi_db_extractor]
|
|
└── unified_data_extractor.py [依赖: multi_db_extractor, data_availability_extractor]
|
|
|
|
scripts/extractors/ (相对独立)
|
|
├── extract_v10_with_supplementary.py [独立]
|
|
└── extract_v7-v9/*.py [旧版，独立]
|
|
```
|
|
|
|
---
|
|
|
|
## 📋 详细依赖分析
|
|
|
|
### 1. 根目录提取器
|
|
|
|
#### ✅ extractor_fusion_complete.py
|
|
```python
|
|
# 依赖：仅标准库 + PyMuPDF
|
|
import re
|
|
import json
|
|
from pathlib import Path
|
|
from dataclasses import dataclass, field, asdict
|
|
from typing import List, Dict, Optional, Set
|
|
import fitz # PyMuPDF
|
|
```
|
|
**结论**：完全独立，可安全移动
|
|
|
|
#### ✅ extractor_fusion_v1.py
|
|
**结论**：备份版本，可归档
|
|
|
|
#### ⚠️ extractor_pc_based.py
|
|
```python
|
|
# 依赖：rule_base_full.json (硬编码路径)
|
|
rule_base_path = Path("/home/xia-lab/gene-family-mas-v2/rule_base_full.json")
|
|
```
|
|
**结论**：依赖规则库文件，移动时需保持相对路径或更新路径
|
|
|
|
#### ✅ llm_extractor_dashscope.py
|
|
**结论**：仅依赖dashscope SDK，可移动
|
|
|
|
#### ✅ llm_relation_extractor.py
|
|
**结论**：仅依赖dashscope SDK，可移动
|
|
|
|
#### ✅ universal_extractor_v2.py
|
|
**结论**：待验证，但通常独立
|
|
|
|
---
|
|
|
|
### 2. src/rag/extractors/ 模块
|
|
|
|
#### 🔗 extract_v12_final.py
|
|
```python
|
|
# 内部依赖 (相对导入)
|
|
from .supplementary_downloader import SupplementaryDownloader
|
|
from .multi_db_extractor import MultiDatabaseExtractor, DatabaseEntry, DataType, Database
|
|
from .data_availability_extractor import DataAvailabilityExtractor, DataAvailabilityEntry
|
|
```
|
|
**结论**：使用相对导入，模块间耦合，需一起移动或保持结构
|
|
|
|
#### 🔗 extract_v12_1_fixed.py
|
|
```python
|
|
# 内置规则，无外部文件依赖
|
|
# 但有内部方法 _extract_relations_rule_based
|
|
```
|
|
**结论**：相对独立，但建议保留在extractors包内
|
|
|
|
#### ✅ supplementary_downloader.py
|
|
**结论**：独立工具类，可单独使用
|
|
|
|
#### 🔗 data_availability_extractor.py
|
|
```python
|
|
# 条件导入
|
|
from .multi_db_extractor import MultiDatabaseExtractor
|
|
```
|
|
**结论**：依赖multi_db_extractor
|
|
|
|
#### 🔗 unified_data_extractor.py
|
|
```python
|
|
from .multi_db_extractor import MultiDatabaseExtractor, DatabaseEntry
|
|
from .data_availability_extractor import DataAvailabilityExtractor, DataAvailabilityEntry
|
|
```
|
|
**结论**：依赖multi_db_extractor和data_availability_extractor
|
|
|
|
#### ✅ 其他工具 (独立)
|
|
- gene_entity_filter.py
|
|
- gene_centric_chunker.py
|
|
- section_supplementer.py
|
|
- geo_extractor.py
|
|
- multi_db_extractor.py
|
|
|
|
---
|
|
|
|
### 3. 脚本中的导入
|
|
|
|
#### 从src.rag导入的脚本
|
|
```bash
|
|
# 以下脚本从 src.rag 导入:
|
|
scripts/build_rag_*.py -> from src.rag.pdf_extractor import PDFExtractor
|
|
scripts/rebuild_index_batch.py -> from src.rag.docling_extractor import ...
|
|
scripts/process_gfpaper_root.py -> from src.rag.docling_extractor import ...
|
|
scripts/test_rag.py -> from src.rag.pdf_extractor import PDFExtractor
|
|
scripts/build_index_from_structured.py -> from src.rag.pdf_extractor import PDFExtractor
|
|
|
|
# 测试脚本:
|
|
test_litchi_v12.py -> from src.rag.extractors.extract_v12_final import ...
|
|
```
|
|
**结论**：这些脚本依赖src.rag包结构，归档时不能破坏
|
|
|
|
---
|
|
|
|
### 4. 规则库依赖
|
|
|
|
```python
|
|
# 硬编码路径引用 rule_base_full.json:
|
|
extractor_pc_based.py: /home/xia-lab/gene-family-mas-v2/rule_base_full.json
|
|
rule_base_loader.py: /home/xia-lab/gene-family-mas-v2/rule_base_full.json
|
|
|
|
# 其他规则文件:
|
|
rule_base_complete.py [Python模块]
|
|
rule_base_priority.json [JSON数据]
|
|
rule_base_loader.py [加载器]
|
|
```
|
|
**结论**：归档时需更新硬编码路径或使用相对路径
|
|
|
|
---
|
|
|
|
## 🗂️ 安全归档策略
|
|
|
|
### 方案A：保守归档 (推荐)
|
|
|
|
**保持不动的文件**:
|
|
```
|
|
# src/rag/ 结构必须保持
|
|
src/rag/extractors/*.py [全部保留]
|
|
src/rag/pdf_extractor.py [被脚本依赖]
|
|
src/rag/docling_extractor.py [被脚本依赖]
|
|
src/rag/entity_extractor.py [被entity_store依赖]
|
|
|
|
# 根目录核心文件
|
|
extractor_fusion_complete.py [主提取器]
|
|
rule_base_full.json [被pc_based依赖]
|
|
```
|
|
|
|
**可归档的文件**:
|
|
```
|
|
# 根目录 - 归档到 extractors/archive/
|
|
extractor_suc5*.py [6个]
|
|
extractor_pc_*.py (除pc_based) [8个，pc_based保留]
|
|
extractor_complete_v1.py
|
|
extractor_enhanced.py
|
|
extractor_final.py
|
|
extractor_hybrid*.py
|
|
extractor_llm_v1.py
|
|
extractor_production_v1.py
|
|
extractor_v2_pro.py
|
|
universal_extractor_v1.py
|
|
|
|
# 文档 - 归档到 docs/
|
|
*.md (除README和EXTRACTOR_CATALOG)
|
|
|
|
# 脚本 - 归档到 scripts/shell/
|
|
*.sh
|
|
|
|
# 测试 - 归档到 tests/
|
|
test_*.py
|
|
test_*.json
|
|
test_*.md
|
|
```
|
|
|
|
### 方案B：激进归档
|
|
|
|
**需要修改代码的归档**:
|
|
```
|
|
# 如果要移动 extractor_pc_based.py:
|
|
# 需要修改其中的硬编码路径:
|
|
# rule_base_path = Path("/home/xia-lab/gene-family-mas-v2/rule_base_full.json")
|
|
# ->
|
|
# rule_base_path = Path(__file__).parent.parent / "data" / "rule_bases" / "rule_base_full.json"
|
|
```
|
|
|
|
---
|
|
|
|
## 📋 归档操作清单
|
|
|
|
### 阶段1：安全归档 (无需修改代码)
|
|
|
|
| 操作 | 命令 | 风险 |
|
|
|------|------|------|
|
|
| 归档SUC5提取器 | `mv extractor_suc5*.py extractors/archive/` | 无 |
|
|
| 归档PC实验版 | `mv extractor_pc_122_types.py extractor_pc_complete*.py ... extractors/archive/` | 低 |
|
|
| 归档旧版提取器 | `mv extractor_complete_v1.py ... extractors/archive/` | 无 |
|
|
| 归档文档 | `mv *.md docs/` (保留README) | 无 |
|
|
| 归档脚本 | `mv *.sh scripts/shell/` | 低 |
|
|
|
|
### 阶段2：需要验证的归档
|
|
|
|
| 操作 | 验证方法 | 风险 |
|
|
|------|----------|------|
|
|
| 移动rule_base*.json | 检查extractor_pc_based.py路径 | 中 |
|
|
| 重构src/rag/extractors | 运行test_litchi_v12.py | 高 |
|
|
|
|
### 阶段3：不建议归档
|
|
|
|
| 文件/目录 | 原因 |
|
|
|-----------|------|
|
|
| `src/rag/extractors/` | 模块间有依赖，脚本引用 |
|
|
| `src/rag/pdf_extractor.py` | 被多个脚本导入 |
|
|
| `src/rag/docling_extractor.py` | 被rebuild_index_batch.py导入 |
|
|
| `extractor_fusion_complete.py` | 主提取器，保持根目录方便使用 |
|
|
|
|
---
|
|
|
|
## 🔧 归档后路径更新
|
|
|
|
### 如果移动 rule_base_full.json:
|
|
|
|
```python
|
|
# extractor_pc_based.py 第X行修改:
|
|
# 原:
|
|
rule_base_path = Path("/home/xia-lab/gene-family-mas-v2/rule_base_full.json")
|
|
|
|
# 改为:
|
|
rule_base_path = Path(__file__).parent / "data" / "rule_bases" / "rule_base_full.json"
|
|
# 或:
|
|
import os
|
|
rule_base_path = Path(os.environ.get("RULE_BASE_PATH", "/home/xia-lab/gene-family-mas-v2/rule_base_full.json"))
|
|
```
|
|
|
|
### 如果创建根目录软链接:
|
|
|
|
```bash
|
|
# 归档后创建软链接方便访问:
|
|
ln -s extractors/current/fusion/extractor_fusion_complete.py extractor.py
|
|
ln -s docs/guides/EXTRACTOR_CATALOG.md EXTRACTOR_CATALOG.md
|
|
```
|
|
|
|
---
|
|
|
|
## ✅ 建议执行顺序
|
|
|
|
1. **先备份**: `git commit -m "before organization"`
|
|
2. **创建目录**: `mkdir -p extractors/archive docs/{changelogs,reports,plans,analysis,guides} scripts/shell`
|
|
3. **归档文档**: 移动所有.md到docs/
|
|
4. **归档提取器**: 移动SUC5和PC实验版到extractors/archive/
|
|
5. **归档脚本**: 移动.sh到scripts/shell/
|
|
6. **测试**: 运行 `python extractor_fusion_complete.py --help` 验证
|
|
7. **创建软链接**: 根目录保留快捷方式
|
|
|
|
---
|
|
|
|
*分析时间：2026-04-12*
|