454 lines
14 KiB
Markdown
Executable File
454 lines
14 KiB
Markdown
Executable File
# PC 实现方案详细设计
|
||
|
||
## PC 数据库特征分析
|
||
|
||
基于对 PC 数据库的观察，PC 的实现方式具有以下特征：
|
||
|
||
### 1. PC 的核心字段
|
||
```
|
||
- source: 源实体 (如 "32 SPL gene(s) (CiSPLs)")
|
||
- source type: 源实体类型 (如 "gene", "gene identifier")
|
||
- relationship: 关系 (如 "identified in")
|
||
- target: 目标实体 (如 "pecan genome")
|
||
- target type: 目标实体类型 (如 "organism")
|
||
- p_source: 章节来源 (如 "34966577_abstract")
|
||
- basis: 证据文本
|
||
- species: 物种
|
||
```
|
||
|
||
### 2. PC 的实体识别特点
|
||
|
||
**复合实体识别**：
|
||
- PC 识别 `"32 SPL gene(s) (CiSPLs)"` 作为一个整体
|
||
- PC 识别 `"SPL gene(s) family"` 作为群体实体
|
||
- PC 识别 `"nine segmentally duplicated gene(s) pairs"` 作为复合概念
|
||
|
||
**实体类型丰富**：
|
||
- `gene`: 单个基因
|
||
- `gene identifier`: 基因标识符
|
||
- `phenotype`: 表型
|
||
- `process`: 过程
|
||
- `organism`: 生物体
|
||
- `treatment`: 处理条件
|
||
|
||
### 3. PC 的关系抽取特点
|
||
|
||
**基于证据文本**：
|
||
- PC 的 `basis` 字段包含完整的证据句子
|
||
- 关系从证据句子中抽取
|
||
|
||
**章节溯源**：
|
||
- PC 记录关系来源的章节 (`p_source`)
|
||
- 如 `34966577_abstract`, `34966577_results1`
|
||
|
||
**关系类型标准化**：
|
||
- PC 使用标准化的关系类型
|
||
- 如 `identified in`, `collinear with`, `involved in`
|
||
|
||
---
|
||
|
||
## 最接近 PC 的详细实现方案
|
||
|
||
### 方案概述
|
||
|
||
采用 **"复合实体识别 + 基于证据的关系抽取 + 章节溯源"** 的三层架构
|
||
|
||
### 第一层：复合实体识别
|
||
|
||
#### 1.1 实体类型扩展
|
||
|
||
```python
|
||
ENTITY_TYPES = {
|
||
# 基础实体
|
||
'gene': r'\b[A-Z][a-z]{1,2}[A-Z]{1,4}\d{1,4}[a-z]?\b',
|
||
|
||
# 复合实体 - 数量+基因
|
||
'gene_group_count': r'(\d+)\s+([A-Z][a-z]{1,2}[A-Z]{1,4})\s+gene\(s\)',
|
||
|
||
# 复合实体 - 数量+基因+别名
|
||
'gene_group_with_alias': r'(\d+)\s+([A-Z][a-z]{1,2}[A-Z]{1,4})\s+gene\(s\)\s+\(\s*([A-Z][a-z]{1,2}[A-Z]{1,4})s\s*\)',
|
||
|
||
# 复合实体 - 基因家族
|
||
'gene_family': r'([A-Z][a-z]{1,2}[A-Z]{1,4})\s+gene\(s\)\s+family',
|
||
|
||
# 复合实体 - 重复基因对
|
||
'duplicated_gene_pairs': r'(\d+|\w+)\s+segmentally\s+duplicated\s+gene\(s\)?\s+pairs?',
|
||
|
||
# 复合实体 - 共线性基因对
|
||
'collinear_gene_pairs': r'(\d+)\s+collinear\s+gene\(s\)?\s+pairs?',
|
||
|
||
# 转录因子群体
|
||
'transcription_factor_group': r'([A-Z][a-z]{1,2}[A-Z]{1,4})\s+transcription\s+factors?',
|
||
}
|
||
```
|
||
|
||
#### 1.2 实体识别流程
|
||
|
||
```python
|
||
class CompositeEntityRecognizer:
|
||
def recognize(self, text):
|
||
entities = []
|
||
|
||
# 步骤1: 识别基础基因实体
|
||
for match in re.finditer(ENTITY_TYPES['gene'], text):
|
||
entities.append({
|
||
'name': match.group(),
|
||
'type': 'gene',
|
||
'start': match.start(),
|
||
'end': match.end()
|
||
})
|
||
|
||
# 步骤2: 识别复合实体
|
||
for match in re.finditer(ENTITY_TYPES['gene_group_with_alias'], text):
|
||
entities.append({
|
||
'name': f"{match.group(1)} {match.group(2)} gene(s) ({match.group(3)}s)",
|
||
'type': 'gene_group',
|
||
'start': match.start(),
|
||
'end': match.end(),
|
||
'count': match.group(1),
|
||
'family': match.group(2),
|
||
'alias': match.group(3)
|
||
})
|
||
|
||
# 步骤3: 实体链接和消歧
|
||
# 将 "CiSPLs" 链接到 "32 SPL gene(s) (CiSPLs)"
|
||
for entity in entities:
|
||
if entity['type'] == 'gene_group':
|
||
# 找到该群体中的所有单个基因
|
||
member_genes = self._find_member_genes(text, entity)
|
||
entity['member_genes'] = member_genes
|
||
|
||
return entities
|
||
|
||
def _find_member_genes(self, text, group_entity):
|
||
"""找到复合实体中的成员基因"""
|
||
# 例如从 "32 SPL gene(s) (CiSPLs)" 中找到所有 CiSPL 基因
|
||
alias = group_entity.get('alias', '')
|
||
pattern = rf'{alias}\d+[a-z]?'
|
||
return re.findall(pattern, text)
|
||
```
|
||
|
||
### 第二层：基于证据的关系抽取
|
||
|
||
#### 2.1 证据句子提取
|
||
|
||
```python
|
||
class EvidenceExtractor:
|
||
def extract_evidence_sentences(self, text, entity):
|
||
"""提取包含实体的证据句子"""
|
||
sentences = []
|
||
|
||
# 分割句子
|
||
sentence_pattern = r'[^.!?]+[.!?]+'
|
||
all_sentences = re.findall(sentence_pattern, text)
|
||
|
||
for sent in all_sentences:
|
||
# 检查句子是否包含实体
|
||
if entity['name'] in sent or self._entity_in_sentence(entity, sent):
|
||
sentences.append({
|
||
'text': sent.strip(),
|
||
'entity': entity,
|
||
'confidence': self._calculate_confidence(entity, sent)
|
||
})
|
||
|
||
return sentences
|
||
|
||
def _entity_in_sentence(self, entity, sentence):
|
||
"""检查实体是否在句子中（支持别名匹配）"""
|
||
# 基础名称匹配
|
||
if entity['name'] in sentence:
|
||
return True
|
||
|
||
# 别名匹配
|
||
if entity.get('alias') and entity['alias'] in sentence:
|
||
return True
|
||
|
||
# 成员基因匹配
|
||
if entity.get('member_genes'):
|
||
for gene in entity['member_genes']:
|
||
if gene in sentence:
|
||
return True
|
||
|
||
return False
|
||
```
|
||
|
||
#### 2.2 关系模式匹配（PC风格）
|
||
|
||
```python
|
||
PC_RELATION_PATTERNS = [
|
||
# identified in 模式
|
||
{
|
||
'relation': 'identified in',
|
||
'patterns': [
|
||
r'(\d+\s+[A-Z][a-z]+[A-Z]+\s+gene\(s\)\s+\([A-Z][a-z]+[A-Z]+s\))\s+were\s+identified\s+in\s+([^\.]+)',
|
||
r'(\d+\s+[A-Z][a-z]+[A-Z]+\s+gene\(s\))\s+were\s+identified\s+in\s+([^\.]+)',
|
||
r'([A-Z][a-z]+[A-Z]+\s+gene\(s\))\s+were\s+identified\s+in\s+([^\.]+)',
|
||
r'([A-Z][a-z]+[A-Z]+\d+[a-z]?)\s+was\s+identified\s+in\s+([^\.]+)',
|
||
],
|
||
'source_type': 'gene_group',
|
||
'target_type': 'organism'
|
||
},
|
||
|
||
# collinear with 模式
|
||
{
|
||
'relation': 'collinear with',
|
||
'patterns': [
|
||
r'([A-Z][a-z]+[A-Z]+\d+[a-z]?)\s+(?:is|was)\s+collinear\s+with\s+([A-Z][a-z]+[A-Z]+\d+[a-z]?)',
|
||
# 列表形式: "CiSPL3a/AtSPL3, CiSPL4b/AtSPL5"
|
||
r'([A-Z][a-z]+[A-Z]+\d+[a-z]?)/([A-Z][a-z]+[A-Z]+\d+[a-z]?)',
|
||
],
|
||
'source_type': 'gene',
|
||
'target_type': 'gene'
|
||
},
|
||
|
||
# involved in 模式
|
||
{
|
||
'relation': 'involved in',
|
||
'patterns': [
|
||
r'([A-Z][a-z]+[A-Z]+\s+gene\(s\)\s+family)\s+(?:is|are)\s+involved\s+in\s+([^\.]+)',
|
||
r'([A-Z][a-z]+[A-Z]+\s+gene\(s\))\s+(?:is|are)\s+involved\s+in\s+([^\.]+)',
|
||
r'([A-Z][a-z]+[A-Z]+\d+[a-z]?)\s+(?:is|are)\s+involved\s+in\s+([^\.]+)',
|
||
# 被动形式
|
||
r'([A-Z][a-z]+[A-Z]+\d+[a-z]?)\s+.*?\s+involved\s+in\s+([^\.]+)',
|
||
],
|
||
'source_type': 'gene',
|
||
'target_type': 'process'
|
||
},
|
||
|
||
# contain 模式
|
||
{
|
||
'relation': 'contain',
|
||
'patterns': [
|
||
r'([A-Z][a-z]+[A-Z]+\s+gene\(s\))\s+contain\s+([^\.]+)',
|
||
r'([A-Z][a-z]+[A-Z]+\d+[a-z]?)\s+contains?\s+([^\.]+)',
|
||
r'([A-Z][a-z]+[A-Z]+\s+gene\(s\))\s+had\s+([^\.]+)',
|
||
],
|
||
'source_type': 'gene',
|
||
'target_type': 'attribute'
|
||
},
|
||
]
|
||
```
|
||
|
||
### 第三层：章节溯源
|
||
|
||
```python
|
||
class SectionTracker:
|
||
def __init__(self):
|
||
self.section_markers = {
|
||
'abstract': r'ABSTRACT|Abstract',
|
||
'intro': r'INTRODUCTION|Introduction',
|
||
'methods': r'MATERIALS\s*&\s*METHODS|Methods',
|
||
'results': r'RESULTS|Results',
|
||
'discussion': r'DISCUSSION|Discussion',
|
||
'conclusion': r'CONCLUSION|Conclusion',
|
||
'references': r'REFERENCES|References',
|
||
}
|
||
|
||
def track_sections(self, text):
|
||
"""追踪文本中的章节"""
|
||
sections = []
|
||
current_pos = 0
|
||
|
||
for section_name, pattern in self.section_markers.items():
|
||
for match in re.finditer(pattern, text, re.IGNORECASE):
|
||
sections.append({
|
||
'name': section_name,
|
||
'start': match.start(),
|
||
'end': len(text) # 临时，后续会更新
|
||
})
|
||
|
||
# 排序并计算章节边界
|
||
sections.sort(key=lambda x: x['start'])
|
||
for i in range(len(sections) - 1):
|
||
sections[i]['end'] = sections[i + 1]['start']
|
||
|
||
return sections
|
||
|
||
def get_section_for_position(self, position, sections):
|
||
"""获取指定位置所属的章节"""
|
||
for section in sections:
|
||
if section['start'] <= position < section['end']:
|
||
return section['name']
|
||
return 'unknown'
|
||
|
||
|
||
### 第四层：关系抽取流程
|
||
|
||
```python
|
||
class PCKnowledgeExtractor:
|
||
"""PC 风格的知识抽取器"""
|
||
|
||
def __init__(self):
|
||
self.entity_recognizer = CompositeEntityRecognizer()
|
||
self.evidence_extractor = EvidenceExtractor()
|
||
self.section_tracker = SectionTracker()
|
||
|
||
def extract(self, text, pmid):
|
||
"""主抽取流程"""
|
||
relations = []
|
||
|
||
# 步骤1: 章节追踪
|
||
sections = self.section_tracker.track_sections(text)
|
||
|
||
# 步骤2: 实体识别
|
||
entities = self.entity_recognizer.recognize(text)
|
||
|
||
# 步骤3: 对每个实体抽取关系
|
||
for entity in entities:
|
||
# 提取证据句子
|
||
evidence_sentences = self.evidence_extractor.extract_evidence_sentences(text, entity)
|
||
|
||
for evidence in evidence_sentences:
|
||
# 匹配关系模式
|
||
for pattern_def in PC_RELATION_PATTERNS:
|
||
for pattern in pattern_def['patterns']:
|
||
matches = re.finditer(pattern, evidence['text'], re.IGNORECASE)
|
||
|
||
for match in matches:
|
||
# 确定源和目标
|
||
if entity['name'] in match.group(1) or match.group(1) in entity['name']:
|
||
source = entity['name']
|
||
target = match.group(2).strip()
|
||
else:
|
||
continue
|
||
|
||
# 确定章节
|
||
section = self.section_tracker.get_section_for_position(
|
||
evidence['text'].index(match.group(0)),
|
||
sections
|
||
)
|
||
|
||
# 构建关系
|
||
relation = {
|
||
'source': source,
|
||
'source_type': pattern_def['source_type'],
|
||
'relation': pattern_def['relation'],
|
||
'target': target,
|
||
'target_type': pattern_def['target_type'],
|
||
'evidence': evidence['text'],
|
||
'section': f"{pmid}_{section}",
|
||
'pmid': pmid,
|
||
'confidence': evidence['confidence']
|
||
}
|
||
|
||
relations.append(relation)
|
||
|
||
return relations
|
||
```
|
||
|
||
---
|
||
|
||
## 关键改进点
|
||
|
||
### 1. 复合实体识别
|
||
|
||
**PC 做法**: 识别 `"32 SPL gene(s) (CiSPLs)"` 作为一个整体实体
|
||
|
||
**V13 改进**:
|
||
```python
|
||
# 添加复合实体模式
|
||
r'(\d+)\s+([A-Z][a-z]+[A-Z]+)\s+gene\(s\)\s+\(\s*([A-Z][a-z]+[A-Z]+)s\s*\)'
|
||
|
||
# 识别结果
|
||
{
|
||
'name': '32 SPL gene(s) (CiSPLs)',
|
||
'type': 'gene_group',
|
||
'count': '32',
|
||
'family': 'SPL',
|
||
'alias': 'CiSPL',
|
||
'member_genes': ['CiSPL2a', 'CiSPL2b', ...] # 自动提取成员
|
||
}
|
||
```
|
||
|
||
### 2. 实体链接
|
||
|
||
**PC 做法**: 将 `"CiSPLs"` 链接到 `"32 SPL gene(s) (CiSPLs)"`
|
||
|
||
**V13 改进**:
|
||
```python
|
||
# 建立实体链接
|
||
for entity in entities:
|
||
if entity['type'] == 'gene_group':
|
||
# 找到所有别名引用
|
||
alias_pattern = rf"\b{entity['alias']}s?\b"
|
||
for match in re.finditer(alias_pattern, text):
|
||
# 链接到复合实体
|
||
entity['references'].append({
|
||
'text': match.group(),
|
||
'position': match.start()
|
||
})
|
||
```
|
||
|
||
### 3. 列表形式关系识别
|
||
|
||
**PC 做法**: 识别 `"CiSPL3a/AtSPL3, CiSPL4b/AtSPL5"` 中的共线性关系
|
||
|
||
**V13 改进**:
|
||
```python
|
||
# 列表形式模式
|
||
r'([A-Z][a-z]+[A-Z]+\d+[a-z]?)/([A-Z][a-z]+[A-Z]+\d+[a-z]?)'
|
||
|
||
# 从列表中提取多个关系
|
||
pairs = re.findall(pattern, "CiSPL3a/AtSPL3, CiSPL4b/AtSPL5, CiSPL6a/AtSPL6")
|
||
for source, target in pairs:
|
||
relations.append({
|
||
'source': source,
|
||
'relation': 'collinear_with',
|
||
'target': target
|
||
})
|
||
```
|
||
|
||
### 4. 灵活的模式匹配
|
||
|
||
**PC 做法**: 支持多种表达方式
|
||
- `"CiSPL gene(s) contain..."`
|
||
- `"CiSPL genes contained..."`
|
||
- `"The CiSPL genes had..."`
|
||
|
||
**V13 改进**:
|
||
```python
|
||
# 使用词干提取或模糊匹配
|
||
from nltk.stem import PorterStemmer
|
||
|
||
stemmer = PorterStemmer()
|
||
|
||
# 匹配 "contain", "contained", "contains", "had" 等
|
||
contain_stems = ['contain', 'contained', 'contains', 'had', 'have']
|
||
```
|
||
|
||
---
|
||
|
||
## 实施优先级
|
||
|
||
### 高优先级（立即实施）
|
||
|
||
1. **复合实体识别**: 识别 `"32 SPL gene(s) (CiSPLs)"` 等
|
||
2. **列表形式关系**: 识别 `"CiSPL3a/AtSPL3"` 共线性
|
||
3. **实体链接**: 将别名链接到复合实体
|
||
|
||
### 中优先级（后续实施）
|
||
|
||
4. **章节溯源**: 记录关系来源章节
|
||
5. **证据句子提取**: 提取完整证据文本
|
||
6. **词形还原**: 支持时态/词形变化
|
||
|
||
### 低优先级（可选）
|
||
|
||
7. **人工校验接口**: 支持人工确认关系
|
||
8. **机器学习优化**: 使用 NER 模型提升识别
|
||
|
||
---
|
||
|
||
## 预期效果
|
||
|
||
实施该方案后，预计：
|
||
|
||
| 指标 | 当前 | 预期 |
|
||
|:---|:---:|:---:|
|
||
| 覆盖率 | 19.9% | 35-40% |
|
||
| identified in | 0 | 15-18 |
|
||
| collinear with | 0 | 12-14 |
|
||
| involved in | 0 | 10-12 |
|
||
| contain | 0 | 5-6 |
|
||
|
||
**关键改进**: 能够识别复合实体和列表形式的关系，大幅提升 PC 关键关系的覆盖率。 |