PlantAI-rag/docs/changelogs/CHANGELOG.md

Changelog

All notable changes to the Gene Family RAG System will be documented in this file.

---

[2026-03-11] RAG 2.0 Revision - Phase 1

Overview

基于5个高级RAG策略链接的全面升级:

• [1] Advanced RAG Strategies (ottomator-agents)
• [2] Anthropic Contextual Retrieval
• [3] Pinecone Rerankers
• [4] Docling Context-Aware Chunking
• [5] Graphiti Temporal Knowledge Graphs

Added

1. Docling PDF Extractor (src/rag/docling_extractor.py)

• 高级PDF解析（表格识别、阅读顺序恢复）
• 多格式导出支持（Markdown、HTML、JSON）
• OCR扫描PDF支持
• PyMuPDF回退机制
• 状态: ✅ 代码完成，Docling已安装，测试中

2. Contextual Retrieval (src/rag/contextual_retrieval.py)

• Anthropic上下文检索实现
• 基因家族名称自动展开（SPL→SQUAMOSA PROMOTER BINDING PROTEIN-LIKE）
• 物种名称自动展开（苦荞→Fagopyrum tataricum）
• 研究方法自动识别
• Contextual BM25索引支持
• Contextual Embedding存储
• 状态: ✅ 完成并测试通过

3. Index Rebuild v2 (scripts/rebuild_index_v2.py)

• 集成Docling和Contextual Retrieval
• 智能分块（语义连贯）
• 增强元数据提取
• 状态: ✅ 代码完成，准备运行

Modified

Dependencies

• Added chardet (✅ 已安装)
• Added jieba (✅ 已安装)
• Added docling (✅ 已在gfpy312环境安装)
  • 注意: Docling使用conda环境gfpy312，非poetry环境
  • 首次运行需下载OCR模型(~40MB)

Environment Setup

# 使用conda环境运行
conda activate gfpy312
python scripts/rebuild_index_v2.py

Test Results

✅ Contextual Retrieval测试通过
   - 上下文前缀生成: 正确
   - 基因家族展开: 正确
   - 物种展开: 正确
   - 查询上下文化: 正确

✅ Docling测试通过
   - 导入成功
   - 初始化成功
   - 正在重建索引（87篇PDF）

Next Steps

1. 等待Docling依赖安装完成
2. 运行rebuild_index_v2.py重建索引
3. 更新hybrid_retriever支持contextualized_content
4. Phase 2: 向量存储升级（Qdrant）

---

---

[2026-03-09] Index Recovery & Search Fixes

Problem Diagnosis

Issue	Root Cause	Status
Search returns no results	Pickle index file corrupted	✅ Fixed
"苦荞SPL" search fails	Default mode="normal" doesn't match Chinese	✅ Fixed
Frontend search fails	Frontend overrides default mode to "normal"	✅ Fixed
SPL literature missing	PDF directory path incorrect	✅ Fixed

Fixes Applied

1. Index Reconstruction (scripts/rebuild_index.py)

   • Added correct PDF directory: /home/xia-lab/wutx/00test/bbav2.0302/paper/gfpaper/buchong/
   • Fixed Unicode encoding errors during PDF processing
   • Rebuild index with 604 chunks (includes FtSPL literature)

2. Search Mode Default (scripts/rag_api.py)

   • Changed default mode from "normal" to "substring"
   • Line 53: mode: str = "substring"

3. Frontend Search Mode (frontend/src/App.vue)

   • Changed default ragMode from 'normal' to 'substring'
   • Line 145: const ragMode = ref('substring')

4. Hybrid Retriever Fix (src/rag/hybrid_retriever.py)

   • Fixed Chunk initialization: id → chunk_id
   • Fixed SQL query: removed non-existent columns

Current Status

• Indexed Documents: 87 PDFs, 604 chunks
• SPL Literature: FtSPL (24 chunks) ✅
• Search Modes Supported: normal, exact, alias, substring, bm25, hybrid
• Default Mode: substring (supports Chinese + English matching)

---

[2026-03-08] Query Rewriting & Reranking

Added

• Query Rewriting Module (src/rag/query_rewriter.py)

  • Semantic keyword expansion for gene families
  • Species-aware expansion (e.g., "苦荞" → "tartary buckwheat")

• Reranking Module (src/rag/reranker.py)

  • BGE-Reranker integration
  • Configurable top-k selection

• Buckwheat Omics Data

  • GSE111937 (Tartary buckwheat transcriptome)

Modified

• Entity Store - Added publish_year, dataset extended fields
• Entity Extractor - Year and omics metadata extraction
• Hybrid Retriever - Integrated Query Rewriting
• RAG API - Integrated Reranking, fixed syntax errors
• Frontend - Restored CDN version

Architecture

User Query
    ↓
[Query Rewriting] ← Semantic keyword expansion
    ↓
[Hybrid Retrieval] ← BM25 + Vector RRF fusion
    ↓
[Reranking] ← BGE-Reranker
    ↓
[Entity Linking] ← Attach datasets/publication year
    ↓
Final Response

---

[Previous Versions]

v0.1 - Initial RAG System

• Basic hybrid retrieval (BM25 + Vector)
• Entity extraction and storage
• Basic chat API