PlantAI-rag Migration Notes

This repository is a local working copy derived from the original gene-family-mas-v2 project.

Current Rules

Do not edit the source project configured by PLANTAI_SOURCE_ROOT; it remains read-only.
Keep current conda environment names for now.
Keep large data/model assets out of the copied repository. Use symlinks or environment variables.
Keep real API keys, passwords, and private local settings out of tracked files.
Keep Git local-only during this cleanup phase: do not add a remote and do not push.
Treat src/rag/extractors/v16_1_2 as the latest extractor entrypoint. It still inherits and calls V15-era implementation files, so edits must include that call chain.

Local Paths

Project root: this repository.
Source project: the original private checkout; do not edit it from this copy.
Large data, papers, PlantConnectome resources, bbav5 workspaces, and model assets are external resources configured through .env or local symlinks.

Configuration

Use .env locally and .env.example as the shareable template. External paths and secrets should enter through env variables, not hard-coded source paths.

Important variables:

PLANTAI_CONDA_ROOT
PLANTAI_CONDA_PROFILE
PC_DATA_ROOT
PLANTAI_SPLIT_DATA_ROOT
PLANTAI_SOURCE_ROOT
RAG_OUTPUT_DIR
RAG_LEARNING_DIR
BGE_RERANKER_ROOT
BBAV5_ROOT
MINERU_CMD
DEEPSEEK_API_KEY
DASHSCOPE_API_KEY
NEO4J_PASSWORD
GF_SECURITY_ADMIN_PASSWORD

Validated Local Baseline

The current local baseline was revalidated on 2026-06-25 using one real Pecan SPL paper, the MinerU CPU pipeline, DeepSeek V4 (deepseek-v4-flash), and the V16.1.2 extractor chain with required PlantConnectome data enabled. Secrets remain only in private .env; no API key is recorded in tracked files.

Strict validation command shape:

${GF_PYTHON:-python} tools/validate_v16_baseline.py \
  --pdf "${PLANTAI_SOURCE_ROOT:?set PLANTAI_SOURCE_ROOT}/output/8ec26cde-ec16-438b-ac2f-5375aae06508/uploads/pecan_SPL_2021.pdf" \
  --pmid 34966577 \
  --output-dir /tmp/plantai_v16_1_2_llm_pc_strict_after_key \
  --cpu-mineru \
  --with-llm \
  --no-vlm \
  --require-pc-data

Validated result:

Status: VALIDATION_PASSED
Output: /tmp/plantai_v16_1_2_llm_pc_strict_after_key
Result JSON: /tmp/plantai_v16_1_2_llm_pc_strict_after_key/result.json
Sections: 11
Chunks: 33
Entities: 712
Relations: 751
LLM relations before evidence gate: 635
LLM candidate relations generated: 642
Rejected LLM candidates: 7
Retained relation fields: 0 missing source, 0 missing target, 0 missing evidence
PC data loaded: 43,561 aliases, 17,618 gene aliases, 788 relationship patterns, 3,636 entity-dictionary sample entries

Runtime note: the strict run had 43 LLM requests and 2 request-timeout failures. The extractor continued, generated candidates from the remaining windows, and passed final validation. This is acceptable for the current baseline but should remain visible in validation_summary.json so timeout frequency can be tracked across future multi-paper runs.

Multi-Paper Strict Regression

On 2026-06-25, the V16.1.2 validation wrapper was run against three real PDFs from the source project's data/input/v16_batch10_20260625 collection. The runs used MinerU CPU mode, DeepSeek V4 relation extraction, required PC data, and wrote all outputs under /tmp/plantai_v16_multi_paper_regression_20260625. No source data or split-data files were copied, cleaned, deduplicated, or modified.

During this regression, --with-llm was narrowed to mean LLM relation extraction only. Per-entity LLM definition generation is now opt-in through --with-llm-definitions, because the previous coupled behavior could add a large, low-observability API tail after relation extraction had already completed. This preserves the definition feature while keeping baseline regression focused on MinerU, PC loading, relation extraction, evidence gates, and final result integrity.

Paper output	Status	Sections	Chunks	Entities	Relations	Before gate	LLM requests	Candidate rows	Rejected rows
`plantsci_2020`	passed	10	24	393	322	357	29	361	4
`aob_mcz153`	passed	8	28	646	590	601	49	752	151
`bmc_s12870_2020`	passed	11	24	631	619	659	33	728	69

All three strict summaries reported PC data available and 0 missing source, 0 missing target, and 0 missing evidence. The bmc_s12870_2020 run was executed after the definition-generation split and confirmed the intended runtime log: LLM定义: 禁用 with LLM关系抽取: 启用.

GPU/API MinerU attempts previously failed due exhausted GPU memory, so CPU pipeline remains the verified fallback. VLM validation is still pending and requires a private DASHSCOPE_API_KEY.

PC_DATA_ROOT is configured through private .env as data/pc. In this local copy, data is a symlink to the source project's data directory, so PC resources are read in place without copying, cleaning, deduplicating, or modifying data.

Phase 2 Validation Entry

The maintained regression entry is:

${GF_PYTHON:-python} tools/validate_v16_baseline.py \
  --pdf "${PLANTAI_TEST_PDF:?set PLANTAI_TEST_PDF to a real local PDF}" \
  --pmid 34966577 \
  --output-dir experiments/phase2_v16_baseline_pecan \
  --cpu-mineru \
  --no-llm \
  --no-vlm

This is a real-paper validation, not a demo. It records validation_summary.json next to result.json.

For strict LLM relation validation, configure DEEPSEEK_API_KEY in private .env and use --with-llm --no-vlm. Add --with-llm-definitions only when the validation target explicitly includes per-entity generated definitions. Add --require-pc-data only after PC_DATA_ROOT points to a readable external directory containing the required PC JSON files.

Current Public Surface

The only supported runtime surface during this cleanup phase is the V16 extractor chain plus the validation/configuration helpers. The source of truth is config/runtime_surface_manifest.json; tools/runtime_surface.py, tools/audit_open_source_readiness.py, and tools/audit_migration_requirements.py all read that manifest instead of maintaining separate runtime file lists.

src.rag.extractors.v16_1_2.V16Extractor
src.rag.extractors.v16_1_2.paper_metadata_enricher_v16_1_2
src.rag.unified
scripts/run_v16_1_2_one_paper.py
scripts/run_v16_batch10_20260625.py
scripts/backfill_v16_paper_metadata.py
tools/validate_v16_baseline.py
tools/audit_functional_surface_parity.py
tools/audit_migration_requirements.py
tools/audit_open_source_readiness.py
tools/runtime_surface.py
config/runtime_surface_manifest.json
src/rag/project_config.py
.env.example

Everything else, including old extractor versions and one-off scripts, is historical or experimental until reviewed. The audit tool reports those files but does not make them blockers for the currently validated runtime path.

Use tools/audit_functional_surface_parity.py to compare source and target code/config surfaces without copying or modifying files. Use tools/audit_migration_requirements.py as the read-only gate for the current migration constraints: source project separation, local-only Git, tracked runtime files, required env keys, no tracked data artifacts, and no current-runtime open-source blockers.

Still Not Fully Clean

Historical scripts, archive files, revision logs, and old extractor versions still contain absolute paths. They were not globally rewritten in this phase because they are not the current runtime entrypoint and broad replacement would risk changing historical records incorrectly.

PC_DATA_ROOT and PLANTAI_SPLIT_DATA_ROOT are independent external inputs. During this code-cleanup phase, do not deduplicate, merge, clean, rename, copy, or rewrite either dataset. The validation wrapper may check that configured paths and required PC files are readable, but data governance remains a later separate phase.

Next cleanup phases should move old scripts into an archive policy, remove unused versions from the runnable surface, and regenerate CodeGraph in this repository after the structure stabilizes.

Default pytest collection is limited to the maintained audit, methodology, contract, and V16.1.2 baseline tests. Historical exploratory scripts remain on disk but are excluded from the default gate because many require private papers, retired absolute paths, or live services.

8.5 KiB Raw Permalink Blame History