PlantAI-rag/PLANTAI_RAG_MIGRATION.md

# PlantAI-rag Migration Notes

This repository is a local working copy derived from the original
`gene-family-mas-v2` project.

## Current Rules

- Do not edit the source project configured by `PLANTAI_SOURCE_ROOT`; it remains
  read-only.
- Keep current conda environment names for now.
- Keep large data/model assets out of the copied repository. Use symlinks or environment variables.
- Keep real API keys, passwords, and private local settings out of tracked files.
- Keep Git local-only during this cleanup phase: do not add a remote and do not push.
- Treat `src/rag/extractors/v16_1_2` as the latest extractor entrypoint. It still inherits and calls V15-era implementation files, so edits must include that call chain.

## Local Paths

- Project root: this repository.
- Source project: the original private checkout; do not edit it from this copy.
- Large data, papers, PlantConnectome resources, bbav5 workspaces, and model
  assets are external resources configured through `.env` or local symlinks.

## Configuration

Use `.env` locally and `.env.example` as the shareable template. External paths and secrets should enter through env variables, not hard-coded source paths.

Important variables:

- `PLANTAI_CONDA_ROOT`
- `PLANTAI_CONDA_PROFILE`
- `PC_DATA_ROOT`
- `PLANTAI_SPLIT_DATA_ROOT`
- `PLANTAI_SOURCE_ROOT`
- `RAG_OUTPUT_DIR`
- `RAG_LEARNING_DIR`
- `BGE_RERANKER_ROOT`
- `BBAV5_ROOT`
- `MINERU_CMD`
- `DEEPSEEK_API_KEY`
- `DASHSCOPE_API_KEY`
- `NEO4J_PASSWORD`
- `GF_SECURITY_ADMIN_PASSWORD`

## Validated Local Baseline

The current local baseline was revalidated on 2026-06-25 using one real Pecan
SPL paper, the MinerU CPU pipeline, DeepSeek V4 (`deepseek-v4-flash`), and the
V16.1.2 extractor chain with required PlantConnectome data enabled. Secrets
remain only in private `.env`; no API key is recorded in tracked files.

Strict validation command shape:

```bash
${GF_PYTHON:-python} tools/validate_v16_baseline.py \
  --pdf "${PLANTAI_SOURCE_ROOT:?set PLANTAI_SOURCE_ROOT}/output/8ec26cde-ec16-438b-ac2f-5375aae06508/uploads/pecan_SPL_2021.pdf" \
  --pmid 34966577 \
  --output-dir /tmp/plantai_v16_1_2_llm_pc_strict_after_key \
  --cpu-mineru \
  --with-llm \
  --no-vlm \
  --require-pc-data
```

Validated result:

- Status: `VALIDATION_PASSED`
- Output: `/tmp/plantai_v16_1_2_llm_pc_strict_after_key`
- Result JSON: `/tmp/plantai_v16_1_2_llm_pc_strict_after_key/result.json`
- Sections: 11
- Chunks: 33
- Entities: 712
- Relations: 751
- LLM relations before evidence gate: 635
- LLM candidate relations generated: 642
- Rejected LLM candidates: 7
- Retained relation fields: 0 missing source, 0 missing target, 0 missing evidence
- PC data loaded: 43,561 aliases, 17,618 gene aliases, 788 relationship
  patterns, 3,636 entity-dictionary sample entries

Runtime note: the strict run had 43 LLM requests and 2 request-timeout failures.
The extractor continued, generated candidates from the remaining windows, and
passed final validation. This is acceptable for the current baseline but should
remain visible in `validation_summary.json` so timeout frequency can be tracked
across future multi-paper runs.

## Multi-Paper Strict Regression

On 2026-06-25, the V16.1.2 validation wrapper was run against three real PDFs
from the source project's `data/input/v16_batch10_20260625` collection. The
runs used MinerU CPU mode, DeepSeek V4 relation extraction, required PC data,
and wrote all outputs under `/tmp/plantai_v16_multi_paper_regression_20260625`.
No source data or split-data files were copied, cleaned, deduplicated, or
modified.

During this regression, `--with-llm` was narrowed to mean LLM relation
extraction only. Per-entity LLM definition generation is now opt-in through
`--with-llm-definitions`, because the previous coupled behavior could add a
large, low-observability API tail after relation extraction had already
completed. This preserves the definition feature while keeping baseline
regression focused on MinerU, PC loading, relation extraction, evidence gates,
and final result integrity.

| Paper output | Status | Sections | Chunks | Entities | Relations | Before gate | LLM requests | Timeouts | Candidate rows | Rejected rows |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| `plantsci_2020` | passed | 10 | 24 | 393 | 322 | 357 | 29 | 0 | 361 | 4 |
| `aob_mcz153` | passed | 8 | 28 | 646 | 590 | 601 | 49 | 0 | 752 | 151 |
| `bmc_s12870_2020` | passed | 11 | 24 | 631 | 619 | 659 | 33 | 0 | 728 | 69 |

All three strict summaries reported PC data available and 0 missing source,
0 missing target, and 0 missing evidence. The `bmc_s12870_2020` run was executed
after the definition-generation split and confirmed the intended runtime log:
`LLM定义: 禁用` with `LLM关系抽取: 启用`.

GPU/API MinerU attempts previously failed due exhausted GPU memory, so CPU
pipeline remains the verified fallback. VLM validation is still pending and
requires a private `DASHSCOPE_API_KEY`.

`PC_DATA_ROOT` is configured through private `.env` as `data/pc`. In this local
copy, `data` is a symlink to the source project's data directory, so PC resources
are read in place without copying, cleaning, deduplicating, or modifying data.

## Phase 2 Validation Entry

The maintained regression entry is:

```bash
${GF_PYTHON:-python} tools/validate_v16_baseline.py \
  --pdf "${PLANTAI_TEST_PDF:?set PLANTAI_TEST_PDF to a real local PDF}" \
  --pmid 34966577 \
  --output-dir experiments/phase2_v16_baseline_pecan \
  --cpu-mineru \
  --no-llm \
  --no-vlm
```

This is a real-paper validation, not a demo. It records
`validation_summary.json` next to `result.json`.

For strict LLM relation validation, configure `DEEPSEEK_API_KEY` in private
`.env` and use `--with-llm --no-vlm`. Add `--with-llm-definitions` only when
the validation target explicitly includes per-entity generated definitions. Add
`--require-pc-data` only after `PC_DATA_ROOT` points to a readable external
directory containing the required PC JSON files.

## Current Public Surface

The only supported runtime surface during this cleanup phase is the V16 extractor
chain plus the validation/configuration helpers. The source of truth is
`config/runtime_surface_manifest.json`; `tools/runtime_surface.py`,
`tools/audit_open_source_readiness.py`, and
`tools/audit_migration_requirements.py` all read that manifest instead of
maintaining separate runtime file lists.

- `src.rag.extractors.v16_1_2.V16Extractor`
- `src.rag.extractors.v16_1_2.paper_metadata_enricher_v16_1_2`
- `src.rag.unified`
- `scripts/run_v16_1_2_one_paper.py`
- `scripts/run_v16_batch10_20260625.py`
- `scripts/backfill_v16_paper_metadata.py`
- `tools/validate_v16_baseline.py`
- `tools/audit_functional_surface_parity.py`
- `tools/audit_migration_requirements.py`
- `tools/audit_open_source_readiness.py`
- `tools/runtime_surface.py`
- `config/runtime_surface_manifest.json`
- `src/rag/project_config.py`
- `.env.example`

Everything else, including old extractor versions and one-off scripts, is
historical or experimental until reviewed. The audit tool reports those files but
does not make them blockers for the currently validated runtime path.

Use `tools/audit_functional_surface_parity.py` to compare source and target
code/config surfaces without copying or modifying files. Use
`tools/audit_migration_requirements.py` as the read-only gate for the current
migration constraints: source project separation, local-only Git, tracked
runtime files, required env keys, no tracked data artifacts, and no
current-runtime open-source blockers.

## Still Not Fully Clean

Historical scripts, archive files, revision logs, and old extractor versions still contain absolute paths. They were not globally rewritten in this phase because they are not the current runtime entrypoint and broad replacement would risk changing historical records incorrectly.

`PC_DATA_ROOT` and `PLANTAI_SPLIT_DATA_ROOT` are independent external inputs.
During this code-cleanup phase, do not deduplicate, merge, clean, rename, copy,
or rewrite either dataset. The validation wrapper may check that configured
paths and required PC files are readable, but data governance remains a later
separate phase.

Next cleanup phases should move old scripts into an archive policy, remove unused versions from the runnable surface, and regenerate CodeGraph in this repository after the structure stabilizes.

Default `pytest` collection is limited to the maintained audit, methodology,
contract, and V16.1.2 baseline tests. Historical exploratory scripts remain on
disk but are excluded from the default gate because many require private papers,
retired absolute paths, or live services.