PlantAI-rag/README.md

# PlantAI-rag

PlantAI-rag is a local working copy of the gene-family literature RAG and
knowledge extraction system. The current maintained extraction entrypoint is
`src.rag.extractors.v16_1_2.V16Extractor`.

This repository is being prepared for a future open-source cleanup. It is not
ready to publish as-is.

## Current Status

- Source project remains read-only and is configured by `PLANTAI_SOURCE_ROOT`.
- This working copy should be used as the project root for cleanup work.
- Large data, papers, and model assets are kept outside Git and referenced by
  `.env` or local symlinks.
- Current conda environment names are intentionally unchanged: `gfpy312` and
  `minerupy312`.
- Real API keys, passwords, and private local paths must stay in `.env`, never
  in tracked files.

## Validated Baseline

Final cleanup acceptance validation was run on 2026-06-25 with a real
Tartary buckwheat SPL paper:

- Input PDF: `${PLANTAI_SOURCE_ROOT}/uploads/64bbee67_Ft_SPL_2019_BMC.pdf`
- Title: `Genome-wide identification of the SPL gene family in Tartary Buckwheat`
- PMID: `31340871`
- DOI: `10.1186/s12870-019-1916-6`
- PlantAI-rag output: `/tmp/plantai_rag_realrun_ftspl`
- Source-project comparison output: `/tmp/source_gfmas_realrun_ftspl`

PlantAI-rag validation command:

```bash
env \
  MINERU_CMD="${MINERU_CMD:?set MINERU_CMD to the local MinerU executable}" \
  PLANTAI_SOURCE_ROOT=/path/to/source-gene-family-mas-v2 \
  "${GF_PYTHON:-python}" \
  tools/validate_v16_baseline.py \
  --pdf "${PLANTAI_SOURCE_ROOT}/uploads/64bbee67_Ft_SPL_2019_BMC.pdf" \
  --pmid 31340871 \
  --doi 10.1186/s12870-019-1916-6 \
  --output-dir /tmp/plantai_rag_realrun_ftspl \
  --cpu-mineru \
  --no-llm \
  --no-vlm \
  --no-pc-data
```

Observed PlantAI-rag result:

- Validation status: `VALIDATION_PASSED`
- V16.1.2 extracted 11 sections, 286 entities, 180 relations, and 24 relation units.
- Retained relations had 0 missing source, 0 missing target, and 0 missing evidence.
- Relation types: `has_property` 72, `localizes_to` 48, `has_gene_id` 31, `contains_domain` 29.
- Output layer: `main_kg` 180 relations.
- Output artifacts included `result.json`, `validation_summary.json`, MinerU block JSONL files,
  relation-unit coverage, repair audit, semantic-review audit, claim-scope audit,
  supplementary-download audit, and supplementary column-mapping audit.

Source-project comparison used the same PDF content and same V16.1.2 runner with LLM off.
The first direct source run wrote MinerU intermediates beside the source PDF and failed before
result generation because the source runner/parser did not honor `--output-dir` for MinerU
intermediates and then refused text fallback. The isolated source-project comparison was rerun
with the PDF copied to `/tmp/plantai_source_compare_input` and
`V16_ALLOW_MINERU_TEXT_FALLBACK=1`; it completed with 143 entities, 180 relations, 0 relation
units, and the same four relation-type counts as PlantAI-rag. This means the maintained
PlantAI-rag runner preserves the supplementary-table relation output and improves the main-text
relation-unit path under the tested MinerU CPU configuration.

The latest strict local baseline used the Pecan SPL paper `pecan_SPL_2021.pdf`
with the MinerU CPU pipeline, DeepSeek V4 relation extraction, required
PlantConnectome data, and the V16.1.2 extractor chain.

Observed result:

- V16.1.2 extracted 11 sections, 712 entities, 751 relations, and 33 chunks.
- LLM relation extraction produced 642 candidate relations and 635 relations
  before the evidence gate.
- All retained relations had non-empty source, target, and evidence fields.
- PC data loaded: 43,561 aliases, 17,618 gene aliases, 788 relationship
  patterns, and 3,636 entity-dictionary sample entries.
- VLM was not enabled in this run.
- The strict run had 43 LLM requests and 2 request-timeout failures; the
  extractor continued and passed final validation.
- GPU/API MinerU attempts failed because available GPU memory was exhausted;
  CPU pipeline is the verified fallback path.

A follow-up three-paper strict regression from
`data/input/v16_batch10_20260625` also passed with required PC data enabled:
`plantsci_2020` (393 entities, 322 relations), `aob_mcz153` (646 entities,
590 relations), and `bmc_s12870_2020` (631 entities, 619 relations). These
runs wrote outputs under `/tmp` and did not copy, clean, deduplicate, or modify
source data.
A strict PC-data validation was also rerun with the four PlantConnectome JSON
files supplied from a local historical source-project PC directory via
`--pc-data-root`. It passed with `PC_DATA_ROOT` available, 43,561 aliases,
17,618 gene aliases, 788 relationship patterns, 3,636 entity-dictionary sample
entries, 286 entities, 180 relations, and 24 chunks. The PC data remained
external and was not copied into this repository.


## Local Configuration

Create a private `.env` from the template:

```bash
cp .env.example .env
```

Fill only local values in `.env`, especially:

- `PC_DATA_ROOT`
- `PLANTAI_SPLIT_DATA_ROOT`
- `PLANTAI_SOURCE_ROOT`
- `BBAV5_ROOT`
- `PLANTAI_CONDA_ROOT`
- `PLANTAI_CONDA_PROFILE`
- `MINERU_CMD`
- `DEEPSEEK_API_KEY`
- `DASHSCOPE_API_KEY`
- `NEO4J_PASSWORD`

For low-memory MinerU validation, use:

```bash
V15_MINERU_BACKEND=pipeline
V15_MINERU_METHOD=txt
V15_MINERU_FORMULA=0
V15_MINERU_TABLE=0
CUDA_VISIBLE_DEVICES=
```

## Local Git Policy

The first Git baseline is local only:

- Do not add a remote.
- Do not push.
- Do not track `.env`, experiments, raw papers, large datasets, model weights,
  CodeGraph indexes, or historical one-off scripts.

Historical files remain on disk for reference, but they are excluded from the
first tracked baseline until reviewed.

## Baseline Validation

Use the maintained validation wrapper before and after cleanup changes:

```bash
${GF_PYTHON:-python} tools/validate_v16_baseline.py \
  --pdf "${PLANTAI_TEST_PDF:?set PLANTAI_TEST_PDF to a real local PDF}" \
  --pmid 34966577 \
  --output-dir experiments/phase2_v16_baseline_pecan \
  --cpu-mineru \
  --no-llm \
  --no-vlm
```

This command writes `result.json` and `validation_summary.json` under the
chosen output directory. It verifies that sections, entities, relations, and
chunks are non-empty and that every retained relation has source, target, and
evidence.
The summary also records optional supplementary table reader capabilities:
`pandas` for CSV/TSV, `openpyxl` for `.xlsx`, `xlrd` for `.xls`, and
`python-docx` for DOCX tables. Missing readers do not rewrite data, but they
explain why downloaded supplementary tables may be skipped.

For strict LLM relation validation, configure `DEEPSEEK_API_KEY` in private
`.env` and run with `--with-llm --no-vlm`. Add `--with-llm-definitions` only
when the validation target explicitly includes per-entity generated definitions.
The wrapper fails before launching MinerU when a required API key or required
external PC data root is missing.

External PC and split-data roots are treated as read-only inputs. The validation
wrapper can report both locations, but it does not copy, deduplicate, merge,
clean, or rewrite either dataset. Use `--pc-data-root` and `--split-data-root`
to override local `.env` values for one run. Add `--require-pc-data` only when
the four required PlantConnectome JSON files must be present and loaded.

External resources are cataloged in `config/external_resources_manifest.json`.
That manifest is intentionally limited to env key names, usage categories, and
read-only repository policy. It must not contain real local paths, API keys,
passwords, or host-specific defaults; put those values only in private `.env`.

## Supported Runtime Surface

The current supported cleanup surface is deliberately narrow. The canonical
classification is stored in `config/runtime_surface_manifest.json` and loaded by
`tools/runtime_surface.py`, so audit tools and migration checks use the same
runtime file list.

- `src.rag.extractors.v16_1_2.V16Extractor`
- `src.rag.extractors.v16_1_2.paper_metadata_enricher_v16_1_2`
- `src.rag.unified`
- `scripts/run_v16_1_2_one_paper.py`
- `scripts/run_v16_batch10_20260625.py`
- `scripts/backfill_v16_paper_metadata.py`
- `tools/validate_v16_baseline.py`
- `tools/audit_functional_surface_parity.py`
- `tools/audit_migration_requirements.py`
- `tools/audit_open_source_readiness.py`
- `tools/runtime_surface.py`
- `config/external_resources_manifest.json`
- `config/runtime_surface_manifest.json`
- `src/rag/project_config.py`
- `.env.example`

Old extractor versions, historical scripts, archived experiments, and report
files remain in the working tree as reference material. They are not treated as
public API until they are reviewed and explicitly promoted.

## Maintained API Entrypoint

Use the narrow maintained API app during cleanup:

```bash
${GF_PYTHON:-python} -m uvicorn api.app:app --host 127.0.0.1 --port 7900
```

This app exposes `/health` and the reviewed KG routes under `/api/v1`. The older
`api/main.py` imports many historical routers and remains on disk for parity, but
it is not part of the first supported open-source runtime surface.

## Open-Source Readiness Audit

Default `pytest` collection is intentionally limited to the maintained
contract, audit, methodology, and V16.1.2 baseline tests. Historical exploratory
scripts under `tests/` and `tests/archive/` are retained on disk, but many of
them require private papers, retired paths, or live services and are not part of
the default open-source gate.

Run the read-only migration and open-source audits before cleanup commits:

```bash
${GF_PYTHON:-python} tools/audit_functional_surface_parity.py
${GF_PYTHON:-python} tools/audit_migration_requirements.py
${GF_PYTHON:-python} tools/audit_open_source_readiness.py
```

The functional parity and migration audits require the original source checkout
to be configured through private `.env` as `PLANTAI_SOURCE_ROOT`, or passed for a
single run with `--source-root`. The repository no longer hard-codes a source
project path.

The functional parity audit compares source and target code/config surfaces. It
excludes data, papers, model assets, cache, and experiment outputs, and it only
reports missing or extra files; it never copies, deletes, or rewrites files.

The migration audit checks that the source project remains separate, Git stays
local-only, required runtime files are tracked, required env keys are present,
and raw data/papers/model weights/experiment outputs are not part of the tracked
baseline.

The audit scans tracked text files for personal absolute paths, credential-like
assignments, and dangerous credential defaults. It reports historical findings
separately from the current runtime surface and does not edit files or data.

## Open Cleanup Work

Cleanup scope is now frozen for this local baseline. Do not continue broad
legacy-script or archive cleanup during this phase. Future changes should be
limited to issues that affect the maintained runtime surface, make audits fail,
or make the real V16/MinerU validation fail.

Remaining pre-publication items are deliberately narrow:

- Keep `PC_DATA_ROOT` and `PLANTAI_SPLIT_DATA_ROOT` as two independent
  external inputs. The strict PC-data validation has passed with a local
  external PC root; do not deduplicate, merge, rewrite, or vendor these
  datasets during the code-cleanup phase.
- Re-run VLM validation only after local `DASHSCOPE_API_KEY` is configured.
- Before any public release, decide whether historical legacy scripts should be
  removed, archived outside the public repository, or explicitly promoted into
  the maintained runtime surface.