PlantAI-rag

PlantAI-rag is a local working copy of the gene-family literature RAG and knowledge extraction system. The current maintained extraction entrypoint is src.rag.extractors.v16_1_2.V16Extractor.

This repository is being prepared for a future open-source cleanup. It is not ready to publish as-is.

Current Status

• Source project remains read-only and is configured by PLANTAI_SOURCE_ROOT.
• This working copy should be used as the project root for cleanup work.
• Large data, papers, and model assets are kept outside Git and referenced by .env or local symlinks.
• Current conda environment names are intentionally unchanged: gfpy312 and minerupy312.
• Real API keys, passwords, and private local paths must stay in .env, never in tracked files.

Validated Baseline

Final cleanup acceptance validation was run on 2026-06-25 with a real Tartary buckwheat SPL paper:

• Input PDF: ${PLANTAI_SOURCE_ROOT}/uploads/64bbee67_Ft_SPL_2019_BMC.pdf
• Title: Genome-wide identification of the SPL gene family in Tartary Buckwheat
• PMID: 31340871
• DOI: 10.1186/s12870-019-1916-6
• PlantAI-rag output: /tmp/plantai_rag_realrun_ftspl
• Source-project comparison output: /tmp/source_gfmas_realrun_ftspl

PlantAI-rag validation command:

env \
  MINERU_CMD="${MINERU_CMD:?set MINERU_CMD to the local MinerU executable}" \
  PLANTAI_SOURCE_ROOT=/path/to/source-gene-family-mas-v2 \
  "${GF_PYTHON:-python}" \
  tools/validate_v16_baseline.py \
  --pdf "${PLANTAI_SOURCE_ROOT}/uploads/64bbee67_Ft_SPL_2019_BMC.pdf" \
  --pmid 31340871 \
  --doi 10.1186/s12870-019-1916-6 \
  --output-dir /tmp/plantai_rag_realrun_ftspl \
  --cpu-mineru \
  --no-llm \
  --no-vlm \
  --no-pc-data

Observed PlantAI-rag result:

• Validation status: VALIDATION_PASSED
• V16.1.2 extracted 11 sections, 286 entities, 180 relations, and 24 relation units.
• Retained relations had 0 missing source, 0 missing target, and 0 missing evidence.
• Relation types: has_property 72, localizes_to 48, has_gene_id 31, contains_domain 29.
• Output layer: main_kg 180 relations.
• Output artifacts included result.json, validation_summary.json, MinerU block JSONL files, relation-unit coverage, repair audit, semantic-review audit, claim-scope audit, supplementary-download audit, and supplementary column-mapping audit.

Source-project comparison used the same PDF content and same V16.1.2 runner with LLM off. The first direct source run wrote MinerU intermediates beside the source PDF and failed before result generation because the source runner/parser did not honor --output-dir for MinerU intermediates and then refused text fallback. The isolated source-project comparison was rerun with the PDF copied to /tmp/plantai_source_compare_input and V16_ALLOW_MINERU_TEXT_FALLBACK=1; it completed with 143 entities, 180 relations, 0 relation units, and the same four relation-type counts as PlantAI-rag. This means the maintained PlantAI-rag runner preserves the supplementary-table relation output and improves the main-text relation-unit path under the tested MinerU CPU configuration.

The latest strict local baseline used the Pecan SPL paper pecan_SPL_2021.pdf with the MinerU CPU pipeline, DeepSeek V4 relation extraction, required PlantConnectome data, and the V16.1.2 extractor chain.

Observed result:

• V16.1.2 extracted 11 sections, 712 entities, 751 relations, and 33 chunks.
• LLM relation extraction produced 642 candidate relations and 635 relations before the evidence gate.
• All retained relations had non-empty source, target, and evidence fields.
• PC data loaded: 43,561 aliases, 17,618 gene aliases, 788 relationship patterns, and 3,636 entity-dictionary sample entries.
• VLM was not enabled in this run.
• The strict run had 43 LLM requests and 2 request-timeout failures; the extractor continued and passed final validation.
• GPU/API MinerU attempts failed because available GPU memory was exhausted; CPU pipeline is the verified fallback path.

A follow-up three-paper strict regression from data/input/v16_batch10_20260625 also passed with required PC data enabled: plantsci_2020 (393 entities, 322 relations), aob_mcz153 (646 entities, 590 relations), and bmc_s12870_2020 (631 entities, 619 relations). These runs wrote outputs under /tmp and did not copy, clean, deduplicate, or modify source data. A strict PC-data validation was also rerun with the four PlantConnectome JSON files supplied from a local historical source-project PC directory via --pc-data-root. It passed with PC_DATA_ROOT available, 43,561 aliases, 17,618 gene aliases, 788 relationship patterns, 3,636 entity-dictionary sample entries, 286 entities, 180 relations, and 24 chunks. The PC data remained external and was not copied into this repository.

Local Configuration

Create a private .env from the template:

cp .env.example .env

Fill only local values in .env, especially:

• PC_DATA_ROOT
• PLANTAI_SPLIT_DATA_ROOT
• PLANTAI_SOURCE_ROOT
• BBAV5_ROOT
• PLANTAI_CONDA_ROOT
• PLANTAI_CONDA_PROFILE
• MINERU_CMD
• DEEPSEEK_API_KEY
• DASHSCOPE_API_KEY
• NEO4J_PASSWORD

For low-memory MinerU validation, use:

V15_MINERU_BACKEND=pipeline
V15_MINERU_METHOD=txt
V15_MINERU_FORMULA=0
V15_MINERU_TABLE=0
CUDA_VISIBLE_DEVICES=

Local Git Policy

The first Git baseline is local only:

• Do not add a remote.
• Do not push.
• Do not track .env, experiments, raw papers, large datasets, model weights, CodeGraph indexes, or historical one-off scripts.

Historical files remain on disk for reference, but they are excluded from the first tracked baseline until reviewed.

Baseline Validation

Use the maintained validation wrapper before and after cleanup changes:

${GF_PYTHON:-python} tools/validate_v16_baseline.py \
  --pdf "${PLANTAI_TEST_PDF:?set PLANTAI_TEST_PDF to a real local PDF}" \
  --pmid 34966577 \
  --output-dir experiments/phase2_v16_baseline_pecan \
  --cpu-mineru \
  --no-llm \
  --no-vlm

This command writes result.json and validation_summary.json under the chosen output directory. It verifies that sections, entities, relations, and chunks are non-empty and that every retained relation has source, target, and evidence. The summary also records optional supplementary table reader capabilities: pandas for CSV/TSV, openpyxl for .xlsx, xlrd for .xls, and python-docx for DOCX tables. Missing readers do not rewrite data, but they explain why downloaded supplementary tables may be skipped.

For strict LLM relation validation, configure DEEPSEEK_API_KEY in private .env and run with --with-llm --no-vlm. Add --with-llm-definitions only when the validation target explicitly includes per-entity generated definitions. The wrapper fails before launching MinerU when a required API key or required external PC data root is missing.

External PC and split-data roots are treated as read-only inputs. The validation wrapper can report both locations, but it does not copy, deduplicate, merge, clean, or rewrite either dataset. Use --pc-data-root and --split-data-root to override local .env values for one run. Add --require-pc-data only when the four required PlantConnectome JSON files must be present and loaded.

External resources are cataloged in config/external_resources_manifest.json. That manifest is intentionally limited to env key names, usage categories, and read-only repository policy. It must not contain real local paths, API keys, passwords, or host-specific defaults; put those values only in private .env.

Supported Runtime Surface

The current supported cleanup surface is deliberately narrow. The canonical classification is stored in config/runtime_surface_manifest.json and loaded by tools/runtime_surface.py, so audit tools and migration checks use the same runtime file list.

• src.rag.extractors.v16_1_2.V16Extractor
• src.rag.extractors.v16_1_2.paper_metadata_enricher_v16_1_2
• src.rag.unified
• scripts/run_v16_1_2_one_paper.py
• scripts/run_v16_batch10_20260625.py
• scripts/backfill_v16_paper_metadata.py
• tools/validate_v16_baseline.py
• tools/audit_functional_surface_parity.py
• tools/audit_migration_requirements.py
• tools/audit_open_source_readiness.py
• tools/runtime_surface.py
• config/external_resources_manifest.json
• config/runtime_surface_manifest.json
• src/rag/project_config.py
• .env.example

Old extractor versions, historical scripts, archived experiments, and report files remain in the working tree as reference material. They are not treated as public API until they are reviewed and explicitly promoted.

Maintained API Entrypoint

Use the narrow maintained API app during cleanup:

${GF_PYTHON:-python} -m uvicorn api.app:app --host 127.0.0.1 --port 7900

This app exposes /health and the reviewed KG routes under /api/v1. The older api/main.py imports many historical routers and remains on disk for parity, but it is not part of the first supported open-source runtime surface.

Open-Source Readiness Audit

Default pytest collection is intentionally limited to the maintained contract, audit, methodology, and V16.1.2 baseline tests. Historical exploratory scripts under tests/ and tests/archive/ are retained on disk, but many of them require private papers, retired paths, or live services and are not part of the default open-source gate.

Run the read-only migration and open-source audits before cleanup commits:

${GF_PYTHON:-python} tools/audit_functional_surface_parity.py
${GF_PYTHON:-python} tools/audit_migration_requirements.py
${GF_PYTHON:-python} tools/audit_open_source_readiness.py

The functional parity and migration audits require the original source checkout to be configured through private .env as PLANTAI_SOURCE_ROOT, or passed for a single run with --source-root. The repository no longer hard-codes a source project path.

The functional parity audit compares source and target code/config surfaces. It excludes data, papers, model assets, cache, and experiment outputs, and it only reports missing or extra files; it never copies, deletes, or rewrites files.

The migration audit checks that the source project remains separate, Git stays local-only, required runtime files are tracked, required env keys are present, and raw data/papers/model weights/experiment outputs are not part of the tracked baseline.

The audit scans tracked text files for personal absolute paths, credential-like assignments, and dangerous credential defaults. It reports historical findings separately from the current runtime surface and does not edit files or data.

Open Cleanup Work

Cleanup scope is now frozen for this local baseline. Do not continue broad legacy-script or archive cleanup during this phase. Future changes should be limited to issues that affect the maintained runtime surface, make audits fail, or make the real V16/MinerU validation fail.

Remaining pre-publication items are deliberately narrow:

• Keep PC_DATA_ROOT and PLANTAI_SPLIT_DATA_ROOT as two independent external inputs. The strict PC-data validation has passed with a local external PC root; do not deduplicate, merge, rewrite, or vendor these datasets during the code-cleanup phase.
• Re-run VLM validation only after local DASHSCOPE_API_KEY is configured.
• Before any public release, decide whether historical legacy scripts should be removed, archived outside the public repository, or explicitly promoted into the maintained runtime surface.