# PlantGeneBench

PlantGeneBench is an offline benchmark harness for bbav5 gene-family agent
planning. It evaluates whether an agent can map plant gene-family analysis
requests to valid stages, dependencies, parameters, and safety boundaries.

The frontend should be used for review and spot checks. The primary benchmark
path is batch execution:

```bash
python benchmarks/plantgenebench/scripts/run_benchmark.py \
  --provider rule \
  --tasks benchmarks/plantgenebench/tasks/plantgenebench_core_seed.jsonl \
  --gold benchmarks/plantgenebench/gold/plantgenebench_core_seed.gold.jsonl \
  --out benchmarks/plantgenebench/results/raw/rule_seed.jsonl

python benchmarks/plantgenebench/scripts/score.py \
  --predictions benchmarks/plantgenebench/results/raw/rule_seed.jsonl \
  --gold benchmarks/plantgenebench/gold/plantgenebench_core_seed.gold.jsonl \
  --out benchmarks/plantgenebench/results/scored/rule_seed.scored.jsonl

python benchmarks/plantgenebench/scripts/summarize.py \
  --scored benchmarks/plantgenebench/results/scored/rule_seed.scored.jsonl \
  --out-prefix benchmarks/plantgenebench/results/summary/rule_seed
```

To evaluate the live bbav5 planning endpoint:

```bash
python benchmarks/plantgenebench/scripts/run_benchmark.py \
  --provider api \
  --api-base http://127.0.0.1:7880 \
  --tasks benchmarks/plantgenebench/tasks/plantgenebench_core_seed.jsonl \
  --gold benchmarks/plantgenebench/gold/plantgenebench_core_seed.gold.jsonl \
  --out benchmarks/plantgenebench/results/raw/live_seed.jsonl
```

The API provider sends `intent.benchmarkMode=true`. Current bbav5 code uses
that explicit flag to skip file-level PreFlight validation and evaluate
planning only. Restart the web service after updating the code; an old process
will still reject empty benchmark jobs for missing input files.

## File Layout

- `tasks/*.jsonl`: benchmark inputs, one task per line.
- `gold/*.gold.jsonl`: structured gold answers keyed by `task_id`.
- `scripts/run_benchmark.py`: batch inference runner.
- `scripts/score.py`: deterministic scorer.
- `scripts/summarize.py`: CSV and Markdown summary generator.
- `prompts/llm_judge.md`: optional rubric for free-text fields only.

## Scoring Policy

Use deterministic scoring wherever possible:

- stage set F1
- forbidden-stage violations
- dependency checks
- required parameter matching
- analysis type matching

Use LLM-as-judge only for fields that cannot be reduced to exact structure,
such as biological rationale or literature evidence extraction. Those fields
should be reported separately from deterministic scores.

## Full 100-Task Evaluation

The full PlantGeneBench-100 workflow evaluates five layers:

- task understanding
- execution planning
- evidence constraint
- biological interpretation
- boundary control

Convert the curated Markdown question set into executable task and gold JSONL:

```bash
/home/xia-lab/miniconda3/envs/test4py311/bin/python \
  benchmarks/plantgenebench/scripts/convert_pgb100_full.py \
  --md 项目记录/2026-06-08_PlantGeneBench_100题正文题库_v1.md \
  --tasks-out benchmarks/plantgenebench/tasks/plantgenebench_100_v1.full.jsonl \
  --gold-out benchmarks/plantgenebench/gold/plantgenebench_100_v1.full.gold.jsonl
```

Run the publication-oriented providers. `deepseek_text` and `deepseek_json` reuse the DeepSeek API configuration from `~/gene-family-mas-v2/.env.v15_deepseek` without copying the key into this repository.

```bash
mkdir -p benchmarks/plantgenebench/results/full

for provider in rule full_system no_evidence deepseek_text deepseek_json; do
  /home/xia-lab/miniconda3/envs/test4py311/bin/python \
    benchmarks/plantgenebench/scripts/run_full_benchmark.py \
    --tasks benchmarks/plantgenebench/tasks/plantgenebench_100_v1.full.jsonl \
    --provider "$provider" \
    --api-base http://127.0.0.1:7880 \
    --deepseek-env /home/xia-lab/gene-family-mas-v2/.env.v15_deepseek \
    --out "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.predictions.jsonl" \
    --timeout 120
done
```

`deepseek_text` is a free-text LLM baseline and must be manually adjudicated. `deepseek_json` is the structured LLM baseline scored against the same planning fields as the system. The legacy `llm_only` provider remains available only as an offline surrogate for engineering smoke tests; do not use it for publication comparisons.

Score each provider and create a manual-review queue:

```bash
for provider in rule full_system no_evidence deepseek_text deepseek_json; do
  /home/xia-lab/miniconda3/envs/test4py311/bin/python \
    benchmarks/plantgenebench/scripts/score_full.py \
    --predictions "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.predictions.jsonl" \
    --gold benchmarks/plantgenebench/gold/plantgenebench_100_v1.full.gold.jsonl \
    --out "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.scored.jsonl" \
    --manual-review-out "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.manual_review.tsv"
done
```

Summarize the benchmark:

```bash
/home/xia-lab/miniconda3/envs/test4py311/bin/python \
  benchmarks/plantgenebench/scripts/summarize_full.py \
  --scored \
    benchmarks/plantgenebench/results/full/plantgenebench_100_v1.rule.scored.jsonl \
    benchmarks/plantgenebench/results/full/plantgenebench_100_v1.full_system.scored.jsonl \
    benchmarks/plantgenebench/results/full/plantgenebench_100_v1.no_evidence.scored.jsonl \
    benchmarks/plantgenebench/results/full/plantgenebench_100_v1.deepseek_text.scored.jsonl \
    benchmarks/plantgenebench/results/full/plantgenebench_100_v1.deepseek_json.scored.jsonl \
  --out-prefix benchmarks/plantgenebench/results/full/plantgenebench_100_v1
```

Main outputs:

- `results/full/*.predictions.jsonl`: raw provider responses normalized by task.
- `results/full/*.scored.jsonl`: machine-scored rows with component scores.
- `results/full/*.manual_review.tsv`: rows that need expert review before publication.
- `results/full/plantgenebench_100_v1.final_report.md`: aggregate provider, layer, backend-status, and error summaries.

Publication caveats:

- `deepseek_text` and `deepseek_json` are the publication-oriented real DeepSeek baselines. The legacy `llm_only` provider is only an offline surrogate and should not be included in publication tables.
- Machine scores are suitable for engineering triage. Evidence faithfulness, biological reasoning, DeepSeek free-text answers, and boundary-control cases must be manually adjudicated from `*.manual_review.tsv` before reporting publication-grade final numbers.
- The Markdown-to-JSONL conversion is deterministic but heuristic. If the question text changes, re-check the generated gold fields before using the benchmark in a manuscript.