144 lines
6.6 KiB
Markdown
Executable File
144 lines
6.6 KiB
Markdown
Executable File
# PlantGeneBench
|
|
|
|
PlantGeneBench is an offline benchmark harness for bbav5 gene-family agent
|
|
planning. It evaluates whether an agent can map plant gene-family analysis
|
|
requests to valid stages, dependencies, parameters, and safety boundaries.
|
|
|
|
The frontend should be used for review and spot checks. The primary benchmark
|
|
path is batch execution:
|
|
|
|
```bash
|
|
python benchmarks/plantgenebench/scripts/run_benchmark.py \
|
|
--provider rule \
|
|
--tasks benchmarks/plantgenebench/tasks/plantgenebench_core_seed.jsonl \
|
|
--gold benchmarks/plantgenebench/gold/plantgenebench_core_seed.gold.jsonl \
|
|
--out benchmarks/plantgenebench/results/raw/rule_seed.jsonl
|
|
|
|
python benchmarks/plantgenebench/scripts/score.py \
|
|
--predictions benchmarks/plantgenebench/results/raw/rule_seed.jsonl \
|
|
--gold benchmarks/plantgenebench/gold/plantgenebench_core_seed.gold.jsonl \
|
|
--out benchmarks/plantgenebench/results/scored/rule_seed.scored.jsonl
|
|
|
|
python benchmarks/plantgenebench/scripts/summarize.py \
|
|
--scored benchmarks/plantgenebench/results/scored/rule_seed.scored.jsonl \
|
|
--out-prefix benchmarks/plantgenebench/results/summary/rule_seed
|
|
```
|
|
|
|
To evaluate the live bbav5 planning endpoint:
|
|
|
|
```bash
|
|
python benchmarks/plantgenebench/scripts/run_benchmark.py \
|
|
--provider api \
|
|
--api-base http://127.0.0.1:7880 \
|
|
--tasks benchmarks/plantgenebench/tasks/plantgenebench_core_seed.jsonl \
|
|
--gold benchmarks/plantgenebench/gold/plantgenebench_core_seed.gold.jsonl \
|
|
--out benchmarks/plantgenebench/results/raw/live_seed.jsonl
|
|
```
|
|
|
|
The API provider sends `intent.benchmarkMode=true`. Current bbav5 code uses
|
|
that explicit flag to skip file-level PreFlight validation and evaluate
|
|
planning only. Restart the web service after updating the code; an old process
|
|
will still reject empty benchmark jobs for missing input files.
|
|
|
|
## File Layout
|
|
|
|
- `tasks/*.jsonl`: benchmark inputs, one task per line.
|
|
- `gold/*.gold.jsonl`: structured gold answers keyed by `task_id`.
|
|
- `scripts/run_benchmark.py`: batch inference runner.
|
|
- `scripts/score.py`: deterministic scorer.
|
|
- `scripts/summarize.py`: CSV and Markdown summary generator.
|
|
- `prompts/llm_judge.md`: optional rubric for free-text fields only.
|
|
|
|
## Scoring Policy
|
|
|
|
Use deterministic scoring wherever possible:
|
|
|
|
- stage set F1
|
|
- forbidden-stage violations
|
|
- dependency checks
|
|
- required parameter matching
|
|
- analysis type matching
|
|
|
|
Use LLM-as-judge only for fields that cannot be reduced to exact structure,
|
|
such as biological rationale or literature evidence extraction. Those fields
|
|
should be reported separately from deterministic scores.
|
|
|
|
## Full 100-Task Evaluation
|
|
|
|
The full PlantGeneBench-100 workflow evaluates five layers:
|
|
|
|
- task understanding
|
|
- execution planning
|
|
- evidence constraint
|
|
- biological interpretation
|
|
- boundary control
|
|
|
|
Convert the curated Markdown question set into executable task and gold JSONL:
|
|
|
|
```bash
|
|
/home/xia-lab/miniconda3/envs/test4py311/bin/python \
|
|
benchmarks/plantgenebench/scripts/convert_pgb100_full.py \
|
|
--md 项目记录/2026-06-08_PlantGeneBench_100题正文题库_v1.md \
|
|
--tasks-out benchmarks/plantgenebench/tasks/plantgenebench_100_v1.full.jsonl \
|
|
--gold-out benchmarks/plantgenebench/gold/plantgenebench_100_v1.full.gold.jsonl
|
|
```
|
|
|
|
Run the publication-oriented providers. `deepseek_text` and `deepseek_json` reuse the DeepSeek API configuration from `~/gene-family-mas-v2/.env.v15_deepseek` without copying the key into this repository.
|
|
|
|
```bash
|
|
mkdir -p benchmarks/plantgenebench/results/full
|
|
|
|
for provider in rule full_system no_evidence deepseek_text deepseek_json; do
|
|
/home/xia-lab/miniconda3/envs/test4py311/bin/python \
|
|
benchmarks/plantgenebench/scripts/run_full_benchmark.py \
|
|
--tasks benchmarks/plantgenebench/tasks/plantgenebench_100_v1.full.jsonl \
|
|
--provider "$provider" \
|
|
--api-base http://127.0.0.1:7880 \
|
|
--deepseek-env /home/xia-lab/gene-family-mas-v2/.env.v15_deepseek \
|
|
--out "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.predictions.jsonl" \
|
|
--timeout 120
|
|
done
|
|
```
|
|
|
|
`deepseek_text` is a free-text LLM baseline and must be manually adjudicated. `deepseek_json` is the structured LLM baseline scored against the same planning fields as the system. The legacy `llm_only` provider remains available only as an offline surrogate for engineering smoke tests; do not use it for publication comparisons.
|
|
|
|
Score each provider and create a manual-review queue:
|
|
|
|
```bash
|
|
for provider in rule full_system no_evidence deepseek_text deepseek_json; do
|
|
/home/xia-lab/miniconda3/envs/test4py311/bin/python \
|
|
benchmarks/plantgenebench/scripts/score_full.py \
|
|
--predictions "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.predictions.jsonl" \
|
|
--gold benchmarks/plantgenebench/gold/plantgenebench_100_v1.full.gold.jsonl \
|
|
--out "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.scored.jsonl" \
|
|
--manual-review-out "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.manual_review.tsv"
|
|
done
|
|
```
|
|
|
|
Summarize the benchmark:
|
|
|
|
```bash
|
|
/home/xia-lab/miniconda3/envs/test4py311/bin/python \
|
|
benchmarks/plantgenebench/scripts/summarize_full.py \
|
|
--scored \
|
|
benchmarks/plantgenebench/results/full/plantgenebench_100_v1.rule.scored.jsonl \
|
|
benchmarks/plantgenebench/results/full/plantgenebench_100_v1.full_system.scored.jsonl \
|
|
benchmarks/plantgenebench/results/full/plantgenebench_100_v1.no_evidence.scored.jsonl \
|
|
benchmarks/plantgenebench/results/full/plantgenebench_100_v1.deepseek_text.scored.jsonl \
|
|
benchmarks/plantgenebench/results/full/plantgenebench_100_v1.deepseek_json.scored.jsonl \
|
|
--out-prefix benchmarks/plantgenebench/results/full/plantgenebench_100_v1
|
|
```
|
|
|
|
Main outputs:
|
|
|
|
- `results/full/*.predictions.jsonl`: raw provider responses normalized by task.
|
|
- `results/full/*.scored.jsonl`: machine-scored rows with component scores.
|
|
- `results/full/*.manual_review.tsv`: rows that need expert review before publication.
|
|
- `results/full/plantgenebench_100_v1.final_report.md`: aggregate provider, layer, backend-status, and error summaries.
|
|
|
|
Publication caveats:
|
|
|
|
- `deepseek_text` and `deepseek_json` are the publication-oriented real DeepSeek baselines. The legacy `llm_only` provider is only an offline surrogate and should not be included in publication tables.
|
|
- Machine scores are suitable for engineering triage. Evidence faithfulness, biological reasoning, DeepSeek free-text answers, and boundary-control cases must be manually adjudicated from `*.manual_review.tsv` before reporting publication-grade final numbers.
|
|
- The Markdown-to-JSONL conversion is deterministic but heuristic. If the question text changes, re-check the generated gold fields before using the benchmark in a manuscript.
|