admin/PlantAI-dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
master
PlantAI-dev/benchmarks/plantgenebench
T
History

README.md

PlantGeneBench

PlantGeneBench is an offline benchmark harness for bbav5 gene-family agent planning. It evaluates whether an agent can map plant gene-family analysis requests to valid stages, dependencies, parameters, and safety boundaries.

The frontend should be used for review and spot checks. The primary benchmark path is batch execution:

python benchmarks/plantgenebench/scripts/run_benchmark.py \
  --provider rule \
  --tasks benchmarks/plantgenebench/tasks/plantgenebench_core_seed.jsonl \
  --gold benchmarks/plantgenebench/gold/plantgenebench_core_seed.gold.jsonl \
  --out benchmarks/plantgenebench/results/raw/rule_seed.jsonl

python benchmarks/plantgenebench/scripts/score.py \
  --predictions benchmarks/plantgenebench/results/raw/rule_seed.jsonl \
  --gold benchmarks/plantgenebench/gold/plantgenebench_core_seed.gold.jsonl \
  --out benchmarks/plantgenebench/results/scored/rule_seed.scored.jsonl

python benchmarks/plantgenebench/scripts/summarize.py \
  --scored benchmarks/plantgenebench/results/scored/rule_seed.scored.jsonl \
  --out-prefix benchmarks/plantgenebench/results/summary/rule_seed

To evaluate the live bbav5 planning endpoint:

python benchmarks/plantgenebench/scripts/run_benchmark.py \
  --provider api \
  --api-base http://127.0.0.1:7880 \
  --tasks benchmarks/plantgenebench/tasks/plantgenebench_core_seed.jsonl \
  --gold benchmarks/plantgenebench/gold/plantgenebench_core_seed.gold.jsonl \
  --out benchmarks/plantgenebench/results/raw/live_seed.jsonl

The API provider sends intent.benchmarkMode=true. Current bbav5 code uses that explicit flag to skip file-level PreFlight validation and evaluate planning only. Restart the web service after updating the code; an old process will still reject empty benchmark jobs for missing input files.

File Layout

  • tasks/*.jsonl: benchmark inputs, one task per line.
  • gold/*.gold.jsonl: structured gold answers keyed by task_id.
  • scripts/run_benchmark.py: batch inference runner.
  • scripts/score.py: deterministic scorer.
  • scripts/summarize.py: CSV and Markdown summary generator.
  • prompts/llm_judge.md: optional rubric for free-text fields only.

Scoring Policy

Use deterministic scoring wherever possible:

  • stage set F1
  • forbidden-stage violations
  • dependency checks
  • required parameter matching
  • analysis type matching

Use LLM-as-judge only for fields that cannot be reduced to exact structure, such as biological rationale or literature evidence extraction. Those fields should be reported separately from deterministic scores.

Full 100-Task Evaluation

The full PlantGeneBench-100 workflow evaluates five layers:

  • task understanding
  • execution planning
  • evidence constraint
  • biological interpretation
  • boundary control

Convert the curated Markdown question set into executable task and gold JSONL:

/home/xia-lab/miniconda3/envs/test4py311/bin/python \
  benchmarks/plantgenebench/scripts/convert_pgb100_full.py \
  --md 项目记录/2026-06-08_PlantGeneBench_100题正文题库_v1.md \
  --tasks-out benchmarks/plantgenebench/tasks/plantgenebench_100_v1.full.jsonl \
  --gold-out benchmarks/plantgenebench/gold/plantgenebench_100_v1.full.gold.jsonl

Run the publication-oriented providers. deepseek_text and deepseek_json reuse the DeepSeek API configuration from ~/gene-family-mas-v2/.env.v15_deepseek without copying the key into this repository.

mkdir -p benchmarks/plantgenebench/results/full

for provider in rule full_system no_evidence deepseek_text deepseek_json; do
  /home/xia-lab/miniconda3/envs/test4py311/bin/python \
    benchmarks/plantgenebench/scripts/run_full_benchmark.py \
    --tasks benchmarks/plantgenebench/tasks/plantgenebench_100_v1.full.jsonl \
    --provider "$provider" \
    --api-base http://127.0.0.1:7880 \
    --deepseek-env /home/xia-lab/gene-family-mas-v2/.env.v15_deepseek \
    --out "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.predictions.jsonl" \
    --timeout 120
done

deepseek_text is a free-text LLM baseline and must be manually adjudicated. deepseek_json is the structured LLM baseline scored against the same planning fields as the system. The legacy llm_only provider remains available only as an offline surrogate for engineering smoke tests; do not use it for publication comparisons.

Score each provider and create a manual-review queue:

for provider in rule full_system no_evidence deepseek_text deepseek_json; do
  /home/xia-lab/miniconda3/envs/test4py311/bin/python \
    benchmarks/plantgenebench/scripts/score_full.py \
    --predictions "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.predictions.jsonl" \
    --gold benchmarks/plantgenebench/gold/plantgenebench_100_v1.full.gold.jsonl \
    --out "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.scored.jsonl" \
    --manual-review-out "benchmarks/plantgenebench/results/full/plantgenebench_100_v1.${provider}.manual_review.tsv"
done

Summarize the benchmark:

/home/xia-lab/miniconda3/envs/test4py311/bin/python \
  benchmarks/plantgenebench/scripts/summarize_full.py \
  --scored \
    benchmarks/plantgenebench/results/full/plantgenebench_100_v1.rule.scored.jsonl \
    benchmarks/plantgenebench/results/full/plantgenebench_100_v1.full_system.scored.jsonl \
    benchmarks/plantgenebench/results/full/plantgenebench_100_v1.no_evidence.scored.jsonl \
    benchmarks/plantgenebench/results/full/plantgenebench_100_v1.deepseek_text.scored.jsonl \
    benchmarks/plantgenebench/results/full/plantgenebench_100_v1.deepseek_json.scored.jsonl \
  --out-prefix benchmarks/plantgenebench/results/full/plantgenebench_100_v1

Main outputs:

  • results/full/*.predictions.jsonl: raw provider responses normalized by task.
  • results/full/*.scored.jsonl: machine-scored rows with component scores.
  • results/full/*.manual_review.tsv: rows that need expert review before publication.
  • results/full/plantgenebench_100_v1.final_report.md: aggregate provider, layer, backend-status, and error summaries.

Publication caveats:

  • deepseek_text and deepseek_json are the publication-oriented real DeepSeek baselines. The legacy llm_only provider is only an offline surrogate and should not be included in publication tables.
  • Machine scores are suitable for engineering triage. Evidence faithfulness, biological reasoning, DeepSeek free-text answers, and boundary-control cases must be manually adjudicated from *.manual_review.tsv before reporting publication-grade final numbers.
  • The Markdown-to-JSONL conversion is deterministic but heuristic. If the question text changes, re-check the generated gold fields before using the benchmark in a manuscript.
Reference in New Issue View Git Blame Copy Permalink