PlantAI-dev/docs/gene_family_scope_execution_audit_20260521.md

Gene Family Scope Execution Audit - 2026-05-21

Purpose

This document records the current execution framework, code links, scenario matrix, and verification state for the gene-family route using the real SPL job:

• Job: job_1779270590216_0038e5
• Family: SPL
• Prefix: Target.SPL
• Target species: Dl, Lc
• Reference species: At
• Comparative species: Cp, Os, PPY, Sl

The goal is to prevent stage-by-stage fixes from becoming disconnected from planning, execution, status, diagnosis, frontend polling, and reproducibility.

Core Code Chain

Layer	Files	Responsibility	Notes
Natural language / intent	apps/web/vue3/main.py, gfa_core/mas/agents/planning_v2.py	Map user request to stage list and params	Must preserve geneFamily, scopeMode, targetSpecies, referenceSpecies, comparativeSpecies, analysisRoute
Stage context	gfa_core/gene_family_stage_context.py	Build scope_id, output_prefix, member_set_id, stage_context	Shared identity used by planning, execution, status, manifests
Scope file	gfa_core/gene_family_scope.py, apps/web/vue3/main.py	Materialize current species scope as JSON	Passed to scoped CLI stages via --scope-file
Execution API	apps/web/vue3/main.py:/api/execute	Prepare scope, check completed status, submit Celery task	Must call unified stage status with scope/route/member set
Worker execution	tasks.py, stage backend code	Route to CLI, allocate queue/CPU, attach QA	Uses stage_catalog.py resource profiles
CLI stages	apps/cli/run_*.py	Actual analysis and manifest writing	Must write scoped manifest using output_prefix
Status truth	gfa_core/mas/tools/stage_status.py	Decide whether a stage is complete	Final manifest + artifacts are source of truth
Diagnosis	gfa_core/mas/agents/diagnosis.py, /api/agent/diagnose_job_stage	Read-only stage audit and safe repair advice	Must not confuse planning/artifact issues with input-cleaning sidecars
Frontend polling	apps/web/vue3/index.html	Poll async tasks and verify completion	Must pass scope/route/member set and trust backend stage_status.completed
Result display	apps/web/vue3/main.py:/api/results, frontend result cards	Group artifacts by stage/result type	Needs route-aware names for predicted/confirmed subfamily outputs

Scenario Matrix

Scenario	Required route	Scope mode	Expected selected species	Main stages	Output identity
Identify target members	multi_target or single_species	target_only	target species only	identify	Target.SPL.{scope_id}.identify.manifest.json, plus Target.{species}.SPL.final_members.*
Predict subfamily	subfamily_prediction	target_plus_reference	target + reference	phylogeny -> reference_classification	Target.SPL.{scope_id}.phylogeny.*, Target.SPL.{scope_id}.reference_classification.*
Confirm subfamily	subfamily_confirmed	same member scope	target + reference or selected set	subfamily_confirm, then optional redraw/reuse	confirmed_subfamily.tsv should become preferred downstream label source
Single species tree	single_species	target_only	one target species	phylogeny	species-scoped output prefix; should not be skipped by unrelated multi-species tree
Multi-target tree	multi_target	target_only	selected targets	phylogeny	target-only multi-species tree without reference
Tree + reference	subfamily_prediction or confirmed route	target_plus_reference	targets + reference	phylogeny	used by reference classification and subfamily tree plots
Motif/domain/structure for prediction review	subfamily_prediction	target_plus_reference	targets + reference	motif, domain, structure	Should use same member set and predicted subfamily labels
Motif/domain/structure for confirmed subfamily	confirmed_subfamily	chosen scope	selected members	motif, domain, structure, combo	Should prefer confirmed labels and avoid recomputing unchanged intermediate files when reusable
Promoter/synteny/karyotype/ppi	route-specific	target or pair scope	depends on stage	downstream stages	Must not assume unscoped {prefix} outputs when scoped outputs exist
Summary	route-aware summary	optional scope file	current scope or all completed	summary	Indexes manifests/artifacts and feeds deep interpretation

Verified On Real Data

Stage	Route	API/task result	Key evidence
identify	target_only:multi_target	success; re-run request now skips as completed	Dl 16, Lc 17 final members; merged 33 after Pfam domain filter
phylogeny	target_plus_reference:subfamily_prediction	success	76 sequences: Dl 22, Lc 37, At 17; tree and MSA generated
reference_classification	target_plus_reference:subfamily_prediction	success	59 target genes classified, 17 At reference tips, labeled tree generated
motif	target_plus_reference:subfamily_prediction	success	API task a481873b-f6a7-4091-bfab-5bd39c141236; scoped FASTA 76 records; scoped subfamily labels; 10 motifs; PNG/PDF/manifest generated
domain	target_plus_reference:subfamily_prediction	success after executor fix	API task a1bd1b6b-a732-4d68-a093-61d1779a33f7; scoped FASTA 76 records; 107 total domain hits; 67 proteins with domains; PNG/PDF/manifest generated
structure	target_plus_reference:subfamily_prediction	success with warning	API task e4dfdbeb-1f38-4ea3-b841-59c18b2d6f46; Dl and Lc scope GFFs used; 59/76 genes with structure; PNG/PDF/manifest generated; postflight reports low ID/GFF overlap warning
combo	target_plus_reference:subfamily_prediction	success after scoped-output/status fix	API task 21dcbc90-9020-4923-b566-7afce12276e2; old unscoped artifacts were first rejected as invalid_outputs; new scoped PDF/PNG and manifest generated with scoped reference-classification subfamily labels
motif/domain/structure/combo rerun	target_plus_reference:subfamily_prediction	success on 2026-05-22	API tasks d893bd75-7991-4862-bbd0-89f8a80e8da7, af84680d-6ccb-438a-8912-d60821c13a62, 27b1f847-015d-41a5-83e1-954f48e7ecd9, a2ea37f2-45a9-4a58-b95f-083f81869642; used real job data and regenerated subfamily evidence combo
identify domain review API	target_only/multi_target	success on 2026-05-22	/api/identify/domain_review?job_id=job_1779270590216_0038e5&prefix=Target.SPL returns 130 rows from merged/Dl/Lc review files; species groups: merged 65, Dl 25, Lc 40
subfamily review API	target_plus_reference:subfamily_prediction	success on 2026-05-22	/api/subfamily/review returns 76 rows, one rect combo evidence image, one circular subfamily tree, no missing evidence
results API	route-aware display	success on 2026-05-22	/api/results puts 77 MSA page images under phylogeny_msa, puts rect combo + circular subfamily tree under reference_classification, keeps combo panel to manifests only
agent plan API	natural language route	success on 2026-05-22	/api/agent/plan for "帮我执行亚家族分类分析" now resolves targetSpecies=[Dl,Lc], referenceSpecies=[At], scopeMode=target_plus_reference, and completed_stages=[phylogeny, reference_classification]

Fixes Already Applied

File	Change	Reason
gfa_core/mas/tools/stage_status.py	Missing manifest member_set_id is compatible when scope matches; explicit mismatch still fails	Older successful manifests lacked top-level member_set_id
gfa_core/mas/tools/stage_status.py	member_set_id scope part is compared order-insensitively	target-Dl-Lc and target-Lc-Dl are the same biological scope
gfa_core/mas/agents/diagnosis.py	multi_target_single_species_stage no longer triggers from successful identify manifest text	Artifact descriptions contain "Single target species..." legitimately
apps/web/vue3/index.html	Async polling passes stage params to status check and trusts backend stage status	Prevent frontend false failure after successful scoped stage
gfa_core/mas/stage_capabilities.py	reference_subfamily is marked as reference-role input	Lets resolver prefer reference labels over target domain-review tables
gfa_core/mas/artifact_resolver.py	reference_subfamily excludes domain_review, InterProScan, BLAST/HMM ID tables and prefers true subfamily files	Prevents planner/frontend from displaying wrong label candidates
gfa_core/mas/context_index.py	Manifest parameters are indexed; .id files are recognized as ID lists; InterProScan/domain-review TSVs remain tables	Enables scoped FASTA discovery and avoids treating domain TSV as member ID list
gfa_core/mas/artifact_resolver.py	family_members_pep prefers phylogeny parameters.scope_input_fasta when available	Aligns semantic preflight with actual scoped execution for motif/domain
apps/cli/run_motif.py, apps/cli/run_domain.py, apps/cli/run_structure.py	Scoped downstream stages search {prefix}.{scope}.reference_classification.subfamily.tsv / confirmed labels before unscoped labels	Ensures prediction/confirmation review plots use the correct route-specific labels
apps/web/pipeline_backend.py	domain dependency check now respects scoped phylogeny manifests; missing dependency returns failure without old manifest fallback	Prevents scoped domain from being blocked by absent unscoped {prefix}.phylogeny.manifest.json
gfa_core/mas/tools/stage_tools.py	Stage success now requires current-stage manifest or explicit success; failure text and manifest-stage mismatch cannot be reported as success	Fixes domain being returned as successful with an old motif manifest
gfa_core/mas/artifact_resolver.py	target_gff candidates are constrained to target species when target scope exists; family_members_id rejects InterProScan/domain TSVs	Prevents structure preflight from showing comparative GFF or domain TSV as selected inputs
apps/cli/run_combo.py	Combo now derives output_prefix from scope_id, writes scoped plot filenames, and searches scoped reference-classification/confirmed subfamily labels before unscoped files	Prevents route-specific combo outputs from overwriting/reusing generic {prefix}.combo... files
apps/web/pipeline_backend.py	Combo backend passes scope_id into resolve_subfamily_for_plotting	Ensures the CLI receives the current route's subfamily table
gfa_core/mas/tools/stage_status.py	Scoped combo completion requires combo artifacts whose filenames contain the requested scope token	Prevents old unscoped combo plots from being treated as completed scoped results
gfa_core/mas/agents/diagnosis.py	Combo was removed from GFF/member overlap postflight checks	Combo consumes upstream structure artifacts and should not be blocked by a fresh GFF overlap check against the wrong species file
apps/web/vue3/index.html	Added unified handleStageCompleted() and call it from batch/natural-language execution, direct execution, and single-stage polling	Keeps click-stage and natural-language paths synchronized for identify domain review and subfamily review dialogs
apps/web/vue3/main.py	/api/identify/domain_review now aggregates merged and per-target review TSV files and reports species groups	Identify domain confirmation works for multi-target jobs instead of only one {prefix}.domain_review.tsv
apps/web/vue3/main.py	/api/subfamily/review resolves the latest scoped reference-classification prefix and evidence assets; evidence is reduced to rect combo + circular subfamily tree	Subfamily review shows the two intended decision plots rather than scattered tree/motif/domain/structure files
apps/web/vue3/main.py, apps/web/vue3/index.html	Added phylogeny_msa result group/stage and moved msa_plot-* pages into it	Prevents dozens of MSA pages from flooding the main phylogeny result panel
gfa_core/gene_family_scope.py, gfa_core/gene_family_stage_context.py	Added scope_summary, scope_hash, and legacy_output_prefix fields	Provides bounded, human-readable scope identity for future short filenames while preserving old long-output compatibility
apps/web/vue3/main.py	Agent plan fallback now fills target species from identify defaults for reference-classification-only plans	Natural-language "亚家族分类" planning can detect already completed scoped phylogeny/reference_classification results

Open Issues To Fix Next

Issue	Impact	Planned action
Confirmed subfamily route reuse not fully verified	Downstream plots may keep predicted labels	Verify subfamily_confirm output preference in tree/motif/domain/structure
gene_subfamily semantic candidates still include non-label tables below the correct top-ranked subfamily file	Mostly UI noise; selected file is correct	Apply the same negative filters used for reference_subfamily to generic gene_subfamily candidates
Structure postflight reports low ID/GFF overlap (22/76) even though final structure plot covers 59 genes	Diagnosis uses one GFF evidence path and does not fully account for multi-target scope aggregation	Make structure postflight overlap check consume scoped GFF source list from the structure manifest
Summary full-scope behavior not verified	Final report may miss scoped outputs	Run summary after downstream stages
Long-running stage polling still creates many local exec sessions during manual testing	Tool warning noise	Prefer API status checks and avoid long sleep sessions