admin/PlantAI-dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
master
PlantAI-dev/docs/GLOBAL_MULTI_AGENT_REPAIR_PLAN_2026-05-20.md
T
potaser f745ee9d7e Initial commit: PlantAI-dev
2026-08-03 18:42:47 +08:00

48 KiB
Executable File
Raw Permalink Blame History

Global Multi-Agent Repair and Orchestration Plan

Date: 2026-05-20

1. Goal

This is a global revision of the multi-agent working mode, not a local DiagnosisAgent-only patch.

Target framework:

User request
  -> structured intent
  -> project context index
  -> stage capability registry
  -> artifact resolver
  -> planner
  -> preflight diagnosis
  -> executor
  -> failure diagnosis + safe repair + bounded retry
  -> postflight validation
  -> result parsing / interpretation
  -> reproducibility and learning record

2. Problems to Fix

  1. User intent is still often routed by keywords and broad task classes.
  2. Existing final/input artifacts are not represented as a unified project context.
  3. Stage input checks are too file-type based.
  4. Prefix is still too user-facing and burdensome.
  5. target/reference/comparative scope is not global.
  6. Diagnosis rules exist but are not mandatory in every failure path.
  7. Repair actions are not first-class plan actions.
  8. Stage contracts, skills, CLI, frontend, and backend can drift from each other.
  9. Real-data testing is not enforced after each revision.

3. Implementation Plan

Phase 0: Documentation and Framework

Status: started.

Deliverables:

  • docs/PROJECT_EXECUTION_FRAMEWORK_2026-05-20.md
  • docs/CODE_FRAMEWORK_INDEX_2026-05-20.md
  • docs/GLOBAL_MULTI_AGENT_REPAIR_PLAN_2026-05-20.md

Purpose:

  • Establish the global code execution framework.
  • Establish a reusable file-level code framework index.
  • Define the required per-stage audit template.

Phase 1: ProjectContextIndex

Status: implemented initial read-only module.

Added:

  • gfa_core/mas/context_index.py

Responsibilities:

  • Scan job input/final/workspace/log/config/scope/manifests.
  • Build structured inventory:
    • files by semantic kind.
    • artifacts by stage.
    • family and species roles.
    • stage status.
    • warnings and repair candidates.

Must reuse:

  • gfa_core/gene_family_scope.py
  • gfa_core/mas/tools/stage_status.py
  • relevant path helpers from apps/web/pipeline_backend.py

Real-data validation:

  • Use the current real job and verify that At/Lc/Dl/Os SPL files, config, final artifacts, and manifests are indexed correctly.

Initial validation result:

  • Command:
    • /home/xia-lab/miniconda3/envs/test4py311/bin/python -m gfa_core.mas.context_index job_1779239565888_3cfb60 --prefix At.SPL --output /tmp/bbav5_context_job_1779239565888_3cfb60.json
  • Result:
    • Module compiled successfully.
    • Real job context index was generated.
    • Detected config_path=/home/xia-lab/wutx/00test/bbav5_0502_Yang_Tihuan.7880/workspace/job_1779239565888_3cfb60/final/config.autofill.draft.xlsx.
    • Detected family=SPL from explicit prefix.
    • Detected target_species=["At"] from explicit prefix.
    • Indexed final files, identify stage intermediate outputs, genome_prepare artifacts, manifests, stage statuses, and logs.

Known limitation after initial implementation:

  • Config row parsing is conservative and may need field-name expansion after reading real config.xlsx variants.
  • Species/family inference from arbitrary file names is heuristic; ArtifactResolver must not rely only on filename inference when manifest/config/scope metadata are available.

Phase 2: StageCapabilityRegistry

Status: implemented initial registry for the first high-risk stage group.

Added:

  • gfa_core/mas/stage_capabilities.py

Initial stage coverage:

  • genome_prepare
  • ncbi_fetch_sequences
  • sequence_extract_by_id
  • identify
  • phylogeny
  • reference_classification
  • motif
  • domain
  • structure
  • combo
  • summary

Initial validation result:

  • Command:
    • /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/mas/stage_capabilities.py
    • /home/xia-lab/miniconda3/envs/test4py311/bin/python -c "from gfa_core.mas.stage_capabilities import list_stage_capabilities, stage_capability_dict; print(list_stage_capabilities()); print(stage_capability_dict('identify')['consumes'][0])"
  • Result:
    • Module compiled successfully.
    • Registry returned 11 covered stages.
    • identify capability exposes target_proteome as semantic input with alternatives genome_prepare_pep, config_pep, and input_pep.

Then extend to:

  • promoter, synteny, interspecies_synteny, karyotype, ppi, expression, pep, annotation.
  • RNA-seq stages.

Each stage capability must define:

  • consumes
  • provides
  • scope_modes
  • input_alternatives
  • preflight_checks
  • postflight_checks
  • repair_candidates
  • rerun_policy
  • cli
  • env

Phase 3: ArtifactResolver

Status: implemented initial read-only resolver.

Added:

  • gfa_core/mas/artifact_resolver.py

Responsibilities:

  • Resolve semantic input needs to concrete paths.
  • Return decision trace explaining why each file was selected.
  • Provide alternatives when a required artifact is missing.

Required resolution examples:

  • target proteome for identify.
  • reference family protein file for phylogeny/reference classification.
  • scoped family member protein FASTA for motif/domain/pep.
  • target GFF for structure/promoter/karyotype/synteny.
  • count matrix and sample metadata for RNA-seq downstream stages.

Integration points:

  • Replace PlanningAgentV2._find_input_file.
  • Use in /api/execute before submitting a task.
  • Use in CLI dispatch when explicit paths can be passed.

Initial validation result:

  • Commands:
    • /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/mas/artifact_resolver.py
    • /home/xia-lab/miniconda3/envs/test4py311/bin/python -m gfa_core.mas.artifact_resolver job_1779239565888_3cfb60 identify --prefix Multi.SPL --species Lc --family SPL --output /tmp/bbav5_resolve_identify_lc_job_1779239565888_3cfb60.json
    • /home/xia-lab/miniconda3/envs/test4py311/bin/python -m gfa_core.mas.artifact_resolver job_1779239565888_3cfb60 phylogeny --prefix At.SPL --species At --family SPL --output /tmp/bbav5_resolve_phylogeny_job_1779239565888_3cfb60.json
  • Result:
    • Module compiled successfully.
    • For identify with species Lc, resolver selected /home/xia-lab/wutx/00test/bbav5_0502_Yang_Tihuan.7880/input/job_1779239565888_3cfb60/Lc.pep.fasta as target_proteome.
    • For identify with species Lc, resolver no longer misclassifies Lc.final_gfmembers.pep.fa as reference_family_pep.
    • For phylogeny with species At, resolver no longer treats genome_prepare whole proteomes as family_members_pep; it reports the missing canonical At.SPL.final_members.pep.fa while still resolving At.SPL.pep.fasta as reference family protein.

Known limitation after initial implementation:

  • Role-specific species resolution still needs scope-aware multi-species support. Current CLI helper accepts one species argument; integration must pass role-specific species lists from scope.
  • Resolver is now wired into PlanningAgentV2 preflight for stages covered by StageCapabilityRegistry.
  • Resolver is not wired into /api/execute yet.

Planning integration validation:

  • Command:
    • PlanningAgentV2._run_preflight_validation({'stages':['identify']}, job_1779239565888_3cfb60, 'Lc.SPL', workspace/job_1779239565888_3cfb60)
  • Result:
    • valid=true
    • no input error
    • selected target_proteome=/home/xia-lab/wutx/00test/bbav5_0502_Yang_Tihuan.7880/input/job_1779239565888_3cfb60/Lc.pep.fasta
  • Command:
    • PlanningAgentV2._run_preflight_validation({'stages':['identify']}, job_1779239565888_3cfb60, 'Multi.SPL', workspace/job_1779239565888_3cfb60)
  • Result:
    • valid=true
    • no input error
    • emits scope_selection warning requiring target species selection instead of silently choosing one proteome.

Phase 4: Intent Object Parser

Modify:

  • gfa_core/mas/agents/router.py
  • gfa_core/mas/agents/intent_v2.py
  • gfa_core/mas/utils/intent_parser.py

New output shape:

{
  "task_type": "pipeline | result_query | file_transform | utility | repair | qa",
  "operation": "identify | preview_fasta_headers | download_sequences | clean_headers",
  "object_ref": "At.SPL.ids",
  "family": "SPL",
  "species": ["At"],
  "scope_hint": "target_only"
}

Required behavior:

  • "根据 At.SPL.ids 下载蛋白序列" -> utility, not missing protein pipeline.
  • "展示下载好的 At.SPL 家族序列基因 id 列" -> result query, not NCBI download.
  • "过滤 protein.fa 的 id 列" -> file transform, not identify.

Phase 5: Mandatory Diagnosis Wiring

Modify:

  • gfa_core/mas/core/langgraph_orchestrator.py
  • gfa_core/mas/agents/execution.py
  • apps/web/vue3/main.py
  • frontend task polling in apps/web/vue3/index.html

Required behavior:

  • preflight issue -> diagnosis.
  • execution failure -> diagnosis.
  • postflight suspicious success -> diagnosis.
  • task failure payload always includes diagnosis status or explicit "diagnosis unavailable" reason.

Partial implementation completed before diagnosis wiring:

  • /api/execute now runs a read-only semantic preflight through _api_semantic_preflight before resource routing and task submission.
  • If semantic inputs are missing, /api/execute returns preflight_failed without submitting the background task.
  • If identify sees multiple target proteomes but no selected target species, /api/execute returns needs_confirmation without submitting the background task.
  • This is not the full diagnosis loop yet; it is the execution-entry guard needed before failure diagnosis and auto repair.
  • Threadpool run_stage_sync failure path now calls _run_failure_diagnosis_for_task, stores tasks[task_id]["diagnosis"], and includes diagnosis in the failed task result.

Validation:

  • Command:
    • _api_semantic_preflight('job_1779239565888_3cfb60','identify','Lc.SPL', {'geneFamily':'SPL','targetSpecies':['Lc']})
  • Result:
    • valid=true
    • needs_confirmation=false
    • selected Lc.pep.fasta as target proteome.
  • Command:
    • _api_semantic_preflight('job_1779239565888_3cfb60','identify','Multi.SPL', {'geneFamily':'SPL','targetSpecies':[]})
  • Result:
    • valid=true
    • needs_confirmation=true
    • issue code identify_target_species_required.
  • Command:
    • _run_failure_diagnosis_for_task('job_1779239565888_3cfb60','identify','Multi.SPL','simulated PFAM 404')
  • Result:
    • returned structured diagnosis with gate=failure, stage=identify, and one issue.
    • LLM fallback was skipped because no LLM API key is configured; deterministic diagnosis still returned.

Remaining:

  • Celery worker task failure path already contains _attach_diagnosis_to_failure, _diagnose_stage_gate, and runtime repair hooks in tasks.py; it should still be included in later end-to-end failure tests.
  • LangGraph execution failure edge has now been changed to route to diagnosis before result parsing.

LangGraph routing implementation:

  • execution now uses _execution_decision.
    • execution_status == "error" -> diagnosis
    • otherwise -> result_parser
  • _diagnosis_node now forwards execution_error, fallback stage, job_id, and prefix into the DiagnosisAgent payload.
  • _diagnosis_node attaches diagnosis output into final_output["diagnosis"] for failed execution states.
  • _diagnosis_decision confirmed-fix path now returns retry, matching the graph edge map.

Validation:

  • Command:
    • /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/mas/core/langgraph_orchestrator.py
  • Result:
    • compile passed.
  • Command:
    • LangGraphOrchestrator.__new__()._execution_decision({'execution_status':'error'})
    • LangGraphOrchestrator.__new__()._execution_decision({'execution_status':'success'})
    • _diagnosis_decision({'pending_confirmation': {'status':'confirmed'}})
  • Result:
    • error routes to diagnosis
    • success routes to result_parser
    • confirmed repair routes to retry
  • Command:
    • _diagnosis_node with simulated failed state for job_1779239565888_3cfb60, stage=identify, prefix=Multi.SPL, execution_error='simulated PFAM 404'
  • Result:
    • diagnosis node ran.
    • stage fallback resolved to identify.
    • final_output contains diagnosis.

Phase 6: Repair Executor

Status: implemented initial repair_plan policy layer and one real auto repair path.

Modify:

  • gfa_core/mas/agents/diagnosis.py
  • optionally add gfa_core/mas/repair_executor.py

Repair policy:

  • auto: apply job-local safe repair and retry once.
  • confirm: show existing-style confirmation dialog.
  • report_only: show actionable diagnosis.
  • dev_fix: write developer repair report; do not patch global source at runtime.

Implemented in DiagnosisAgent:

  • REPAIR_POLICY_BY_CODE maps diagnosis issue codes to one of auto, confirm, report_only, or dev_fix.
  • diagnose_job_stage and diagnose_stage_gate now return repair_plan.
  • apply_runtime_repairs only executes actions classified as auto; report-only and confirm-only actions are not silently applied.
  • protein_fasta_stop_markers is now a real automatic job-local repair.
  • _apply_protein_stop_marker_fix can repair a FASTA path supplied in issue evidence, not only final/{prefix}.final_members.pep.fa.

Current repair policy table:

  • auto:
    • protein_fasta_stop_markers
    • identify_pfam_remote_404
    • pfam_remote_404
    • identify_missing_hmm_or_pfam
    • ncbi_family_reference_link_missing
    • manifest_missing
    • manifest_missing_stage_incomplete
    • genome_prepare_final_gff_empty
    • genome_prepare_pep_looks_like_genome
    • genome_prepare_cds_looks_like_genome
  • confirm:
    • hmm_or_pfam_missing
  • report_only:
    • gene_gff_id_overlap_low
    • manifest_missing_stage_incomplete
    • manifest_missing
    • manifest_failed
    • upstream_genome_prepare_partial_success
  • dev_fix:
    • safe patch candidates that imply global source-code changes.

Validation:

  • Command:
    • /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/mas/agents/diagnosis.py tasks.py apps/web/vue3/main.py
  • Result:
    • compile passed.
  • Command:
    • DiagnosisAgent(None).diagnose_stage_gate('job_1779239565888_3cfb60','identify','Multi.SPL','failure')
  • Result:
    • returned repair_plan.
    • issue manifest_missing_stage_incomplete is classified as report_only.
    • no auto repair or confirmation is triggered for an incomplete stage manifest alone.
  • Real data command:
    • scanned real identify output workspace/job_1779239565888_3cfb60/identify/Dl/final/Dl.final_gfmembers.pep.fa.
  • Real data result:
    • the FASTA contained terminal * stop markers.
    • apply_runtime_repairs with issue protein_fasta_stop_markers created workspace/job_1779239565888_3cfb60/identify/Dl/final/Dl.final_gfmembers.pep.clean.fa.
    • repair metadata was written to workspace/job_1779239565888_3cfb60/identify/Dl/final/Dl.final_gfmembers.pep.clean.fa.repair.json.
    • 22 sequences were scanned, 22 sequences had stop markers, 22 stop markers were removed.
    • the clean FASTA was verified with rg -n "\*", and no * remained.
    • the original FASTA was not modified.
  • Real PFAM fallback validation:
    • run_identify.py now reads Required_Pfams from config.xlsx/config.autofill.draft.xlsx in input/, workspace/, or workspace/final/.
    • local HMM discovery now includes assets/hmm and assets/examples/test_input.
    • Pfam-A discovery now includes /home/xia-lab/00.share_data/hmm/Pfam-A.hmm.
    • DiagnosisAgent.apply_runtime_repairs now handles identify_pfam_remote_404, pfam_remote_404, and identify_missing_hmm_or_pfam by preparing a job-local query HMM under input/<job_id>/genome_prepare/hmm_cache/.
    • Real job repair test created /home/xia-lab/wutx/00test/bbav5_0502_Yang_Tihuan.7880/input/job_1779239565888_3cfb60/genome_prepare/hmm_cache/PF03110.hmm and /home/xia-lab/wutx/00test/bbav5_0502_Yang_Tihuan.7880/input/job_1779239565888_3cfb60/genome_prepare/hmm_cache/PF03110.diagnosis_repair.json.
    • repair_plan for identify_pfam_remote_404 is now policy=auto, action=prepare_local_pfam_hmm, and requires_user_confirmation=false.
    • Full real identify test was run on target Lc, family SPL, reference At, using /tmp/bbav5_identify_pfam_fallback_real as workspace.
    • The run used local PF03110.hmm, completed HMMER + BLAST + InterProScan, and wrote /tmp/bbav5_identify_pfam_fallback_real/final/CodexPFAM.SPL.identify.manifest.json with status=success.
    • The real identify test produced 33 Lc SPL candidate members and /tmp/bbav5_identify_pfam_fallback_real/final/CodexPFAM.SPL.final_members.pep.fa.
  • Real NCBI reference-link validation:
    • run_ncbi_fetch_sequences.py already creates {Species}.{Family}.pep.fa for new runs.
    • DiagnosisAgent now also diagnoses old/partial jobs where *.ncbi_fetch.protein.fa exists but {Species}.{Family}.pep.fa is missing.
    • Issue code is ncbi_family_reference_link_missing, policy is auto, action is create_family_reference_pep_link.
    • Test used real protein FASTA copied from workspace/job_1779022508209_52539d/final/At.SPL.ncbi_fetch.protein.fa.
    • Repair created workspace/codex_ncbi_link_test/final/At.SPL.pep.fa -> At.SPL.ncbi_fetch.protein.fa.
    • The symlink was verified with readlink -f and head; the first FASTA header was >At2g47070 ....
  • Real manifest recovery validation:
    • DiagnosisAgent now recovers missing manifests only from verified non-empty final artifacts.
    • Initial coverage is intentionally limited to identify and ncbi_fetch_sequences.
    • identify recovery requires both {prefix}.final_members.pep.fa and {prefix}.final_members.id.
    • ncbi_fetch_sequences recovery requires at least one valid NCBI final artifact such as protein FASTA, CDS FASTA, or mapping TSV.
    • manifest_missing and manifest_missing_stage_incomplete now have policy auto, action recover_manifest_from_final_artifacts.
    • Recovery refuses to run if the same job/stage is currently running.
    • Recovery writes recovered_manifest.recovered=true and a warning that original parameters/elapsed time are unavailable.
    • Real identify test copied final artifacts from /tmp/bbav5_identify_pfam_fallback_real/final/ into workspace/codex_manifest_identify_test2/final/.
    • Recovered manifest: workspace/codex_manifest_identify_test2/final/CodexPFAM.SPL.identify.manifest.json.
    • Recovered identify metrics: final_members=33, final_member_ids=33, sequence_count=33, status=success.
    • Real NCBI test copied At.SPL.ncbi_fetch.protein.fa, At.SPL.ncbi_fetch.cds.fa, and At.SPL.ncbi_fetch.mapping.tsv into workspace/codex_manifest_ncbi_test2/final/.
    • Recovered manifest: workspace/codex_manifest_ncbi_test2/final/At.SPL.ncbi_fetch.manifest.json.
    • Recovered NCBI metrics: protein_records=17, mapping_rows=17, status=success.
    • should_rerun_stage is now action-specific: manifest recovery does not request rerun; PFAM HMM preparation does request identify rerun.

Remaining initial auto repairs:

  • pep/cds genome-like re-extraction from genome+GFF.
  • artifact re-resolution when final artifact exists but preflight missed it.

Important limitation:

  • Generic non-identify hmm_or_pfam_missing remains confirm; the automatic local HMM fallback is currently scoped to identify where family/config semantics are available and validated.

Phase 7: Frontend UX

Status: partially implemented for diagnosis/repair-plan display.

Modify:

  • apps/web/vue3/index.html

Rules:

  • Reuse existing ElementPlus dialog style.
  • Hide prefix for gene-family stages.
  • Show family and target/reference/comparative selection.
  • Auto-generate internal prefix.
  • Apply 40-second countdown globally.
  • Provide pause countdown button globally.
  • Always show diagnosis popup after failed stage if diagnosis was run.

Implemented:

  • The existing ElementPlus diagnosis dialog is reused; no new dialog framework was introduced.
  • /api/agent/diagnose_job_stage now treats repair_plan actions with policy confirm or dev_fix as confirmable diagnosis records, not only legacy safe_patch_candidate issues.
  • Diagnosis dialogs now use a dynamic title:
    • 诊断结果 for read-only diagnosis.
    • 参数修复确认 for executable job-local patch actions.
    • 诊断确认 for cases requiring user-supplied files/parameters/resources.
    • 自动修复结果 for auto-repair actions reported by the executor.
    • 代码修订建议 for global source-code repair candidates.
  • repair_plan.actions is displayed directly in the dialog.
  • Actions are labeled by policy:
    • auto -> 自动修复
    • confirm -> 需确认
    • report_only -> 仅报告
    • dev_fix -> 代码修订
  • The parameter diff block is hidden for report-only diagnostics, so manifest_missing_stage_incomplete no longer looks like an executable parameter repair.
  • Footer text now explains whether the result is read-only, confirm-only, auto-repair reporting, or source-code repair.

Validation:

  • Command:
    • /home/xia-lab/.nvm/versions/node/v22.22.0/bin/node -e "... new vm.Script(inlineScript) ..."
  • Result:
    • js-parse-ok
  • Command:
    • DiagnosisAgent(None)._build_repair_plan('identify', [{'code':'identify_pfam_remote_404', ...}])
  • Result:
    • returned requires_user_confirmation=true.
    • this previous result is superseded by Phase 6 PFAM fallback implementation; current action policy is auto, action is prepare_local_pfam_hmm.

Remaining:

  • Prefix-free target/reference/comparative selection dialogs still need to be finished globally.
  • The 40-second countdown and pause button must be audited across every dialog, not only the batch execute dialog.
  • The frontend can display repair_plan, but PFAM fallback and manifest recovery still need backend repair executors before their buttons can perform real automatic actions.

Phase 8: Contracts, Skills, and Reproducibility

Modify:

  • gfa_core/schemas/stages/*_contract_v1.md
  • skills/*/SKILL.md
  • gfa_core/reproducibility_v2.py
  • stage manifest writers where needed.

Required additions:

  • semantic consumes/provides.
  • scope behavior.
  • resolver decision trace.
  • context snapshot path.
  • repair actions applied.
  • retry count.

Phase 9: Real Data Test Pass

Primary test:

  • Family: SPL
  • Targets: Lc, Dl
  • Reference: At
  • Comparative: Os

Test sequence:

  1. Config autofill and scope creation.
  2. Data preparation.
  3. NCBI sequence download from At.SPL.ids.
  4. Identify for selected targets.
  5. Phylogeny with target+reference scope.
  6. Reference classification.
  7. Motif/domain/structure.
  8. Combo joint plot.
  9. Summary aggregation.
  10. Failure simulations:
    • PFAM 404.
    • stop marker in protein.
    • missing manifest but valid final artifact.
    • wrong/missing protein file path.

Each phase is not complete until real output files and manifests are verified.

4. Execution Order for Code Revisions

  1. Finish the documentation baseline.
  2. Add ProjectContextIndex.
  3. Add stage capabilities for the first stage group.
  4. Add ArtifactResolver.
  5. Wire resolver into planning and /api/execute.
  6. Wire mandatory diagnosis into execution failure paths.
  7. Add safe repair executor and bounded retry.
  8. Update frontend dialogs.
  9. Update contracts/skills/repro.
  10. Run full real-data test and record results in docs/revisions/.

5. Revision Log

2026-05-20 genome_prepare runtime repair and context resolver fix

Changed:

  • gfa_core/mas/agents/diagnosis.py
    • Added genome_prepare / genome_prepare_v2 to STAGE_SCRIPT_MAP, fixing false "stage is running" blocks caused by matching the plain string genome_prepare in diagnostic commands.
    • Expanded genome/GFF candidate discovery for genome_prepare_pep_looks_like_genome and genome_prepare_cds_looks_like_genome.
    • GFF candidates are now species-ranked; unrelated all-GFF fallback is only used when there is exactly one candidate.
    • pep/cds extraction now moves invalid files to diagnosis_repair_backups/ before writing replacements. This prevents writing through symlinks to source genome files.
    • Successful extraction returns should_rerun_stage=true, so the formal genome_prepare stage can rerun longest-transcript selection and ID unification.
  • gfa_core/mas/context_index.py
    • Project context now prefers the real upload directory input/<job_id> over the non-existent workspace/<job_id>/input.
    • Species/family inference now treats Pp.pep.fasta / Pp.cds.fasta as species-only files, while preserving {Species}.{Family}.pep.fa parsing.

Real data validation:

  • Created isolated job input/codex_genome_reextract_pp_20260520.
  • Used real Pp.genome.fasta and Pp_final.gff3 from job_1779239565888_3cfb60.
  • Simulated invalid pep/cds as genome-like nucleotide FASTA.
  • Diagnosis before repair:
    • genome_prepare_pep_looks_like_genome blocking.
    • genome_prepare_cds_looks_like_genome warning.
  • Runtime repair:
    • repair_success=True
    • should_rerun_stage=True
    • gffread returned 0.
  • Post-repair profiles:
    • Pp.pep.fasta: protein-like, not genome-like.
    • Pp.cds.fasta: nucleotide CDS-like, not genome-like.
  • Re-running preflight diagnosis after repair returned status=pass, issues=[].
  • Resolver validation:
    • python -m gfa_core.mas.artifact_resolver codex_genome_reextract_pp_20260520 identify --prefix Multi.SPL --species Pp --family SPL
    • resolved target_proteome from real input/<job_id>/Pp.pep.fasta.

Incident and mitigation:

  • An early symlink-based simulation exposed a real risk: writing gffread output to a symlink path can truncate the symlink target.
  • The affected real file input/job_1779239565888_3cfb60/Lc.genome.fasta was immediately restored from input/job_1779239565888_3cfb60/Lc/Lc.genome.fasta.
  • The code now moves existing pep/cds paths out of the way before gffread writes replacements, preventing this failure mode.

2026-05-20 semantic resolver execution context

Changed:

  • apps/web/vue3/main.py
    • /api/execute now writes semantic preflight decisions into the actual execution params, not only the frontend task record.
    • API responses now expose resolved_artifacts, so the frontend can show which real input/final artifact was selected.
  • tasks.py
    • Added worker-side attach_semantic_execution_context().
    • Celery rebuilds resolver decisions before preflight diagnosis and stage execution when the request did not already provide them.
    • Success and failure payloads now include semantic_preflight and resolved_artifacts.

Why this matters:

  • The planning UI, Celery worker, and DiagnosisAgent now share the same resolver evidence.
  • Direct worker calls no longer depend on the frontend having run semantic preflight first.
  • This closes the gap where the UI could say an input was resolved, while the worker/diagnosis still behaved as if only raw prefix/file-extension guessing existed.

Real data validation:

  • Command:
    • attach_semantic_execution_context('job_1779239565888_3cfb60','identify','Multi.SPL',{'targetSpecies':['Lc'],'geneFamily':'SPL'})
  • Result:
    • resolvedArtifacts['Lc:target_proteome'] resolved to real file input/job_1779239565888_3cfb60/Lc.pep.fasta.
    • context input directory resolved to input/job_1779239565888_3cfb60.
  • Command:
    • _api_semantic_preflight('job_1779239565888_3cfb60','identify','Multi.SPL',{'targetSpecies':['Lc'],'geneFamily':'SPL'})
  • Result:
    • valid=True
    • needs_confirmation=False
    • resolved Lc:target_proteome from the genome_prepare manifest artifact.
    • config path resolved to input/job_1779239565888_3cfb60/config.xlsx.

2026-05-20 frontend scope-first gene-family confirmation

Changed:

  • apps/web/vue3/index.html
    • Gene-family stages no longer expose prefix as the primary editable field in the main parameter area or execution dialogs.
    • Batch confirmation now shows one scope-first panel for gene-family stages:
      • Family input.
      • Target checkbox group.
      • Reference checkbox group.
      • Comparative checkbox group.
      • Internal output naming preview.
    • Stage confirmation dialogs reuse the existing Element Plus dialog style and now allow Family to be filled when config/scope has no family.
    • Switching scope mode now synchronizes reference/comparative selections instead of leaving stale selections.
    • Batch countdown pauses when Family or Target species is missing, preventing automatic execution before the key biological scope is confirmed.
    • Fallback/simple planning path also loads scope options and no longer references an undefined stagesDetail.

Validation:

  • JavaScript parse check:
    • js-parse-ok scripts=1
  • Python compile check:
    • py_compile apps/web/vue3/main.py tasks.py passed.

2026-05-20 promoter and summary scope close-out

Changed:

  • apps/web/pipeline_backend.py
    • summary now returns the scoped summary manifest when a scope_file is present, instead of always returning {prefix}.summary.manifest.json.
  • apps/cli/run_promoter.py
    • Gzipped GFF3/genome inputs are materialized to plain files before gffread/seqkit.
    • Reused PlantCARE tabs copied from outside the workspace are copied into the current analysis directory before artifact registration.
    • seqkit subseq stdout is no longer echoed into logs; the FASTA is still captured and written to gene.upstream.fasta.
    • Reused PlantCARE tab files are now checked against the extracted promoter FASTA IDs. A completely non-overlapping tab fails early with a failure manifest instead of producing an empty visualization under a success status.

Real data validation:

  • Workspace: workspace/codex_scope_phy_ref_fast_test
  • Summary scope: SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference
    • Backend run_stage("summary", ..., scope_file=...) returned:
      • final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.summary.manifest.json
    • This verifies that the backend return path now matches the scoped output generated by run_summary.py.
  • Promoter scope: SPL.target-Lc.ref-At.comp-Os.target_only
    • Real Lc GFF3 and genome were decompressed/reused successfully.
    • gffread converted GFF3 to GTF.
    • 33 target Lc IDs matched the GTF and 33 upstream promoter sequences were extracted.
    • A reused Ft PlantCARE tab was intentionally tested against the Lc promoter FASTA and failed with:
      • PlantCARE result IDs do not match the extracted promoter FASTA IDs
    • Failure manifests were written for both generic and scoped promoter manifests, which is the intended behavior for mismatched external PlantCARE results.
  • Compile check:
    • py_compile apps/cli/run_promoter.py apps/web/pipeline_backend.py apps/cli/run_summary.py passed.

2026-05-20 downstream scoped manifest returns and multi-target synteny

Changed:

  • apps/web/pipeline_backend.py
    • ortholog, kaks, karyotype, synteny, interspecies_synteny, and ppi now prefer returning the scoped manifest path when a scope_file is present and the scoped manifest exists.
    • synteny multi-target scope is no longer passed directly to the single-target CLI. The backend splits target_species into one child scope per target species, runs each child scope separately, and writes an aggregate manifest for the original multi-target scope.
    • Failed single-scope synteny now returns the synteny manifest if it exists, instead of leaking an unrelated previously selected manifest to the caller.

Real data validation:

  • Workspace: workspace/codex_scope_phy_ref_fast_test
  • Scope: SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference
  • ortholog:
    • Re-ran real Lc/Dl target family proteins against At SPL reference proteins.
    • Returned scoped manifest:
      • final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.ortholog.manifest.json
    • Metrics: ortholog_pairs=54.
  • kaks:
    • Re-ran real Lc→At and Dl→At comparisons.
    • Returned scoped manifest:
      • final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.kaks.manifest.json
    • Metrics: species_runs=2, input_pairs=54.
  • karyotype:
    • Re-ran real Lc and Dl karyotype plotting.
    • Returned scoped manifest:
      • final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.karyotype.manifest.json
    • Metrics: species_count=2, success_count=2, failure_count=0.
  • synteny:
    • The first direct multi-target run failed with the expected single-target CLI error and revealed an incorrect backend return path.
    • After revision, the backend split the multi-target scope into:
      • SPL.target-Lc.ref-At.comp-Os.target_plus_reference
      • SPL.target-Dl.ref-At.comp-Os.target_plus_reference
    • Both child runs completed successfully.
    • Aggregate scoped manifest:
      • final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.synteny.manifest.json
    • Aggregate metrics: species_runs=2, success_count=2, failure_count=0.
    • Child metrics:
      • Lc: total_alignments=261053, synteny_blocks=49
      • Dl: total_alignments=135791, synteny_blocks=49
  • Compile check:
    • py_compile apps/web/pipeline_backend.py passed.

2026-05-20 interspecies synteny and diagnosis coverage

Changed:

  • apps/web/pipeline_backend.py
    • interspecies_synteny returns the scoped manifest when the scoped manifest exists.
  • gfa_core/mas/agents/diagnosis.py
    • Added diagnosis policies for:
      • plantcare_id_mismatch
      • multi_target_single_species_stage
      • interspecies_pair_no_hits
    • Generic gene-family stage audit now checks both stage log text and manifest text, so errors recorded only in manifest can still be diagnosed.
    • Stage log lookup no longer falls back to the latest unrelated log when a stage-specific log is missing. This prevents false diagnosis caused by log cross-talk between stages.

Real data validation:

  • interspecies_synteny
    • Ran real Lc/Dl target species against At reference with DIAMOND requested.
    • IQ-TREE completed on 72 proteins.
    • Returned scoped manifest:
      • final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.interspecies_synteny.manifest.json
    • Status: partial_success
    • Tree: success.
    • Pairwise synteny:
      • Dl-At: success, anchor_count=12181, marked_block_count=1.
      • Lc-At: failed because reciprocal search produced no hits after DIAMOND fallback to BLASTP.
    • Metrics included pair_success_count=1, pair_total_count=2, blast_fallback_count=1.
  • Diagnosis validation:
    • DiagnosisAgent.diagnose_job_stage(..., "promoter", "Multi.SPL") now reports:
      • manifest_failed
      • plantcare_id_mismatch
    • DiagnosisAgent.diagnose_job_stage(..., "interspecies_synteny", "Multi.SPL") now reports:
      • interspecies_pair_no_hits
      • empty_required_output
    • The promoter diagnosis no longer incorrectly imports the latest interspecies synteny log.
  • Compile check:
    • py_compile gfa_core/mas/agents/diagnosis.py apps/web/pipeline_backend.py passed.

2026-05-20 interspecies synteny compressed FASTA repair

Changed:

  • apps/cli/run_interspecies_synteny.py
    • Added materialize_plain_fasta(...) for reciprocal DIAMOND/BLASTP searches.
    • .gz protein FASTA inputs are now decompressed into the stage analysis directory before search tools are called.
    • DIAMOND/BLASTP stderr is retained in the pair result.
    • Empty-output diagnostics now include per-output file sizes in blast_output_stats.
    • Pairwise manifest records now include blast_stderr, diamond_stderr, and blast_output_stats.

Why this matters:

  • The previous real run produced partial_success for Lc/Dl vs At. Dl-At succeeded, but Lc-At produced no hits after DIAMOND fell back to BLASTP.
  • The Lc command path used /home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.pep.sim.fa.gz directly in BLASTP. After materializing the same file to plain FASTA, Lc→At BLASTP immediately produced hits, showing that the previous no-hit result was an input handling issue rather than a true absence of homology.

Real data validation:

  • Test input:
    • Lc whole proteome: /home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.pep.sim.fa.gz
    • At whole proteome: /home/xia-lab/wutx/plantAnalysisAgent/ref_data/ath/Arabidopsis_thaliana.TAIR10.pep.all.fa
  • Validation command directly called run_reciprocal_blast(...) on real Lc/At files in:
    • workspace/codex_scope_phy_ref_fast_test/interspecies_synteny/repair_test_lc_at
  • Observed after repair:
    • Plain FASTA was created:
      • Lc.search.Lchinesis_genome.pep.sim.fa, 25 MB.
    • Lc→At BLAST output was no longer empty:
      • tmp.Lc_At.blast, 20247 lines, 1.4 MB at the time the validation was stopped.
  • The independent validation BLASTP process was stopped after confirming non-empty real hits to avoid leaving a long-running full-proteome test consuming CPU. No background validation process remained.
  • Compile check:
    • py_compile apps/cli/run_interspecies_synteny.py passed.

2026-05-20 interspecies synteny full rerun success

Changed:

  • apps/web/pipeline_backend.py
    • When final/{prefix}.interspecies_tree.nwk exists, the backend now passes --reuse-tree to run_interspecies_synteny.py.
    • This prevents a pairwise-input fix from repeatedly spending ~18 minutes in IQ-TREE when the tree is already available for the same prefix/scope.
  • apps/cli/run_interspecies_synteny.py
    • Search-safe protein FASTA materialization now handles both compressed input and invalid protein characters before DIAMOND/BLASTP.

Real data validation:

  • Workspace: workspace/codex_scope_phy_ref_fast_test
  • Scope: SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference
  • Backend command included --reuse-tree and reused:
    • final/Multi.SPL.interspecies_tree.nwk
  • Full stage rerun completed with status success.
  • Updated scoped manifest:
    • final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.interspecies_synteny.manifest.json
  • Metrics:
    • tree_success=True
    • synteny_success=True
    • pair_success_count=2
    • pair_total_count=2
    • blast_fallback_count=0
    • blast_engine_used=["diamond", "diamond"]
  • Pair details:
    • Lc-At:
      • blast_success=True
      • engine=diamond
      • fallback=None
      • anchor_count=9153
      • output sizes: tmp.Lc_At.blast=14055578, tmp.At_Lc.blast=11822370, Lc.At.blast=25877948
    • Dl-At:
      • blast_success=True
      • engine=diamond
      • fallback=None
      • anchor_count=12181
      • output sizes: tmp.Dl_At.blast=7318933, tmp.At_Dl.blast=10131817, Dl.At.blast=17450750
  • Diagnosis validation:
    • DiagnosisAgent.diagnose_job_stage(..., "interspecies_synteny", "Multi.SPL") returned no issues after the successful rerun.
  • Process check:
    • No lingering run_interspecies_synteny, DIAMOND, BLASTP, or IQ-TREE process remained after validation.

2026-05-20 promoter final GFF3 and PPI external validation

Changed:

  • apps/cli/run_promoter.py
    • Scope mode now prefers genome_prepare normalized final GFF3 files before raw scope GFF records.
    • Lookup order includes:
      • input/{Species}_final.gff3
      • input/{Species}.final.gff3
      • input/{Species}/{Species}_final.gff3
      • workspace/final/{Species}_final.gff3
    • Raw whole-genome GFF/GFF3 from scope is now only a fallback.
  • skills/ppi/run.py
    • Ortholog-mapped query IDs are de-duplicated after mapping to STRING species IDs.
    • STRING URL now uses the requested --score value instead of hard-coded required_score=400.
    • Empty STRING network results now still write useful metrics with zero-valued interaction counts.

Real data validation:

  • Promoter:
    • Scope: SPL.target-Lc.ref-At.comp-Os.target_only
    • The stage now logs and uses:
      • input/job_1779239565888_3cfb60/Lc_final.gff3
    • It no longer uses the raw scope GFF:
      • /home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.Chr.gff3.gz
    • With final GFF3, 31 Lc target transcripts/genes matched and 31 upstream sequences were extracted.
    • The intentionally mismatched Ft PlantCARE tab was still correctly rejected by the PlantCARE ID overlap check.
  • PPI:
    • Ran real external STRING query through the backend with target Lc/Dl, reference At, score 700.
    • Scoped manifest:
      • final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.ppi.manifest.json
    • STRING URL now contains 11 unique At query IDs and required_score=700.
    • Metrics are no longer empty:
      • input_genes=11
      • genes_in_network=0
      • total_interactions=0
      • score bucket counts all 0 at threshold 700.
  • Compile check:
    • py_compile apps/cli/run_promoter.py skills/ppi/run.py apps/web/pipeline_backend.py passed.

2026-05-20 phylogeny scope input resolution

Changed:

  • apps/cli/run_phylogeny.py
    • Kept scope JSON as the first source of truth for selected_species.
    • Added reference/comparative family protein fallback resolution when scope lacks a usable family_pep record.
    • Checked common real project paths:
      • input/{Species}.{Family}.pep.fa
      • input/{Species}.{Family}.pep.fasta
      • input/{Species}/{Species}.{Family}.pep.fa
      • input/{Species}/{Species}.{Family}.pep.fasta
      • final/{Species}.{Family}.pep.fa
      • final/{Species}.{Family}.ncbi_fetch.protein.fa
    • Error messages now include the actual checked paths for each missing species.
    • Scope manifest information now records the inferred input directory used for fallback resolution.

Why this matters:

  • A user selecting target_plus_reference should get exactly target species plus reference species in the tree input.
  • Reference species such as At should not disappear just because the frontend/config scope record missed a family protein file that exists in input/ or final/.
  • A user selecting target_only should not silently build a tree for every species in the project.

Real data validation:

  • Job: job_1779239565888_3cfb60
  • Scope: target Lc,Dl, reference At, comparative Os, mode target_plus_reference, family SPL.
  • Resulting phylogeny input FASTA:
    • Lc: 37 sequences from real identify output.
    • Dl: 22 sequences from real identify output.
    • At: 17 sequences from real input/job_1779239565888_3cfb60/At.SPL.pep.fa.
    • Total: 76 sequences.
  • Target-only validation:
    • Scope: target Lc, reference At, comparative Os, mode target_only.
    • Resulting phylogeny input FASTA contained only Lc: 37 sequences.
  • Compile check:
    • py_compile apps/cli/run_phylogeny.py gfa_core/gene_family_scope.py apps/web/pipeline_backend.py apps/web/vue3/main.py passed.

2026-05-20 reference classification scope consistency

Changed:

  • apps/web/pipeline_backend.py
    • Added resolve_scoped_tree_from_phylogeny().
    • reference_classification now resolves scope_file before selecting the tree.
    • When scope is present, it prefers the matching scoped phylogeny manifest:
      • final/{prefix}.{scope_id}.phylogeny.manifest.json
      • then final/{prefix}.phylogeny.manifest.json
      • then final/{prefix}.tree.nwk as fallback.
  • apps/cli/run_reference_classification.py
    • find_target_proteins() can now reuse parameters.scope_input_fasta from the matching scoped phylogeny manifest.
    • Auto-built joint reference tree no longer depends only on final/{prefix}.final_members.pep.fa; scope runs can provide their exact phylogeny input FASTA.

Why this matters:

  • Reference classification should classify the tree built for the user's selected target/reference/comparative scope, not whatever unscoped final/{prefix}.tree.nwk happens to contain.
  • If the user chose a narrow target-only or target-plus-reference scope, downstream classification now has a direct path back to that same scope's phylogeny manifest.

Real data validation:

  • Workspace: workspace/codex_scope_phy_ref_fast_test
  • Input: input/job_1779239565888_3cfb60
  • Scope: family SPL, target Lc,Dl, reference At, comparative Os, mode target_plus_reference.
  • Resolver validation:
    • Resolved tree from scoped phylogeny manifest to final/Multi.SPL.tree.nwk.
    • Resolved target protein source from existing final/scoped phylogeny inputs.
  • Full script validation:
    • run_reference_classification.py --prefix Multi.SPL ... --scope-file /tmp/bbav5_refclass_scope_test.json exited 0.
    • Output manifest status: success.
    • Scope id recorded: SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.
    • Selected species recorded: Lc,Dl,At.
    • Classified target genes: 55.
    • Reference genes: 17.
    • Scope filter applied with 0 dropped assignment rows.
    • Scoped manifest was written.
  • Compile check:
    • py_compile apps/cli/run_reference_classification.py apps/web/pipeline_backend.py passed.

2026-05-20 downstream scope consistency for structure/motif/domain/combo

Changed:

  • apps/cli/run_structure.py
    • Added --tree.
    • Added scoped tree resolution from the matching phylogeny manifest.
    • Scope mode now uses target-species GFF3 files with the same tree used by phylogeny/reference classification.
  • apps/cli/run_motif.py
    • Added --tree.
    • Added scoped tree resolution from the matching phylogeny manifest.
    • Motif manifest now always records meme_xml and meme_results artifacts, including reused MEME results.
  • apps/cli/run_domain.py
    • Replaced latest-manifest tree lookup with scoped phylogeny tree resolution.
  • apps/web/pipeline_backend.py
    • Structure, motif, and domain now pass the same scoped tree to CLI when scope_file is present.
  • apps/cli/run_combo.py
    • Combo now resolves MEME XML from the matching scoped motif manifest instead of blindly scanning the shared motif/analysis/meme_out directory.
    • It still requires scoped manifests for phylogeny, motif, structure, and domain when scope is present.

Why this matters:

  • Tree, subfamily, motif, structure, domain, and combo plots now refer to the same selected species scope.
  • A later run with a different scope is less likely to silently reuse stale tree or MEME XML from another scope.
  • Structure remains target-GFF based, while tree/subfamily can include reference species for subfamily interpretation.

Real data validation:

  • Workspace: workspace/codex_scope_phy_ref_fast_test
  • Scope: family SPL, target Lc,Dl, reference At, comparative Os, mode target_plus_reference.
  • structure:
    • Used Lc and Dl target GFF3 files.
    • Used the scoped phylogeny tree with 72 tips.
    • Processed 55 target genes.
    • Generated structure PNG/PDF and scoped manifest.
  • motif:
    • Used scoped phylogeny FASTA with 72 proteins.
    • Used the same scoped tree and reference-classification subfamily file.
    • Reused matching MEME signature after the first run.
    • Generated motif PNG/PDF and scoped manifest.
  • domain:
    • Used scoped phylogeny FASTA with 72 proteins.
    • Used the same scoped tree and subfamily file.
    • hmmscan completed successfully.
    • Significant domain hits: 92.
    • Proteins with domains: 63/72.
    • Generated domain PNG/PDF and scoped manifest.
  • combo:
    • Used scoped phylogeny tree.
    • Used scoped motif manifest to resolve MEME XML.
    • Used scoped structure and domain manifests.
    • Generated Multi.SPL.combo_tree_motif_structure_domain.pdf and scoped combo manifest.
  • Compile check:
    • py_compile apps/cli/run_motif.py apps/cli/run_combo.py apps/cli/run_domain.py apps/cli/run_structure.py apps/web/pipeline_backend.py passed.
  • Real data scope validation:
    • Job: job_1779239565888_3cfb60
    • build_gene_family_scope(...) returned:
      • prefix: Multi.SPL
      • family: SPL
      • target: Lc, Dl
      • reference: At
      • comparative: Cp, Os, Pp, Sl
  • HTTP validation note:
    • curl http://127.0.0.1:7880/api/gene-family/scope-options... could not connect in this shell session, so endpoint-level verification was not completed here. The same underlying scope builder was validated directly with real data.

2026-05-20 frontend countdown and pause audit

Changed:

  • apps/web/vue3/index.html
    • Ordinary parameter confirmation dialog now shows a clear paused state after the countdown is paused.
    • Stage confirmation and batch confirmation remain on 40-second countdowns.
    • Stale comments that still said "15-second countdown" were corrected to "40-second countdown".
    • Interspecies synteny helper text was corrected from 60 seconds to 40 seconds.
    • Diagnosis/repair confirmation explicitly states that repair will not auto-execute and requires manual confirmation. This is intentionally not a countdown auto-apply path because diagnosis fixes can modify job-local inputs or prepare patches.

Validation:

  • Text audit:
    • No remaining 15秒, 15 秒, 60秒, 60 秒, or 60 seconds countdown text in apps/web/vue3/index.html.
  • JavaScript parse check:
    • js-parse-ok scripts=1
  • Python compile check:
    • py_compile apps/web/vue3/main.py tasks.py passed.
Reference in New Issue View Git Blame Copy Permalink