# Gene Family Scope Refactor Plan 本文档记录基因家族分析链路的代码通读结果、当前直接逻辑、关联代码、主要风险、修订方案和真实数据验收计划。 测试基准固定为： - target species: `Lc,Dl` - reference species: `At` - comparative species: `Os` - family: `SPL` - real data job: `job_1779115921147_d4e0a7` - primary input dir: `input/job_1779115921147_d4e0a7` - primary workspace: `workspace/job_1779115921147_d4e0a7` ## 1. 总体结论 当前系统仍然以 `prefix` 作为几乎唯一的分析身份，例如 `Multi.SPL`、`Lc.SPL`、`At.SPL`。这在单物种串行分析中能工作，但在多物种、target/comparative/reference 混合分析中不够： - `prefix` 只能表达一个字符串，不能表达本次分析使用了哪些 target、reference、comparative。 - 多个不同选择范围会写入同一批 `final/{prefix}.*` 文件。 - 下游阶段多数只读 `final/{prefix}.final_members.*`、`final/{prefix}.tree.nwk`，不知道这些文件对应哪个物种组合。 - 部分阶段会从“最新 manifest”或固定中间目录取文件，容易串用旧结果。 - 前端弹窗只在 `identify` 和 `interspecies_synteny` 上部分支持物种选择，其他阶段没有统一选择范围。 因此需要引入统一的 `scope` 作为一等概念。`prefix` 保留为兼容输出前缀，`scope` 表达真实分析范围。 建议 scope 字段： ```json { "family": "SPL", "prefix": "Multi.SPL", "scope_id": "SPL.target-Lc-Dl.ref-At.comp-Os", "target_species": ["Lc", "Dl"], "reference_species": ["At"], "comparative_species": ["Os"], "selected_species": ["Lc", "Dl", "At", "Os"], "stage_scope": "target_plus_reference_plus_comparative" } ``` ## 2. 代码链路通读记录 ### 2.1 前端入口与弹窗 主要文件：`apps/web/vue3/index.html` 当前直接逻辑： - 参数确认弹窗在 `index.html:1718` 附近定义。 - basic parameters 仍展示 `confirmParams.prefix`，`index.html:1730`。 - `identify` 独有 target/comparative/reference 展示在 `index.html:1803` 附近，但实际选择都写入 `confirmParams.targetSpecies`。 - `reference_classification` 只提供 `referenceTargetPrefix` 和 `referencePrefix`，`index.html:1889` 附近，没有 target/ref 物种勾选。 - 批量执行弹窗的 identify 物种选择在 `index.html:2448` 附近，仍只针对 identify。 - `showConfirm()` 在 `index.html:5149` 附近将 `confirmParams.prefix = normalizedPrefix(params.prefix)`。 - 只有 `interspecies_synteny` 调用 `loadInterspeciesSpeciesOptions()`，`index.html:5125` 附近。 - 单阶段执行时，`executeSingleStageWithParams()` 只对 `identify` 传 `targetSpecies`，只对 `interspecies_synteny` 传 `targetSpecies/refSpeciesList`，`index.html:5809-5877`。 - 普通执行 `doExecute()` 会把大量参数统一发送，包括 `targetSpecies/refSpeciesList`，`index.html:6343-6404`，但后端和 CLI 不一定消费。 - agent plan 展示仍打印 `Prefix`，`index.html:7668`。 - direct execution 只对 `identify` 和 `reference_classification` 做少量参数分支，`index.html:7768-7794`。 - 自动倒计时默认 15 秒，`index.html:5191`，批量倒计时也单独维护。 关联代码： - `/api/execute` 请求进入 `apps/web/vue3/main.py`。 - `/api/agent/plan` 的返回被 `runPlanForIntent()` 消费，再进入批量执行。 - `/api/stage-status` 只按 `job_id/stage/prefix` 查状态，没有 scope。 当前风险： - 用户选择 target/ref/comparative 后，只有少数阶段收到选择。 - `reference_classification/motif/domain/structure/combo` 仍按 prefix 读全局文件。 - 前端让用户理解和填写 prefix，用户负担过高。 - 倒计时太短且没有统一暂停机制。 ### 2.2 Web API 与任务执行 主要文件：`apps/web/vue3/main.py` 当前直接逻辑： - `ExecuteRequest` 定义 `prefix` 默认 `"Test"`，并有 `targetSpecies/refSpeciesList`，`main.py:1353-1378`。 - `run_stage_sync()` 解析 effective prefix 后调用 `pipeline_backend.run_stage()`，`main.py:1604-1711`。 - `targetSpecies` 只作为参数传给 backend；是否生效由 backend 的 stage 分支决定。 - `_load_identify_species_selection()` 从 config/input 推断 identify 可选物种，`main.py:230`。 - `_infer_family_from_config_or_final()` 和 `_apply_inferred_family_to_intent()` 尝试从 config/final 推断 family 和 prefix，`main.py:157-227`。 - `/api/execute` 会先查 stage status，仍然以 `prefix` 为粒度，`main.py:2505-2534`。 - `/api/stage-status` 调用 `get_unified_stage_status(job_id,prefix,stage)`，`main.py:3060-3066`。 - `parse_natural_language()` 仍从文本里抽 prefix 或构造 `species.family`，`main.py:3344-3526`。 - agent chat 中 explicit species 会强制改写 `intent["prefix"] = f"{species}.{family}"`，`main.py:3769`，多物种会变成 `Multi.family`。 - agent plan 阶段只在含 `identify` 且没有 targetSpecies 时补 `identifySpeciesSelection`，`main.py:4535-4562`。 关联代码： - 实际 CLI 调度在 `apps/web/pipeline_backend.py`。 - 状态判断在 `gfa_core/mas/tools/stage_status.py`。 - 多智能体 stage 工具在 `gfa_core/mas/tools/stage_tools.py`。 当前风险： - API 层没有统一 `scope` 请求模型。 - effective prefix 会从历史 manifest/final 自动推断，可能掩盖用户本次选择。 - status 只看 prefix，无法判断同一个 prefix 下不同物种组合是否已经完成。 ### 2.3 后端 CLI 调度 主要文件：`apps/web/pipeline_backend.py` 当前直接逻辑： - `run_stage()` 是所有 stage 的核心后端分发函数。 - `identify` 分支调用 `run_identify.py`，并传 `--target-species`，`pipeline_backend.py:1149` 附近。 - `phylogeny` 分支优先使用 `final/{prefix}.final_members.pep.fa`，并会去除蛋白末尾 `*`，`pipeline_backend.py:1257-1339`。 - `reference_classification` 分支固定读 `final/{prefix}.tree.nwk`，并用 `reference_target_prefix or prefix.split(".")[0]`，`pipeline_backend.py:1366-1400`。 - `structure` 分支固定检查 `final/{prefix}.tree.nwk`，并调用 `run_structure.py --prefix {prefix}`，`pipeline_backend.py:1422-1444`。 - `domain` 分支传 `--proteins final/{prefix}.final_members.pep.fa`，`pipeline_backend.py:1450-1483`。 - `motif` 分支传 `--fasta final/{prefix}.final_members.pep.fa`，`pipeline_backend.py:1511-1534`。 - `pep/promoter/synteny/karyotype/expression/ppi/kaks/combo` 也基本围绕 `{prefix}` 拼文件。 - `combo` 分支只检查 `final/{prefix}.{stage}.manifest.json` 是否存在，但 `run_combo.py` 内部仍可能找最新 manifest，`pipeline_backend.py:3030-3059`。 - `genome_prepare` 分支可以跑多物种准备，`pipeline_backend.py:3706` 之后。 关联代码： - 传入参数来自 `main.py:run_stage_sync()` 和前端 `index.html`。 - 每个 CLI 自己还会再次自动发现文件和 manifest。 当前风险： - backend 是 scope 断开的主要位置：参数收到了，但大多数 stage 没有用。 - 多数 stage 默认读 canonical 文件，导致同 prefix 不同分析范围互相覆盖。 - `reference_classification` 用字符串 target prefix，不足以表达 Lc+Dl 双目标。 ### 2.4 意图识别、stage catalog、多智能体工具层 主要文件： - `gfa_core/mas/core/fast_intent.py` - `gfa_core/mas/stage_catalog.py` - `gfa_core/mas/tools/stage_tools.py` 当前直接逻辑： - `fast_intent.py` 用 `ANALYSIS_KEYWORDS` 判定阶段，`fast_intent.py:9`。 - 物种识别是简单 species code list，`fast_intent.py` 中 `SPECIES_KEYWORDS`。 - 返回字段主要是 `stage/stages/gene_family/species/prefix`，`fast_intent.py:86-362`。 - `stage_catalog.py` 定义 `STAGE_DEPENDENCIES`，如 `combo` 依赖 `identify/phylogeny/motif/structure/domain`，`stage_catalog.py:60`。 - `complete_dependencies()` 只按 stage 补依赖，不关心 scope，`stage_catalog.py:382`。 - `StageToolExecutor.execute()` 调用 `pipeline_backend.run_stage()`，并只传 `targetSpecies/refSpeciesList` 等已有参数，`stage_tools.py:139-245`。 关联代码： - agent plan 使用这些 intent 和 stage catalog 输出计划。 - frontend `runPlanForIntent()` 消费 plan 并发起 batch execute。 当前风险： - 多智能体计划没有 scope 字段，无法要求下游复用同一物种选择。 - 依赖完成判断无法区分不同 scope。 - repair/diagnosis agent 即使诊断出问题，也没有“修复 scope 输入/重建 stage input”的规范动作。 ### 2.5 状态、manifest 校验和复现 主要文件： - `gfa_core/mas/tools/stage_status.py` - `apps/cli/validate_manifest.py` - `gfa_core/reproducibility_v2.py` 当前直接逻辑： - `manifest_path_for()` 固定返回 `final/{prefix}.{stage}.manifest.json`，`stage_status.py:51`。 - `get_stage_status()` 读取该 manifest 并校验 artifact 是否存在，`stage_status.py:142`。 - `list_completed_stages()` glob `f"{prefix}.*.manifest.json"`，`stage_status.py:233-259`。 - `validate_manifest.py` 校验 JSON schema 和 artifact 文件存在，`validate_manifest.py:30-104`。 - `ReproducibilityManager` 只以 `job_id/prefix` 建立复现目录和命令记录，`reproducibility_v2.py:57-344`。 关联代码： - 前端 `/api/stage-status`、agent completed stage、batch 跳过逻辑依赖 stage_status。 - 每个 CLI 产出的 manifest 被 validate/status/repro 后续使用。 当前风险： - 有 manifest 不代表本次选择范围正确。 - artifact 存在不代表来自正确 target/ref/comparative。 - 不同 scope 下的复现命令会混在同一个 prefix 目录。 ### 2.6 契约与技能 主要目录： - `gfa_core/schemas/stages/` - `skills/` 当前直接逻辑： - identify contract 只要求 `final/{prefix}.final_members.id/fa`，没有 scope。 - phylogeny contract 固定输入 `final/{prefix}.final_members.pep.fa` 和 identify manifest。 - motif/domain/structure/combo contract 也固定 `prefix` 文件和固定中间路径。 - combo contract 明确写死 `motif/analysis/meme_out/meme.xml` 和 `domain/analysis/results.domtblout`。 - skills 文档和 run.py 多数仍以 `prefix` 作为外部接口。 当前风险： - 即使 CLI 修了，契约不改也会把后续开发拉回 prefix-only。 - combo contract 的固定中间路径会继续制造串结果。 ## 3. 各分析阶段现状与关联 ### 3.1 Identify 主要文件：`apps/cli/run_identify.py` 当前直接逻辑： - 从 prefix 解析 family，或从 `config.xlsx` Analysis sheet 读取 family，`run_identify.py:147`、`552`、`803`。 - `find_pep_files()` 能读取 config 并按 `target_species` 过滤，`run_identify.py:678-743`。 - CLI 参数支持 `--target-species`，`run_identify.py:1926`。 - main 中读取 `selected_target_species` 并传给 `find_pep_files()`，`run_identify.py:1995-1998`。 - 逐物种跑 HMM/BLAST 后合并成 `final/{effective_prefix}.final_members.pep.fa/id`，`run_identify.py:2235-2239`。 - manifest 中已有 `target_species` 和 per-species artifacts，`run_identify.py:2244-2304`。 - InterProScan 是非阻塞 domain check，不是最终成员筛选器，`run_identify.py:2239` 后。 关联代码： - backend `identify` 分支传 `--target-species`。 - phylogeny/motif/domain/structure 读取 identify 的 final members。 - identify_refine 会改写 final members，但没有 scope。 主要风险： - identify 已支持 target，但没有 reference/comparative 角色定义。 - merged final members 覆盖同 prefix 的上一次选择。 - manifest 没有 scope_id 和输入 hash，不足以被下游校验。 ### 3.2 Phylogeny 主要文件：`apps/cli/run_phylogeny.py` 当前直接逻辑： - 默认读取 `final/{prefix}.identify.manifest.json`，`run_phylogeny.py:177-215`。 - 从 identify manifest 中找 `final_proteins/final_members/proteins`。 - 输出 `final/{prefix}.pep.msa.fa`、`final/{prefix}.tree.nwk`、manifest，`run_phylogeny.py:902-973`。 - CLI 有 `--proteins` override，`run_phylogeny.py:1003-1005`。 - tree plotting 的 subfamily 查找与 reference_classification 输出不完全统一。 关联代码： - reference_classification、motif、domain、structure、combo 都读 tree。 主要风险： - 没有 stage-level selection，无法表达 target-only、target+ref、target+comp 等不同树。 - 同 prefix 不同树会覆盖 `final/{prefix}.tree.nwk`。 ### 3.3 Reference Classification 主要文件：`apps/cli/run_reference_classification.py` 当前直接逻辑： - 目标树默认 `final/{prefix}.tree.nwk`。 - 如树中没有参考 tip，会用 target proteins + reference proteins 构建 helper joint tree。 - target 判断依赖 `--target-prefix`，reference 判断依赖 `--reference-prefix`。 - 输出 `final/{prefix}.reference_classification.*` 和兼容 `{prefix}.geneID.subfamily`。 关联代码： - motif/domain/structure 优先读取 `reference_classification.subfamily.tsv`。 - combo 应使用 reference_classification 的 subfamily。 主要风险： - 多目标物种 `Lc,Dl` 无法用单个 `target_prefix` 准确表达。 - 选择一个物种做分类时，现有逻辑可能仍用全体 tree。 - 没有 scope 校验，可能用不含 At 的树或旧 helper tree。 ### 3.4 Motif 主要文件：`apps/cli/run_motif.py` 当前直接逻辑： - `prepare_directories()` 固定 `motif/analysis/meme_out`，`run_motif.py:57-61`。 - 默认从 identify manifest 或 `final/{prefix}.final_members.pep.fa` 取蛋白，`run_motif.py:128-167`。 - `run_meme()` 检查 `meme_out/meme.xml` 存在就复用，`run_motif.py:229-322`。 - 输出 final motif 图、seqlogo、report、manifest。 关联代码： - combo 固定找 `motif/analysis/meme_out/meme.xml`。 主要风险： - 不同 scope 复用同一个 meme.xml。 - 不同参数或 FASTA 改变时没有严格 hash 判断。 ### 3.5 Domain 主要文件：`apps/cli/run_domain.py` 当前直接逻辑： - `find_latest_manifest(final_dir, stage)` 只按 `*.{stage}.manifest.json` 找最新，不带 prefix，`run_domain.py:108-114`。 - 默认从 identify/phylogeny latest manifest 或 `final/{prefix}.*` 取输入。 - `run_hmmscan()` 固定写 `domain/analysis/results.domtblout`，`run_domain.py:239-286`。 - manifest artifacts 记录 domtblout 路径，`run_domain.py:856-864`。 关联代码： - combo 从 domain manifest 读取 domtblout。 主要风险： - latest manifest 可能跨 prefix。 - domtblout 固定路径会被覆盖。 - 没有 scope 一致性。 ### 3.6 Structure 主要文件：`apps/cli/run_structure.py` 当前直接逻辑： - 默认 gene ID 文件为 `final/{prefix}.final_members.id`，`run_structure.py:606`。 - 默认 tree 为 `final/{prefix}.tree.nwk`。 - GFF3 自动解析为 `{species_short}_final.gff3`，其中 species_short 来自 prefix，`run_structure.py:76`。 - 优先读取 `final/{prefix}.reference_classification.subfamily.tsv`，`run_structure.py:141-168`。 - 输出 `structure/analysis/{prefix}.structure.filtered.gff3`、`{prefix}.structure.tsv` 和 final 图。 关联代码： - combo 读取 structure manifest 中的 filtered gff3。 主要风险： - `Multi.SPL` 无法映射到 `Lc_final.gff3` 和 `Dl_final.gff3`。 - 多物种结构需要 per-species GFF 过滤再合并或分图。 - tree tip、gene ID、GFF ID 没有统一校验。 ### 3.7 Combo 主要文件：`apps/cli/run_combo.py` 当前直接逻辑： - `find_meme_xml()` 优先固定 `motif/analysis/meme_out/meme.xml`，`run_combo.py:36-39`。 - `find_latest_manifest(workspace, stage, prefix)` 实际 glob `*{stage}*.manifest.json`，忽略 prefix，`run_combo.py:56-62`。 - phylogeny/structure/domain 都通过 latest manifest 获取，`run_combo.py:166-224`。 - domtblout 从 domain manifest artifact 获取。 - 输出 `final/{prefix}.combo.manifest.json`。 关联代码： - backend combo 只检查 prefix manifest 存在，然后调用 CLI。 主要风险： - 这是当前最容易串结果的阶段。 - 即使 backend 检查 prefix manifest，CLI 仍可能读取别的最新 manifest。 - meme XML 仍是固定目录。 ### 3.8 下游阶段概览 需要后续逐个迁移： - `pep`: 读 `final/{prefix}.final_members.pep.fa`，应改为 scope FASTA。 - `promoter`: 读单物种 genome/GFF/gene ID，scope 多物种时需 per-species。 - `synteny`: 偏单物种，backend 当前 gene ID 查找不够稳。 - `interspecies_synteny`: 已经有较好的 target/ref 物种选择和 config groups，可作为迁移模板。 - `karyotype`: 已支持按 config 物种循环，但 family ID 仍来自全局 final members。 - `expression`: 目前从 input/final glob gene ID，容易拿错；表达应明确单物种或显式选择表达物种。 - `ppi/kaks/ortholog`: 当前多依赖 prefix 和 interspecies outputs，需要等 scope 底座后再改。 ## 4. 真实测试数据现状 已确认可用文件： - `input/job_1779115921147_d4e0a7/config.xlsx` - `input/job_1779115921147_d4e0a7/Lc.pep.fasta` - `input/job_1779115921147_d4e0a7/Dl.pep.fasta` - `input/job_1779115921147_d4e0a7/Os.pep.fasta` - `input/job_1779115921147_d4e0a7/At.SPL.pep.fa` - `input/job_1779115921147_d4e0a7/At.SPL.subfamily` - `input/job_1779115921147_d4e0a7/Lc_final.gff3` - `input/job_1779115921147_d4e0a7/Dl_final.gff3` - `input/job_1779115921147_d4e0a7/Os_final.gff3` - `input/job_1779115921147_d4e0a7/Ref_combined.SPL.pep.fasta` - `workspace/job_1779022508209_52539d/final/At.SPL.ncbi_fetch.protein.fa` 已有历史结果： - `workspace/job_1779115921147_d4e0a7/final/Multi.SPL.genome_prepare.manifest.json` - `workspace/job_1779115921147_d4e0a7/final/Target.SPL.identify.manifest.json` - `workspace/job_1779115921147_d4e0a7/final/Lc.identify.manifest.json` 后续真实测试不能只检查编译，必须检查： - CLI exit code。 - manifest `status == success`。 - manifest `scope_id` 与本次选择一致。 - 输入 FASTA、tree tips、gene IDs、GFF IDs 的集合一致。 - target/reference/comparative 是否按本阶段定义参与。 - 不同 scope 的中间目录不互相覆盖。 - 故意制造 scope 不一致时，stage 必须拒绝继续。 ## 5. 修订总体原则 1. `prefix` 只作为兼容输出前缀，不再承载完整分析范围。 2. 新增 `scope` 作为 stage 输入和 manifest 必填字段。 3. 每个阶段只读同一 scope 的上游 manifest。 4. 中间目录必须 scope 隔离。 5. 兼容旧文件名，但新逻辑优先读 scope manifest。 6. 一个阶段改完并通过真实数据测试后，再改下一个阶段。 7. 前端使用原有 Element Plus 弹窗技术，不换新框架。 8. config.xlsx 的 `Analysis!Family` 是 family 的优先来源；没有则让用户填写。 9. 默认示例使用 `At.WRKY`，不再全局默认 `Ft.SPL` 或 `Multi.SPL`。 ## 6. 分阶段修订计划 ### Phase 0: Scope 底座 目标： - 建立统一 scope 解析、写入、校验能力。 - 不改变阶段生物学结果，只先让所有阶段能接收/记录 scope。 拟修改文件： - 新增 `gfa_core/gene_family_scope.py` - 修改 `apps/web/vue3/main.py` - 修改 `apps/web/pipeline_backend.py` - 修改 `gfa_core/mas/tools/stage_status.py` - 修改 `gfa_core/reproducibility_v2.py` - 修改 `gfa_core/schemas/core/workspace_contract_v1.md` 核心能力： - 从 `config.xlsx` 读取 family 和物种角色。 - 生成 `scope_id`。 - 生成 stage input manifest。 - 校验 upstream manifest 的 scope 是否匹配。 - 保留 canonical manifest，同时写 scope manifest。 真实测试： - 在 `job_1779115921147_d4e0a7` 解析出 family `SPL`。 - 解析出 target `Lc,Dl`、reference `At`、comparative `Os`。 - 生成 scope JSON 后，检查所有文件存在且 hash 可计算。 通过标准： - scope JSON 非空。 - species roles 正确。 - `prefix` 自动生成为 `Multi.SPL` 或由兼容逻辑给出，但不要求用户理解 prefix。 ### Phase 1: Identify 目标： - identify 只鉴定用户选择的 target species。 - manifest 写入 scope。 - 输出 per-species artifacts 和 scope artifacts。 拟修改文件： - `apps/cli/run_identify.py` - `apps/web/pipeline_backend.py` - `apps/web/vue3/index.html` - `gfa_core/schemas/stages/identify_contract_v1.md` - `skills/gene-identify/SKILL.md` 真实测试命令形态： ```bash /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_identify.py \ --input-dir input/job_1779115921147_d4e0a7 \ --workspace workspace/ \ --prefix Multi.SPL \ --target-species Lc,Dl \ --required-pfams PF03110 \ --cpu 4 ``` 通过标准： - final members 只包含 `Lc,Dl`。 - 不包含 `At` 和 `Os`。 - `PF03110` 被记录到 parameters。 - manifest 含 `scope_id`、`target_species=["Lc","Dl"]`。 ### Phase 2: Phylogeny 目标： - 支持用户选择建树范围。 - tree/msa 文件和 manifest 记录 scope。 - 不同建树范围不互相覆盖。 拟修改文件： - `apps/cli/run_phylogeny.py` - `apps/web/pipeline_backend.py` - `apps/web/vue3/index.html` - `gfa_core/schemas/stages/phylogeny_contract_v1.md` - `skills/phylogeny/SKILL.md` 测试矩阵： - target only: `Lc,Dl` - target + reference: `Lc,Dl,At` - target + comparative + reference: `Lc,Dl,Os,At` 通过标准： - tree tips 数量等于 scope FASTA 记录数。 - manifest 的 upstream identify scope 匹配。 - 不同 scope 输出在不同中间目录。 ### Phase 3: Reference Classification 目标： - 由 selected target species + reference species 驱动分类。 - 支持 `Lc,Dl` 多目标物种。 - At 只作为参考，不写入目标 assignment。 拟修改文件： - `apps/cli/run_reference_classification.py` - `apps/web/pipeline_backend.py` - `apps/web/vue3/index.html` - `gfa_core/schemas/stages/reference_classification_contract_v1.md` 通过标准： - assignment/subfamily 目标行只包含 `Lc,Dl`。 - reference evidence 来自 `At.SPL.subfamily`。 - 如果 tree scope 不含 At 且需要 At，必须构建 helper tree 并记录。 ### Phase 4: Motif 目标： - MEME 输出目录 scope 隔离。 - 复用 MEME 前校验 FASTA hash、参数、scope_id。 拟修改文件： - `apps/cli/run_motif.py` - `apps/cli/run_combo.py` - `gfa_core/schemas/stages/motif_contract_v1.md` - `skills/motif-analysis/SKILL.md` 通过标准： - `Lc,Dl` 和 `Lc` 两次运行生成不同 meme_out。 - 修改 scope 后不会复用旧 `meme.xml`。 ### Phase 5: Domain 目标： - hmmscan 输出 scope 隔离。 - 去除 latest manifest 跨 prefix 风险。 拟修改文件： - `apps/cli/run_domain.py` - `apps/cli/run_combo.py` - `gfa_core/schemas/stages/domain_contract_v1.md` - `skills/domain-analysis/SKILL.md` 通过标准： - `domain/analysis/{scope_id}/results.domtblout` 存在。 - manifest domtblout path 指向 scope 目录。 - 故意放入其他 prefix manifest 时不会被读取。 ### Phase 6: Structure 目标： - 多目标物种时按物种 GFF 过滤，再合并绘图输入或输出分物种结构图。 - tree tips、gene ID、GFF gene ID 做一致性检查。 拟修改文件： - `apps/cli/run_structure.py` - `apps/web/pipeline_backend.py` - `gfa_core/schemas/stages/structure_contract_v1.md` - `skills/structure/SKILL.md` 通过标准： - Lc/Dl 的结构分别从 `Lc_final.gff3`、`Dl_final.gff3` 解析。 - 缺失 ID 报告到 metrics，不静默失败。 ### Phase 7: Combo 目标： - 只合并同一 scope 的 phylogeny/motif/domain/structure。 - 不再从固定 `meme_out` 或 latest manifest 取文件。 拟修改文件： - `apps/cli/run_combo.py` - `apps/web/pipeline_backend.py` - `gfa_core/schemas/stages/combo_contract_v1.md` 通过标准： - 同 scope 正常出组合图。 - scope 不一致时明确失败并列出不一致 stage。 ### Phase 8: 下游阶段 迁移顺序： 1. `pep` 2. `promoter` 3. `synteny` 4. `karyotype` 5. `interspecies_synteny` 6. `expression` 7. `ortholog` 8. `ppi` 9. `kaks` 10. `summary` 每个阶段独立改、独立真实测试。 ## 7. 前端统一方案 目标： - prefix 不再作为主输入。 - 弹窗使用原有 Element Plus dialog/form/checkbox 技术。 - 所有基因家族阶段都显示统一的 species role selector。 - family 从 config.xlsx 自动填；没有则留空让用户填。 - 倒计时全局 40 秒，并增加暂停按钮。 前端字段： - `confirmParams.family` - `confirmParams.scopeMode` - `confirmParams.targetSpecies` - `confirmParams.referenceSpecies` - `confirmParams.comparativeSpecies` - `confirmParams.prefixAdvanced` - `confirmParams.countdownPaused` scopeMode 建议选项： - `target_only` - `target_plus_reference` - `target_plus_comparative` - `target_plus_reference_plus_comparative` - `custom` ## 8. 诊断/修复 agent 后续定位 当前诊断 agent 的主要问题： - 它看 log 和 manifest，但没有 stage-specific 可执行修复策略。 - 它不能主动修代码。 - 它没有把这次修订逻辑固化成规则。 后续需要给诊断 agent 增加“可修复动作库”： - `scope_mismatch`: 建议重新生成 stage scope input，或阻止下游执行。 - `wrong_manifest_scope`: 指向正确 manifest 或要求重跑上游。 - `fixed_intermediate_collision`: 清理/隔离旧中间目录后重跑。 - `missing_reference_family_pep`: 从 NCBI fetch 结果或 `At.SPL.pep.fa` 建立兼容软链接。 - `protein_stop_marker`: 调用已有 clean fasta 逻辑。 - `id_mismatch`: 运行 ID normalize/映射检查，不直接猜测替换。 这些动作要写入代码和文档，不能只停留在聊天说明。 ## 9. 逐项执行记录 后续每完成一个阶段，在这里追加： ```text Phase: Modified files: Real test command: Real test data: Result: Manifest: Artifacts: Known residual risks: Next phase gate: ``` 当前状态： - 文档创建完成。 - Phase 0 scope 解析模块已开始落地。 ```text Phase: 0 Scope 底座 Modified files: - gfa_core/gene_family_scope.py - apps/cli/resolve_scope.py - docs/gene_family_scope_refactor_plan.md Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/gene_family_scope.py apps/cli/resolve_scope.py - /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/resolve_scope.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/job_1779115921147_d4e0a7 --prefix Multi.SPL --family SPL --target-species Lc,Dl --reference-species At --comparative-species Os --scope-mode target_plus_reference_plus_comparative Real test data: - input/job_1779115921147_d4e0a7/config.xlsx - input/job_1779115921147_d4e0a7/At.SPL.pep.fa - Lc/Dl/Os pep/gff/genome rows from config.xlsx Result: - scope_id = SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference_plus_comparative - family = SPL - target_species = Lc,Dl - reference_species = At - comparative_species = Os - selected_species = Lc,Dl,At,Os Artifacts: - workspace/job_1779115921147_d4e0a7/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference_plus_comparative.scope.json Known residual risks: - At whole-genome pep/cds rows did not resolve in this job, but At.SPL family_pep resolved and is sufficient for reference classification. - Existing stages do not yet consume scope JSON. Next phase gate: - Identify must accept --scope-file, default target species from scope, and write scope fields into manifest. ``` ```text Phase: 1 Identify scope 接入 Modified files: - apps/cli/run_identify.py - apps/web/pipeline_backend.py - apps/web/vue3/main.py - gfa_core/mas/tools/stage_tools.py - docs/gene_family_scope_refactor_plan.md Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_identify.py apps/web/pipeline_backend.py apps/web/vue3/main.py gfa_core/mas/tools/stage_tools.py - /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/resolve_scope.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/codex_scope_identify_test2 --prefix Multi.SPL --family SPL --target-species Lc,Dl --reference-species At --comparative-species Os --scope-mode target_only --output workspace/codex_scope_identify_test2/final/Multi.SPL.identify_scope.json - /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_identify.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/codex_scope_identify_test2 --prefix Multi.SPL --scope-file workspace/codex_scope_identify_test2/final/Multi.SPL.identify_scope.json --required-pfams PF03110 --cpu 4 --skip-ref-warning - Direct backend run_stage('identify') with scope_file in workspace/codex_scope_backend_identify_test. Real test data: - target: Lc,Dl - reference: At - comparative: Os - family: SPL / PF03110 Result: - CLI identify status = success - CLI target_species = Lc,Dl - CLI selected_species = Lc,Dl - CLI final members = 55 total; Lc=33, Dl=22, At-like=0, Os-like=0 - CLI InterProScan status = success - Backend identify status = success - Backend final members = 59 total; Lc=37, Dl=22, At-like=0, Os-like=0 - Backend/CLI count difference is explained by parameter difference: backend default blast_evalue is 1e-5, CLI test used 1e-7. Artifacts: - workspace/codex_scope_identify_test2/final/Multi.SPL.identify.manifest.json - workspace/codex_scope_identify_test2/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_only.identify.manifest.json - workspace/codex_scope_identify_test2/final/Multi.SPL.final_members.id - workspace/codex_scope_identify_test2/final/Multi.SPL.final_members.pep.fa - workspace/codex_scope_backend_identify_test/final/Multi.SPL.identify.manifest.json Known residual risks: - Frontend does not yet generate scope JSON automatically. - stage_status still checks prefix manifest only; scope-aware completed checks are not done. - Downstream stages still ignore scope. Next phase gate: - Phylogeny must accept scope/stage FASTA and ensure tree tips match selected species. ``` ```text Phase: 2 Phylogeny scope 接入 Modified files: - apps/cli/run_phylogeny.py - apps/web/pipeline_backend.py - docs/gene_family_scope_refactor_plan.md Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_phylogeny.py apps/web/pipeline_backend.py - target-only CLI: /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_phylogeny.py --workspace workspace/codex_scope_phy_target_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_target_fast_test/final/Multi.SPL.phylogeny_target_scope.json --cpu 4 --iqtree-params '-m LG+G -nt 4' --tree-layout rectangular - target+reference CLI: /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_phylogeny.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --cpu 4 --iqtree-params '-m LG+G -nt 4' --tree-layout rectangular - backend run_stage('phylogeny') with scope_file: workspace/codex_scope_backend_phy_test Real test data: - target-only: Lc,Dl SPL members from real identify output - target+reference: Lc,Dl SPL members + At.SPL.pep.fa Result: - target-only CLI status = success - target-only CLI scope counts = Lc 33, Dl 22 - target-only CLI scope_sequence_count = 55 - target-only CLI MSA count = 55 - target-only CLI tree tips = 55 - target+reference CLI status = success - target+reference CLI scope counts = Lc 33, Dl 22, At 17 - target+reference CLI scope_sequence_count = 72 - target+reference CLI MSA count = 72 - target+reference CLI tree tips = 72 - backend phylogeny status = success - backend phylogeny scope counts = Lc 33, Dl 22 - backend phylogeny MSA/tree tips = 55/55 Artifacts: - workspace/codex_scope_phy_target_fast_test/final/Multi.SPL.phylogeny.manifest.json - workspace/codex_scope_phy_target_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_only.phylogeny.manifest.json - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny.manifest.json - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.phylogeny.manifest.json - workspace/codex_scope_backend_phy_test/final/Multi.SPL.phylogeny.manifest.json Known residual risks: - Frontend does not yet generate scope JSON automatically. - target+comparative+reference requires Os family members. Current behavior should fail clearly unless Os has already been identified or Species.Family.pep.fa is provided. - The backend still performs stop-marker cleanup on canonical final_members, while scope FASTA currently uses per-species artifacts directly. A later shared FASTA sanitizer should be applied to scope-built FASTA before all sequence stages. - MFP model selection on target+reference took too long in a validation run and was stopped; fixed-model LG+G was used for scope verification. The production default remains configurable. Next phase gate: - Reference classification must consume selected target/reference species from scope and must not classify At as target. ``` ```text Phase: 3 Reference classification scope 接入 Modified files: - apps/cli/run_reference_classification.py - apps/web/pipeline_backend.py - docs/gene_family_scope_refactor_plan.md Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_reference_classification.py apps/web/pipeline_backend.py - /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_reference_classification.py --workspace workspace/codex_scope_phy_ref_fast_test --input-dir input/job_1779115921147_d4e0a7 --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --reference-subfamily input/job_1779115921147_d4e0a7/At.SPL.subfamily --reference-prefix At --overwrite-gene-subfamily - Direct backend run_stage('reference_classification') with the same scope_file/reference_subfamily_file. Real test data: - Joint tree from Phase 2 target+reference: Lc,Dl + At, 72 tips - Reference subfamily table: input/job_1779115921147_d4e0a7/At.SPL.subfamily Result: - CLI reference_classification status = success - Backend reference_classification status = success - assignment rows = 55 - assignment target composition = Lc 33, Dl 22, At 0 - subfamily rows = 72, retaining At 17 reference rows plus Lc/Dl rows for joint-tree plotting - manifest scope_id = SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference - manifest target_species = Lc,Dl; reference_species = At; comparative_species = Os - scoped manifest exists. Artifacts: - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.assignment.tsv - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.subfamily.tsv - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.confidence.tsv - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.manifest.json - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.reference_classification.manifest.json Known residual risks: - The internal tree script still accepts a single target_prefix and reference_prefix; the wrapper now filters assignment rows by scope target IDs, but a future cleanup should move scope awareness into scripts/reference_classification_from_tree.py directly. - The compatibility file final/Multi.SPL.geneID.subfamily intentionally includes At reference rows when the classification scope includes reference species. Downstream plotting stages must use scope-aware filtering if they need target-only plots. - Frontend still needs to generate/pass scope JSON from the target/comparative/reference checklist automatically. Next phase gate: - Structure/motif/domain/combo must consume scope/subfamily consistently and reuse the scoped reference classification outputs without rerunning unrelated species. ``` ```text Phase: 4 Structure scope 接入 Modified files: - apps/cli/run_structure.py - apps/web/pipeline_backend.py - docs/gene_family_scope_refactor_plan.md Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_structure.py apps/web/pipeline_backend.py - /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_structure.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --subfamily workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.subfamily.tsv --cpu 4 - Direct backend run_stage('structure') with the same scope_file. Real test data: - target GFF3 from scope: Lc=Lchinesis_genome.Chr.gff3.gz, Dl=Longan.v20190918.gff3.gz - target IDs: final/Multi.SPL.final_members.id, 55 IDs - tree/subfamily: Lc,Dl + At reference classification from Phase 2/3 Result: - CLI structure status = success - Backend structure status = success - gzip GFF3 input is supported - scanned GFF3 lines = 1,260,473 - filtered GFF3 rows = 353 - genes_with_structure = 55/55 - structure TSV ID check: final_ids=55, tsv_ids=55, missing=0, extra=0 - species composition in structure TSV: Lc 33, Dl 22 - structure PNG/PDF generated. Artifacts: - workspace/codex_scope_phy_ref_fast_test/structure/analysis/Multi.SPL.structure.filtered.gff3 - workspace/codex_scope_phy_ref_fast_test/structure/analysis/Multi.SPL.structure.tsv - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.structure.png - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.structure.pdf - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.structure.manifest.json - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.structure.manifest.json Known residual risks: - The structure plot uses a joint 72-tip tree but structure features only exist for target species Lc/Dl. This is intentional for target structure visualization after reference-guided classification, but downstream combo needs explicit scope-aware panel rules. - run_structure.py still writes the first GFF3 path in the legacy cmd field for compatibility; the authoritative source list is parameters.gff3_sources. Next phase gate: - Motif must use scoped target proteins and scoped subfamily consistently, and must not silently include reference/comparative proteins unless the selected scope requests them. ``` ```text Phase: 5 Motif scope 接入 Modified files: - apps/cli/run_motif.py - apps/web/pipeline_backend.py - docs/gene_family_scope_refactor_plan.md Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_motif.py apps/web/pipeline_backend.py - /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_motif.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --subfamily workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.subfamily.tsv --nmotifs 3 --minw 6 --maxw 100 --cpu 4 - Direct backend run_stage('motif') with the same scope_file and motif parameters. Real test data: - scoped protein FASTA from phylogeny: phylogeny/analysis/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.input.pep.fa - sequence composition: Lc,Dl target + At reference, 72 proteins - subfamily file: final/Multi.SPL.reference_classification.subfamily.tsv Result: - CLI motif status = success - Backend motif status = success - MEME ran on 72 scoped proteins, not target-only final_members - motif_count = 3 - total_sites = 109 - motif plot PNG/PDF generated - scoped manifest exists - MEME now records input FASTA hash and motif parameters in motif/analysis/meme_out/run_signature.json; existing results are reused only when the signature matches. Artifacts: - workspace/codex_scope_phy_ref_fast_test/motif/analysis/meme_out/meme.xml - workspace/codex_scope_phy_ref_fast_test/motif/analysis/meme_out/run_signature.json - workspace/codex_scope_phy_ref_fast_test/motif/analysis/Multi_SPL.motif_prot.txt - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.motif.png - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.motif.pdf - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.motif.manifest.json - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.motif.manifest.json Known residual risks: - The motif analysis directory is still the legacy shared path motif/analysis. The new MEME signature prevents stale MEME reuse, but future multi-scope parallel execution should use per-scope analysis subdirectories. - If MEME reruns, the summary file is now regenerated; older outputs from pre-signature runs may be replaced during the first post-fix execution. Next phase gate: - Domain must use the same scoped protein FASTA/subfamily rules as motif and should avoid stale hmmscan outputs when scope or thresholds change. ``` ```text Phase: 6 Domain scope 接入 Modified files: - apps/cli/run_domain.py - apps/web/pipeline_backend.py - docs/gene_family_scope_refactor_plan.md Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_domain.py apps/web/pipeline_backend.py - /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_domain.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --subfamily workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.subfamily.tsv --evalue 1e-5 --cpu 4 - Direct backend run_stage('domain') with the same scope_file. Real test data: - scoped protein FASTA from phylogeny: 72 proteins, Lc/Dl target + At reference - Pfam database: /home/xia-lab/00.share_data/hmm/Pfam-A.hmm Result: - CLI domain status = success - Backend domain status = success - hmmscan elapsed = 38.3 s in CLI test - proteins_total = 72 - proteins_with_domains = 63 - significant_domain_hits = 84 - unique_domain_types = 8 - top domain = SBP, 50 hits - domain PNG/PDF generated - scoped manifest exists. Artifacts: - workspace/codex_scope_phy_ref_fast_test/domain/analysis/results.domtblout - workspace/codex_scope_phy_ref_fast_test/domain/analysis/results.tblout - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.domain.png - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.domain.pdf - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.domain.manifest.json - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.domain.manifest.json Known residual risks: - hmmscan outputs still use the legacy shared domain/analysis/results.* filenames. This is safe for serial execution but not for two scopes running in parallel in the same workspace. - Domain stage does not cache by input signature; it reruns hmmscan, which avoids stale output but costs time. Next phase gate: - Combo must read scoped manifests/artifacts from phylogeny, motif, structure and domain instead of relying on the latest manifest glob alone. ``` ```text Phase: 7 Combo scope 接入 Modified files: - apps/cli/run_combo.py - apps/web/pipeline_backend.py - docs/gene_family_scope_refactor_plan.md Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_combo.py apps/web/pipeline_backend.py - /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_combo.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --subfamily workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.subfamily.tsv --evalue 1e-5 - Direct backend run_stage('combo') with the same scope_file. Real test data: - scoped phylogeny, motif, structure and domain manifests from Phases 2, 5, 4 and 6 - subfamily file from Phase 3 Result: - CLI combo status = success - Backend combo status = success - dependency check uses scoped manifests when scope_file is supplied - combo PDF/PNG generated - scoped combo manifest exists. Artifacts: - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.combo_tree_motif_structure_domain.pdf - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.combo_tree_motif_structure_domain.png - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.combo.manifest.json - workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.combo.manifest.json Known residual risks: - run_combo.py still finds meme.xml by legacy workspace search because motif manifest does not expose meme.xml as an artifact. The current test is correct for serial execution; a later cleanup should add meme_xml to motif artifacts and make combo consume it from the scoped motif manifest. - Combo output filename remains legacy prefix-based, so concurrent scopes in one workspace can overwrite each other. Scoped manifest records which scope produced the current plot. Next phase gate: - Frontend/router needs to generate/pass scope_file automatically from target/reference/comparative choices; status/diagnostic/repair should become scope-aware. ``` ```text Phase: 8 Frontend/API scope 自动传递与状态检查 Modified files: - apps/web/vue3/main.py - apps/web/vue3/index.html - gfa_core/mas/tools/stage_status.py - tasks.py - docs/gene_family_scope_refactor_plan.md Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/web/vue3/main.py gfa_core/mas/tools/stage_status.py tasks.py - Direct API helper: get_gene_family_scope_options(job_id='job_1779115921147_d4e0a7', prefix='Multi.SPL') - Direct API helper: _ensure_gene_family_scope_file(job_id='job_1779115921147_d4e0a7', prefix='Multi.SPL', stage='motif', target=Lc,Dl, reference=At, comparative=Os, scopeMode=target_plus_reference) - Direct /api/execute function call: execute_stage(ExecuteRequest(stage='phylogeny', job_id='codex_scope_phy_ref_fast_test', prefix='Multi.SPL', scopeFile='workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json')) - Direct /api/stage-status function call: get_stage_status(job_id='codex_scope_phy_ref_fast_test', stage='combo', prefix='Multi.SPL', scope_id='SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference') Real test data: - input/job_1779115921147_d4e0a7/config.xlsx - workspace/codex_scope_phy_ref_fast_test scoped manifests from Phases 2-7 Result: - /api/gene-family/scope-options returns family=SPL, target_species=Lc,Dl, reference_species=At, comparative_species from config. - _ensure_gene_family_scope_file writes final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.scope.json. - /api/execute with an existing scopeFile checks scoped phylogeny manifest and skips as already_completed instead of using the unscoped manifest. - /api/stage-status with scope_id returns completed=True for scoped combo. - Element Plus confirmation dialogs now expose analysis range as Target / Target+Reference / Target+Comparative / All, with checkboxes for target/reference/comparative species. - Countdown defaults are now 40 s and include a "暂停倒计时" button for normal, batch and per-stage confirmation dialogs. Known residual risks: - The browser UI was validated by code-path inspection and backend function tests, not by an automated Playwright click-through in a live browser. - Comparative species are listed from config; they are not auto-selected unless the selected scope mode includes comparative or upstream stage params specify them. - Natural-language router still needs deeper prompt/rule cleanup so it consistently sets scopeMode when the user says target/comparative/reference in free text. Next phase gate: - Diagnostic/repair agent should get explicit gene-family repair playbooks for ID mismatches, stop codons, missing/ambiguous GFF/PEP/HMM inputs, and scoped manifest/final mismatch. ``` ```text Phase: 9 Diagnostic/repair agent scope 接入与 stop-marker 真实修复 Modified files: - gfa_core/mas/agents/diagnosis.py - apps/cli/run_phylogeny.py - apps/cli/run_motif.py - apps/cli/run_domain.py - docs/gene_family_scope_refactor_plan.md Code relationship summary: - tasks.py calls DiagnosisAgent.diagnose_stage_gate() before and after StageToolRegistry execution. - diagnose_stage_gate() previously searched only final/{prefix}.{stage}.manifest.json or latest unscoped manifests, then checked legacy final_members files. - Scoped stages now pass params.scopeFile from frontend/API/backend; DiagnosisAgent reads scope_id from scopeFile and loads final/{prefix}.{scope_id}.{stage}.manifest.json first. - Protein-consuming stages now resolve protein input from scoped manifest parameters: protein_fasta, protein_source and scope_input_fasta. - GFF/member stages now read target GFF paths from scopeFile when available instead of guessing Multi_final.gff3 or the first final/*.gff3. - structure/combo no longer trigger protein stop-marker checks because they do not directly consume protein FASTA. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_motif.py apps/cli/run_domain.py apps/cli/run_phylogeny.py gfa_core/mas/agents/diagnosis.py - Diagnosis preflight repair: DiagnosisAgent.diagnose_stage_gate('codex_scope_phy_ref_fast_test','motif','Multi.SPL', gate='preflight', params={'scopeFile': 'workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json'}) DiagnosisAgent.apply_runtime_repairs(...) - /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_domain.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --cpu 4 - /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_motif.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --nmotifs 3 --minw 6 --maxw 100 --cpu 4 - Postflight gate for motif/domain/structure/combo with the same scopeFile. Real test data: - scope: SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference - protein input before repair: workspace/codex_scope_phy_ref_fast_test/phylogeny/analysis/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.input.pep.fa - protein input after repair: workspace/codex_scope_phy_ref_fast_test/phylogeny/analysis/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.input.pep.clean.fa Result: - Diagnosis preflight detected protein_fasta_stop_markers on the real scoped FASTA. - apply_runtime_repairs generated the clean scoped FASTA and a second preflight returned pass with no runtime repair candidates. - domain CLI used input.pep.clean.fa, completed hmmscan successfully, and produced 72 proteins / 84 significant hits / 8 unique domains. - motif CLI used input.pep.clean.fa, reran MEME because the input hash changed, produced 3 motifs / 109 sites, and generated PNG/PDF plus scoped manifest. - postflight diagnosis results: motif pass ok [] domain pass ok [] structure pass ok [] combo pass ok [] Behavioral fixes: - Scoped manifest/final mismatch no longer causes diagnosis to inspect stale unscoped manifests first. - Stop-marker repair is now job-local and actually affects downstream CLI inputs. - ID/GFF low-overlap is treated as a reportable job-local diagnosis; the agent can write a diagnosis report, but it does not silently rewrite biological IDs. - Plot-only combo no longer warns about empty metrics. Known residual risks: - The natural-language router still needs a separate pass so free text consistently maps to scopeMode and selected species. - ID mismatch repair remains intentionally conservative: automatic semantic ID rewriting needs an explicit mapping rule or user confirmation. - Motif/domain still write legacy shared analysis directories; serial real-data execution is validated, but parallel same-workspace multi-scope runs can still overwrite intermediate files. Next phase gate: - Router/plan/skill/contract cleanup should make scope explicit in natural-language planning, stage contracts and reproducibility records. ``` ```text Phase: 10 Router/plan/skill/contract/repro scope 清理 Modified files: - gfa_core/mas/utils/intent_parser.py - gfa_core/mas/core/fast_intent.py - apps/web/vue3/main.py - gfa_core/mas/tools/stage_tools.py - gfa_core/reproducibility_v2.py - apps/web/pipeline_backend.py - gfa_core/schemas/stages/identify_contract_v1.md - gfa_core/schemas/stages/phylogeny_contract_v1.md - gfa_core/schemas/stages/motif_contract_v1.md - gfa_core/schemas/stages/domain_contract_v1.md - gfa_core/schemas/stages/structure_contract_v1.md - gfa_core/schemas/stages/combo_contract_v1.md - skills/gene-identify/SKILL.md - skills/phylogeny/SKILL.md - skills/motif-analysis/SKILL.md - skills/domain-analysis/SKILL.md - gfa_core/mas/core/orchestrator.py - gfa_core/mas/agents/discovery.py - apps/web/vue3/static/conversation-sidebar.js - gfa_core/pipeline_api_integration.py Code relationship summary: - Natural-language parsing now emits targetSpecies/referenceSpecies/comparativeSpecies/scopeMode instead of only prefix/species. - /api/chat, /api/agent/chat and /api/agent/plan preserve these scope fields and put them into params for /api/execute. - _ensure_gene_family_scope_file can now use the parsed natural-language scope fields directly to write a scope JSON from config.xlsx. - Stage tool metadata now describes the real scoped inputs; structure uses final_members.id + target GFF3, not protein FASTA. - ReproducibilityManager now records scope_id, scope_file and scope content in progress.json, commands.sh and provenance.json. - pipeline_backend passes scope_file/scope_id into ReproducibilityManager. - Stage contracts and skills now state that scope, not prefix alone, defines the analysis range. Behavioral fixes: - Removed runtime-path Ft.SPL defaults from orchestrator/discovery fallbacks. Generic RNA-seq fallback prefix is now "analysis"; gene-family fallback leaves family/species/prefix blank until config/user input supplies them. - Natural-language example "目标物种Lc和Dl，参考物种At，比较物种Os，SPL基因家族成员鉴定和进化树" parses to: targetSpecies = Lc,Dl referenceSpecies = At comparativeSpecies = Os scopeMode = target_plus_reference_plus_comparative prefix = Multi.SPL - Lowercase family names such as "spl" are normalized to SPL in the fast keyword intent path. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/mas/utils/intent_parser.py gfa_core/mas/core/fast_intent.py apps/web/vue3/main.py gfa_core/mas/tools/stage_tools.py gfa_core/reproducibility_v2.py apps/web/pipeline_backend.py gfa_core/mas/core/orchestrator.py gfa_core/mas/agents/discovery.py gfa_core/pipeline_api_integration.py gfa_core/mas/agents/result_parser.py gfa_core/mas/agents/result_parser_cumulative.py - parse_gene_family_scope_selection / parse_intent_from_query / fast_intent_recognition on: "请用目标物种Lc和Dl，参考物种At，比较物种Os，进行SPL基因家族成员鉴定和进化树分析" "帮我执行Lc和Dl的spl基因家族鉴定" - apps.web.vue3.main.parse_natural_language + _ensure_gene_family_scope_file with real job config: job_1779115921147_d4e0a7 Real test result: - parse_natural_language returned prefix Multi.SPL and params containing targetSpecies Lc,Dl; referenceSpecies At; comparativeSpecies Os; scopeMode target_plus_reference_plus_comparative; geneFamily SPL. - _ensure_gene_family_scope_file wrote: workspace/job_1779115921147_d4e0a7/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference_plus_comparative.scope.json - ReproducibilityManager loaded the existing real scope file and resolved: scope_id = SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference target_species = Lc,Dl reference_species = At comparative_species = Os Known residual risks: - Historical docs, backups and test fixtures still contain Ft.SPL as historical data. Active runtime fallbacks were cleaned; historical records were not rewritten. - Full browser click-through of /api/agent/plan -> confirmation dialog -> /api/execute is still not automated by Playwright. - Some older result interpretation text still uses prefix as prose context; it no longer controls execution scope, but future interpretation cleanup should display scope_id/species sets to users. Next phase gate: - Update result interpretation/reporting so summaries display family + scope species sets rather than relying on prefix wording. ``` ```text Phase: 11 Result interpretation/report scope 化 Modified files: - gfa_core/mas/agents/result_parser.py - gfa_core/mas/agents/interpretation_evidence.py - gfa_core/mas/agents/result_parser_cumulative.py Code relationship summary: - ResultParserAgent now extracts scope context from manifest top-level fields, parameters or scope blocks. - Single-stage reports now include an explicit "分析范围" section with family, scope_id, scope_mode, target/reference/comparative/selected species. - Prompt metric bundles and LLM digests now carry scope, so downstream interpretation can cite the real analysis range. - Cumulative reports now accept optional scope_id, auto-discover a scope_id from matching manifests when omitted, and prefer scoped manifests before legacy manifests. - Evidence-first cumulative interpretation now filters manifests by scope_id and uses the scope label in report title/biological story. - The fallback cumulative mixin was updated to keep the same scope fields, avoiding a future import path reverting to prefix-only wording. Behavioral fixes: - "Multi.SPL" is no longer presented as if it were the biological species range in reports. - Real scope "SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference" reports as: family = SPL target_species = Lc,Dl reference_species = At comparative_species = Os selected_species = Lc,Dl,At - Cumulative interpretation can now distinguish scoped manifests from legacy manifests for the same prefix. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/mas/agents/result_parser.py gfa_core/mas/agents/result_parser_cumulative.py gfa_core/mas/agents/interpretation_evidence.py - ResultParserAgent._parse_result and _generate_report using: workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.domain.manifest.json - ResultParserAgent.generate_cumulative_report using real completed stages: phylogeny,motif,domain,structure,combo Real test result: - Parsed scope label: SPL家族；目标物种=Lc,Dl；参考物种=At；比较物种=Os - Single-stage markdown includes: 基因家族 SPL, Scope ID SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference, 目标物种 Lc/Dl, 参考物种 At, 比较物种 Os, 实际参与分析物种 Lc/Dl/At. - Cumulative report auto-discovered scope_id when scope_id was omitted and loaded the scoped manifests. Known residual risks: - Evidence helper sibling-manifest lookup still falls back to legacy sibling names for some cross-stage raw-file parsing; scoped report selection works, but sibling evidence enrichment can be made fully scope-aware later. Next phase gate: - Continue with the next analysis stage or reporting polish pass; prioritize any active user-facing confusion before broad refactors. ``` ```text Phase: 11b Evidence report hardcoded-prefix cleanup Modified files: - gfa_core/mas/agents/interpretation_evidence.py - gfa_core/mas/agents/result_parser.py Behavioral fixes: - Removed active result-report hardcoded Ft.SPL/Ath SPL wording from synteny, interspecies_synteny, PPI and publication-claim templates. - Evidence report text now uses the current scope label/family label for target/reference family statements. - Identify reference-count fallback no longer only recognizes SPL_Ath.fasta; it accepts reference-role, known-family or family-named inputs from manifest. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/mas/agents/result_parser.py gfa_core/mas/agents/result_parser_cumulative.py gfa_core/mas/agents/interpretation_evidence.py - ResultParserAgent.generate_cumulative_report with real scope job codex_scope_phy_ref_fast_test. Real test result: - Cumulative report scope label is: SPL家族；目标物种=Lc,Dl；参考物种=At；比较物种=Os - Serialized cumulative report no longer contains active "Ft.SPL" or "Ath SPL" wording. ``` ```text Phase: 12 Promoter scope guard and single-target resolution Modified files: - apps/cli/run_promoter.py - apps/web/pipeline_backend.py - gfa_core/mas/tools/stage_tools.py - gfa_core/schemas/stages/promoter_contract_v1.md Code relationship summary: - run_promoter.py now accepts --scope-file. - In scope mode, promoter resolves GFF/GFF3 and genome FASTA from the single selected target species instead of guessing from prefix. - Promoter refuses multi-target scope because promoter extraction cannot safely merge different genomes/GFFs in one run. - Success and failure manifests now record scope_id, scope_mode, scope_file, target/reference/comparative/selected species and actual gff3/genome/gene_id files. - pipeline_backend passes scope_file to run_promoter.py and skips legacy "first GFF/first genome" selection when scope is available. - Stage metadata and contract now state that promoter is a single-target stage. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_promoter.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py gfa_core/mas/agents/interpretation_evidence.py - Multi-target guard test: /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_promoter.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/codex_promoter_scope_guard_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --plantcare-tab workspace/codex_scope_phy_ref_fast_test/promoter/analysis/nonexistent.tab - Single-target real scope resolution: build_gene_family_scope(... target_species=['Lc'] ...) + resolve_promoter_scope_inputs() Real test result: - Multi-target Lc,Dl scope failed early with clear message: Promoter 阶段一次只能分析一个目标物种；当前 scope target_species=['Lc', 'Dl']。请在弹窗中只选择一个 target，或分别运行每个目标物种。 - Failure manifest was written to: workspace/codex_promoter_scope_guard_test/final/Multi.SPL.promoter.manifest.json and contains scope_id + target_species. - Single-target Lc scope resolved real files: /home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.Chr.gff3.gz /home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.Chr.fasta.gz Known residual risks: - Full PlantCARE execution was not run because it depends on external web/email service. Scope input resolution and multi-target guard were validated with real project data. - Promoter still writes legacy output names for compatibility; scoped manifest is written in addition when execution reaches success. Next phase gate: - Move to synteny/interspecies_synteny, which need careful target/comparative/reference separation and should not infer species from Multi.. ``` ```text Phase: 13 Synteny scope guard and full-proteome target resolution Modified files: - apps/cli/run_synteny.py - apps/web/pipeline_backend.py - gfa_core/mas/tools/stage_tools.py - gfa_core/schemas/stages/synteny_contract_v1.md Code relationship summary: - run_synteny.py now accepts --scope-file. - In scope mode, synteny resolves target pep/GFF/genome from the single selected target species. - Synteny refuses multi-target scope because it is an intraspecies MCScanX stage and cannot safely combine multiple genomes in one run. - validate_inputs now prioritizes full target proteome over identify final family proteins; identify proteins are only a legacy fallback. - final/{prefix}.final_members.id is now used as the preferred family gene list for Circos filtering. - Success and failure manifests now record scope fields and write a scoped manifest copy when scope_id is present. - pipeline_backend passes scope_file to run_synteny.py and skips legacy "first GFF/first genome" selection when scope is available. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_synteny.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py - Multi-target guard test: /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_synteny.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/codex_synteny_scope_guard_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --cpu 1 - Single-target real scope resolution: resolve_synteny_scope_inputs(workspace/codex_promoter_scope_guard_test/final/Multi.SPL.Lc.promoter.scope.json) Real test result: - Multi-target Lc,Dl scope failed early with clear message: Synteny 阶段是种内共线性，一次只能分析一个目标物种；当前 scope target_species=['Lc', 'Dl']。请在弹窗中只选择一个 target，或分别运行每个目标物种。 - Failure manifest was written to: workspace/codex_synteny_scope_guard_test/final/Multi.SPL.synteny.manifest.json and contains scope_id + target_species. - Single-target Lc scope resolved real files: /home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.pep.sim.fa.gz /home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.Chr.gff3.gz /home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.Chr.fasta.gz Known residual risks: - Full MCScanX execution was not run in this pass because it can be long and depends on local tool availability/data scale. Scope input resolution and multi-target guard were validated with real project data. - Some legacy behavior still allows identify final proteins as fallback when no full target proteome is found; this is retained for compatibility but is not the recommended path. Next phase gate: - Move to interspecies_synteny. It should use target/reference/comparative roles explicitly rather than scanning arbitrary protein files. ``` ```text Phase: 14 Interspecies synteny scope selection and gzip-safe real data discovery Modified files: - apps/cli/run_interspecies_synteny.py - apps/web/pipeline_backend.py - gfa_core/mas/tools/stage_tools.py - gfa_core/schemas/stages/interspecies_synteny_contract_v1.md Code relationship summary: - run_interspecies_synteny.py now accepts --scope-file. - Scope selection is applied before discovery: target_species -> target side selected reference/comparative species -> ref side - In scope mode, species files are resolved from scope files first. - In scope mode, full proteome can only come from scope `pep`; family_pep is not treated as all_pep. - reference/comparative family_pep can still be used for the interspecies family tree. - gzip FASTA/GFF files are now read through open_text(), which supports .gz ref_data paths. - Success/failure manifests now record scope_id, scope_mode, scope_file, target/reference/comparative/selected species and write a scoped manifest copy when scope_id is present. - pipeline_backend passes scope_file to run_interspecies_synteny.py and skips legacy input pep precheck when scope is available. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_interspecies_synteny.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py - Real scope discovery with: workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json - Real target+reference+comparative discovery with: workspace/job_1779115921147_d4e0a7/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference_plus_comparative.scope.json - Failure manifest write test with real scope metadata. Real test result: - target_plus_reference scope selected: targets = Lc,Dl refs = At target_all_peps = Lc,Dl ref_all_peps = empty because At all-pep path is missing in scope ref_family_protein = generated selected_ref.family.pep.fa - target_plus_reference_plus_comparative scope selected: targets = Lc,Dl refs = At,Os target_all_peps = Lc,Dl ref_all_peps = Os only ref_gff = At,Os - Failure manifest records: scope_id = SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference target_species = Lc,Dl reference_species = At comparative_species = Os Known residual risks: - Full MAFFT/IQ-TREE/MCScanX execution was not run in this pass because it can be long. The corrected scope selection, gzip-safe file discovery and manifest behavior were validated with real project data. - At whole-proteome path is absent in the current scope file, so interspecies MCScanX for At should remain partial/skipped unless the At all-pep file is added or genome_prepare provides it. This is correct behavior; the system must not substitute At.SPL.pep.fa as whole proteome. Next phase gate: - Move to karyotype, which is another target-side stage and must not run all targets together unless explicitly designed for multi-target output. ``` ```text Phase: 15 Karyotype target-scope selection and gzip GFF support Modified files: - apps/cli/run_karyotype.py - apps/web/pipeline_backend.py - gfa_core/mas/tools/stage_tools.py - gfa_core/schemas/stages/karyotype_contract_v1.md Code relationship summary: - run_karyotype.py now accepts --scope-file. - In scope mode, karyotype uses only `scope_file.target_species` and resolves each target species' GFF/genome from scope. - Multi-target scope remains supported, but output is per target species (`{Species}.{Family}.karyotype.*`), not one merged cross-species karyotype. - gzip GFF files are now read through open_text(), supporting real ref_data *.gff3.gz inputs. - Success/failure manifests now record scope_id, scope_mode, scope_file, target/reference/comparative/selected species and write a scoped manifest copy when scope_id is present. - pipeline_backend passes scope_file to run_karyotype.py. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_karyotype.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py - discover_species_inputs() using real scope: workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json - build_simplified_gff() against real gzip GFFs for Dl and Lc using real final_members IDs. - write_manifest() scope-copy test. Real test result: - Scope discovery selected only target species: Lc, Dl - Resolved real files: Lc: Lchinesis_genome.Chr.gff3.gz + Lchinesis_genome.Chr.fasta.gz Dl: Longan.v20190918.gff3.gz + Longan_Chr.v20190918.fasta.gz - GFF matching succeeded on gzip inputs: Dl sample: 5/5 matched rows Lc sample: 5/5 matched rows - Manifest writes both: ```text Phase: 16 PEP scoped protein-set input Modified files: - apps/cli/run_pep.py - apps/web/pipeline_backend.py - gfa_core/mas/tools/stage_tools.py - gfa_core/schemas/stages/pep_contract_v1.md Code relationship summary: - run_pep.py now accepts --scope-file and --fasta. - In scope mode, PEP resolves the same protein FASTA used by the matching phylogeny scope manifest (`parameters.scope_input_fasta`), then removes terminal/internal `*` into a job-local clean FASTA when needed. - target_only scope may fall back to `final/{prefix}.final_members.pep.fa`; multi-species/reference/comparative scopes must use the phylogeny-scoped merged FASTA unless --fasta is explicitly provided. - Success manifests record scope_id, scope_mode, scope_file, target/reference/comparative/selected species, family, and write a scoped manifest copy. - pipeline_backend passes the scoped FASTA into the existing PEP skill path and annotates the copied manifest with scope metadata, so the current faster skill implementation remains compatible. Real test command: - Compile run_pep.py / pipeline_backend.py / stage_tools.py. - Resolve PEP input from the real target+reference scope and verify it points to the phylogeny-scoped clean FASTA. - Run the PEP skill on that real scoped FASTA and verify final manifest contains scope metadata and the expected protein count. Real test result: - py_compile passed. - CLI resolver: `/home/xia-lab/wutx/00test/bbav5_0502_Yang_Tihuan.7880/workspace/codex_scope_phy_ref_fast_test/phylogeny/analysis/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.input.pep.clean.fa` - Backend resolver uses the same scoped clean FASTA instead of `final/{prefix}.final_members.pep.clean.fa`, preventing cross-scope overwrite. - PEP skill loaded 72 real sequences and produced: `workspace/codex_pep_scope_fast_test/final/Multi.SPL.pep.manifest.json` `workspace/codex_pep_scope_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.pep.manifest.json` - Manifest contains `scope_id=SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference` in both top-level fields and `parameters`. ``` ```text Phase: 17 Expression single-target scope selection Modified files: - apps/cli/run_expression.py - apps/web/pipeline_backend.py - gfa_core/mas/tools/stage_tools.py - gfa_core/schemas/stages/expression_contract_v1.md Code relationship summary: - Expression is treated as a single-target-species stage because expression matrices are species-specific. - pipeline_backend now reads scope_file for expression and rejects multi-target scopes before execution. - In scope mode, backend resolves member IDs from `identify/{Species}/final/{Species}.final_gfmembers.id` instead of globbing any `*geneID*` or multi-species `final/{prefix}.final_members.id`. - Removed the duplicated expression/gene/sample discovery block that could overwrite the first resolved inputs. - run_expression.py accepts `--scope-file`, validates that exactly one target species is selected, allows the scoped per-species gene ID file to satisfy upstream validation, records scope fields in manifest, and writes a scoped manifest copy. - comparative_summary.target_species now uses the scoped target species instead of parsing `Multi` from `Multi.SPL`. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_expression.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py - Resolver test against real Lc/Dl identify outputs from `workspace/codex_scope_phy_ref_fast_test/identify/{Species}/final/{Species}.final_gfmembers.id`. - Full expression run with a small real Lc member subset expression matrix and `rnaR` Rscript. - Multi-target guard run with target_species=[Lc,Dl]. Real test result: - py_compile passed. - Resolver selected: `workspace/codex_scope_phy_ref_fast_test/identify/Lc/final/Lc.final_gfmembers.id` `workspace/codex_scope_phy_ref_fast_test/identify/Dl/final/Dl.final_gfmembers.id` - Single-target Lc expression run succeeded in `workspace/codex_expression_scope_fast_test`, using: `workspace/codex_expression_scope_fast_test/identify/Lc/final/Lc.final_gfmembers.id` - R generated heatmap/correlation/barplot PDF+PNG and expression/correlation txt outputs. - Manifest contains `scope_id=SPL.target-Lc.ref-At.comp-Os.target_only`, `gene_count=33`, and comparative_summary.target_species=`Lc`. - Multi-target scope failed early with the intended message: `Expression 阶段一次只能分析一个目标物种；当前 scope target_species=['Lc', 'Dl']。请在弹窗中只选择一个 target，或分别运行每个目标物种。` ``` ```text Phase: 18 Ortholog target-to-reference scope mapping Modified files: - apps/cli/run_ortholog.py - apps/web/pipeline_backend.py - gfa_core/mas/tools/stage_tools.py - gfa_core/schemas/stages/ortholog_contract_v1.md Code relationship summary: - run_ortholog.py had the broken inline CONDA_BIN statement repaired. - pipeline_backend ortholog stage now reads scope_file when provided. - In scope mode, target family proteins are resolved per species from: `identify/{Species}/final/{Species}.final_gfmembers.pep.fa`, then compatible `{Species}.{Family}.pep.fa` fallbacks. - Reference protein resolves from scope `family_pep` first, then compatible `{Reference}.{Family}.pep.fa`; it no longer depends only on hardcoded Ath/At filenames. - Multi-target scope runs one BLASTP job per target species against the first reference species, then merges mappings. - Output mapping now includes source/target species columns: `Source_ID Target_ID Identity(%) Source_Species Target_Species` - Manifest records scope fields and writes a scoped manifest copy. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_ortholog.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py - run_stage('ortholog') with real scope: `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json` target: Lc,Dl; reference: At; family: SPL. Real test result: - py_compile passed. - BLASTP 2.17.0+ was available. - Ortholog ran two real BLASTP jobs: Lc final family proteins vs `input/job_1779115921147_d4e0a7/At.SPL.pep.fa` Dl final family proteins vs `input/job_1779115921147_d4e0a7/At.SPL.pep.fa` - Produced 54 mapped pairs. - Output: `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.ortholog_pairs.txt` `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.orthologs.txt` `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.ortholog_pairs.raw.txt` `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.ortholog_blastp.txt` `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.ortholog.manifest.json` - Example mapping: `LITCHI014255.m1 At2g47070 49.81 Lc At` ``` ```text Phase: 19 PPI scoped ortholog mapping input Modified files: - apps/cli/run_ppi.py - apps/web/pipeline_backend.py - gfa_core/mas/tools/stage_tools.py - gfa_core/schemas/stages/ppi_contract_v1.md Code relationship summary: - run_ppi.py had the broken inline CONDA_BIN statement repaired. - pipeline_backend PPI stage now reads scope_file when provided. - In scope mode, reference_species is converted to the STRING species code (`At -> ath`, `Os -> osa`). - If an ortholog mapping exists, PPI builds a scoped source gene ID file from `{prefix}.orthologs.txt` / `{prefix}.ortholog_pairs.txt` and passes the mapping to the existing PPI skill. - Multi-target scope now requires Ortholog first; PPI will not send mixed local IDs from multiple species directly to STRING. - PPI manifest annotation uses the shared scope manifest helper and writes a scoped manifest copy after a real PPI run. Real-data validation command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_ppi.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py - build_ppi_gene_ids_from_ortholog_mapping() with real ortholog output: `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.orthologs.txt` - skills.ppi.run.parse_ortholog_mapping() on the same real mapping. Real-data validation result: - py_compile passed. - From real Lc/Dl -> At ortholog mapping, PPI source ID extraction produced 54 source genes: `workspace/codex_scope_phy_ref_fast_test/ppi/analysis/codex_ppi_source_ids.txt` - STRING species code resolution: `At -> ath`, `Os -> osa` - Existing PPI skill parsed 54 source→At mappings, e.g.: `LITCHI014255.m1 -> At2g47070` Known residual risk: - Full STRING network querying was not executed in this pass because it depends on external network service availability. The scoped input/mapping path was validated with real ortholog data; a complete online PPI run should be verified separately when network access is allowed. ``` ```text Phase: 20 Ka/Ks scoped target-reference runs and YN00 parser correction Modified files: - apps/cli/run_kaks_v2.py - apps/web/pipeline_backend.py - gfa_core/mas/tools/stage_tools.py - gfa_core/schemas/stages/kaks_contract_v1.md Code relationship summary: - run_kaks_v2.py now has a valid CONDA_BIN/PATH block. - run_kaks_v2.py can read gzipped FASTA inputs. - run_kaks_v2.py now matches IDs case-insensitively after removing transcript/version suffixes, so `At2g47070` can match `AT2G47070.1`. - run_kaks_v2.py now detects pal2nal error output instead of passing an empty/error codon alignment into yn00. - YN00 parsing was corrected. The old parser incorrectly treated columns `S/N/t` as `Ka/Ks/KaKs`; the corrected parser reads `dN/dS/omega`. - pipeline_backend KaKs stage now reads scope_file. - In scope mode, KaKs splits multi-target scopes into per-target runs: `{prefix}.{Target}_to_{Reference}` - Target CDS/PEP and reference CDS/PEP are resolved per species. Reference PEP is intentionally resolved from the same whole-genome annotation source as reference CDS, not from `{Reference}.{Family}.pep.fa`, because Ka/Ks requires CDS and protein from the same transcript set. - Aggregate and scoped manifests are written: `{prefix}.kaks.manifest.json` `{prefix}.{scope_id}.kaks.manifest.json` Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_kaks_v2.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py - Resolve real scope files: Lc CDS: Lchinesis_genome.sim.cds.fa.gz Dl CDS: longan.v20190918.sim.CDS.gz At CDS: Arabidopsis_thaliana.TAIR10.cds.all.fa At PEP: Arabidopsis_thaliana.TAIR10.pep.all.fa - Split real ortholog mapping: Lc->At: 33 pairs Dl->At: 21 pairs - Run full scoped KaKs through run_stage('kaks'). Real test result: - py_compile passed. - Initial test with `At.SPL.pep.fa` failed pal2nal consistency checks, proving that NCBI/downloaded family PEP cannot be mixed with TAIR10 CDS for Ka/Ks. - After switching reference PEP to TAIR10 whole PEP and fixing YN00 parser, full run succeeded: `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.Lc_to_At.kaks_results.csv` `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.Dl_to_At.kaks_results.csv` `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.kaks.manifest.json` `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.kaks.manifest.json` - Successful pairs: Lc->At: 33/33 Dl->At: 21/21 - Corrected example values: `LITCHI014255.m1, At2g47070, Ka=0.3281, Ks=1.3925, Ka/Ks=0.2356` `Dil.01g033640.1.t1, At5g18830, Ka=0.3490, Ks=1.5298, Ka/Ks=0.2282` - Aggregate manifest records: `species_runs=2`, `input_pairs=54`, `scope_id=SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference`. ``` ```text Phase: 21 Summary scoped objective result index Modified files: - apps/cli/run_summary.py - apps/web/pipeline_backend.py - gfa_core/mas/tools/stage_tools.py - gfa_core/schemas/stages/summary_contract_v1.md Code relationship summary: - run_summary.py was replaced with a self-contained manifest aggregator. The old file imported a missing summary_agent module and could not run as a reliable CLI stage. - Summary is defined as an objective result index and QC report, not deep biological interpretation. - Summary scans `final/*.manifest.json`, excludes summary/minimal self manifests, filters child manifests whose prefix does not match the requested prefix, groups results by `scope_id`, and records stage status, key metrics, artifacts and consistency checks. - Summary supports gene-family and RNA-seq style manifests because it only depends on the shared manifest schema fields. - In scope mode, Summary writes both a global summary and a scoped summary: `{prefix}.summary_report.md` `{prefix}.summary.files.tsv` `{prefix}.summary.manifest.json` `{prefix}.{scope_id}.summary_report.md` `{prefix}.{scope_id}.summary.files.tsv` `{prefix}.{scope_id}.summary.manifest.json` - pipeline_backend now runs Summary through `apps/cli/run_summary.py` and no longer requires an identify manifest as the seed. Real test command: - /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_summary.py - Run CLI summary with real scope: `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json` - Run backend `run_stage('summary')` on the same real workspace. - Run CLI summary on an existing real RNA-seq manifest workspace: `workspace/job_1778662928678_06813e`, prefix `Os`. Real test result: - py_compile passed. - CLI and backend Summary both succeeded. - Summary grouped the real Lc,Dl target / At reference / Os comparative SPL run into: `SPL.target-Lc-Dl.ref-At.comp-Os.target_only`: identify `SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference`: combo, domain, kaks, motif, ortholog, phylogeny, reference_classification, structure - Output report: `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.summary_report.md` - Output artifact index: `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.summary.files.tsv` - Output manifest: `workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.summary.manifest.json` - Aggregate manifest records: `manifest_count=9`, `scope_count=2`, `artifact_count=103`. - RNA-seq manifest aggregation also succeeded: `workspace/job_1778662928678_06813e/final/Os.summary_report.md` `workspace/job_1778662928678_06813e/final/Os.summary.files.tsv` `workspace/job_1778662928678_06813e/final/Os.summary.manifest.json` - RNA-seq summary grouped 13 unscoped stage manifests, including: annotation, config_autofill, rnaseq_align, rnaseq_candidate_family, rnaseq_de, rnaseq_fgsea, rnaseq_go_enrich, rnaseq_kegg_enrich, rnaseq_qc, rnaseq_quant, rnaseq_sample_qc, rnaseq_wgcna, samples_autofill. - RNA-seq aggregate manifest records: `manifest_count=13`, `scope_count=1`, `artifact_count=305`. Known residual risks: - Summary currently reports objective artifacts/metrics/status only. It intentionally does not infer biological conclusions or recommend subfamilies; that should be a separate deep interpretation stage built on top of these indexed outputs. ``` ```text Phase: 22 Full real-data end-to-end validation and fixes Modified files: - apps/web/pipeline_backend.py - apps/cli/run_reference_classification.py - apps/cli/run_promoter.py - apps/cli/run_synteny.py Validation workspace: - workspace/codex_full_flow_real_20260519_235947 Real data: - target species: Lc,Dl - reference species: At - comparative species: Os - family: SPL - input dir: input/job_1779115921147_d4e0a7 Issues found and fixed: - Phylogeny advanced IQ-TREE params beginning with `-` were passed as a separate argv value, so argparse treated the value as another option. pipeline_backend now passes these as `--iqtree-params=` and `--mafft-params=`. - reference_classification auto-selected `Target.SPL.geneID.subfamily`, which already contained target labels, instead of the pure At reference file. It now prefers `{reference_prefix}.{family}.subfamily`, for example `At.SPL.subfamily`. - promoter failed on non-UTF-8 stderr from external tools. The command runner now decodes subprocess output with replacement instead of raising UnicodeDecodeError. - synteny failed or fell back to very slow BLASTP because whole-proteome FASTA files contained terminal `*`/`.` stop markers and DIAMOND v2 uses `--threads`, not `--thread`. Synteny now writes a BLAST-safe cleaned protein FASTA and uses the correct DIAMOND option. Final validation result: - Final summary report: `workspace/codex_full_flow_real_20260519_235947/final/Multi.SPL.summary_report.md` - Final summary manifest: `workspace/codex_full_flow_real_20260519_235947/final/Multi.SPL.summary.manifest.json` - Final summary artifact index: `workspace/codex_full_flow_real_20260519_235947/final/Multi.SPL.summary.files.tsv` - Summary recorded: `manifest_count=18`, `scope_count=3`, `artifact_count=181` - Final consistency check: `No manifest-level consistency issues detected` Successful final stages: - target_plus_reference scope: identify, phylogeny, reference_classification, structure, domain, motif, pep, combo, karyotype, interspecies_synteny, ortholog, ppi, kaks - Lc target_only scope: expression, promoter, synteny - Dl target_only scope: promoter, synteny Key final metrics: - Identify/phylogeny protein set: 76 sequences = 59 target + 17 At reference. - Reference classification: target_gene_count=59, reference_gene_count=17. - Domain: proteins_total=76, proteins_with_domains=67, domain_coverage_rate=0.8816. - Motif: sequence_count=76. - Ortholog: ortholog_pairs=58. - Ka/Ks: species_runs=2, input_pairs=58. - Karyotype: species_count=2, success_count=2. - Interspecies synteny: status=success, Lc.At and Dl.At collinearity files produced. - Promoter: Lc and Dl target_only runs both succeeded. - Synteny: Lc gene_count=31896, Dl gene_count=37142. Known residual risks: - The validation used a generated full_clean scope for interspecies_synteny because the original scope had empty At whole-proteome paths. The next front-end/config revision should ensure whole-proteome reference paths are filled in the generated scope by default. - Full phylogeny with ModelFinder is slow for validation. A future UI/plan option should expose fast verification mode such as `-m LG+G4 -nt -fast`. ``` ```text Phase: 23 Automatic scope artifact and reference-path repair Modified files: - apps/cli/run_promoter.py - apps/cli/run_synteny.py - gfa_core/gene_family_scope.py Purpose: - Remove manual intervention for two remaining scope problems: 1. Single-target stages wrote scope manifests but their artifacts still pointed to generic files, so Lc/Dl runs could overwrite each other. 2. Generated scopes could leave reference whole-proteome paths empty when config.xlsx recorded files stored in local ref_data instead of input/final. Fixes: - promoter and synteny now materialize scope-specific artifact copies when `scope_id` is present. Generic legacy outputs are still written for compatibility, but scoped manifests point to stable paths containing the sanitized scope ID. - promoter failure handling now writes both generic and scoped failure manifests when a scope file is provided. - promoter PATH initialization was repaired so the test4py311 tool directory is actually prepended; this fixed a real `gffread` not found failure. - gene_family_scope now searches local reference-data roots, including `/home/xia-lab/wutx/plantAnalysisAgent/ref_data`, by the config original file name when input/workspace/final lookup fails. Real-data validation: - Scope generation: `/tmp/codex_scope_autofill_test.json` resolved At whole-genome gff/cds/genome/pep paths from local ref_data, including: `/home/xia-lab/wutx/plantAnalysisAgent/ref_data/ath/Arabidopsis_thaliana.TAIR10.pep.all.fa` - Synteny rerun with real Lc and Dl target_only scopes: `Multi.SPL.SPL.target-Lc.ref-At.comp-Os.target_only.synteny.manifest.json` `Multi.SPL.SPL.target-Dl.ref-At.comp-Os.target_only.synteny.manifest.json` Both scoped manifests point to scope-specific blast/circos/report artifacts and `missing=0` for registered artifact paths. - Promoter rerun with real Lc and Dl target_only scopes using existing local PlantCARE tab: `Multi.SPL.SPL.target-Lc.ref-At.comp-Os.target_only.promoter.manifest.json` `Multi.SPL.SPL.target-Dl.ref-At.comp-Os.target_only.promoter.manifest.json` Both scoped manifests point to scope-specific report and PlantCARE/table artifacts and `missing=0` for registered artifact paths. Validation commands: - `/home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_promoter.py apps/cli/run_synteny.py gfa_core/gene_family_scope.py` - `/home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/resolve_scope.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/codex_full_flow_real_20260519_235947 --prefix Multi.SPL --family SPL --target-species Lc,Dl --reference-species At --comparative-species Os --scope-mode target_plus_reference --output /tmp/codex_scope_autofill_test.json` - Real Lc/Dl synteny and promoter reruns in `workspace/codex_full_flow_real_20260519_235947`. Known residual risks: - The promoter validation intentionally used an existing local PlantCARE tab to avoid submitting to the external PlantCARE service. That tab was empty after filtering, so R promoter visualizations reported "no lines available in input"; the stage still completed and scoped artifact/manifest behavior was validated. ``` ## 2026-05-26 追加记录：Stage Requirements 与 Agent 职责边界 KaKs/PPI/scope 讨论暴露出 PlanningAgent、DiagnosisAgent、ResultParserAgent 与执行层之间存在职责混杂：静态依赖、动态需求、artifact 解析、完成状态、修复策略在多个文件重复实现。 详细问题归类、非破坏迁移计划和验收矩阵见：[`docs/stage_requirements_agent_refactor_plan_2026-05-26.md`](stage_requirements_agent_refactor_plan_2026-05-26.md)。