1759 lines
91 KiB
Markdown
Executable File
1759 lines
91 KiB
Markdown
Executable File
# Gene Family Scope Refactor Plan
|
||
|
||
本文档记录基因家族分析链路的代码通读结果、当前直接逻辑、关联代码、主要风险、修订方案和真实数据验收计划。
|
||
|
||
测试基准固定为：
|
||
|
||
- target species: `Lc,Dl`
|
||
- reference species: `At`
|
||
- comparative species: `Os`
|
||
- family: `SPL`
|
||
- real data job: `job_1779115921147_d4e0a7`
|
||
- primary input dir: `input/job_1779115921147_d4e0a7`
|
||
- primary workspace: `workspace/job_1779115921147_d4e0a7`
|
||
|
||
## 1. 总体结论
|
||
|
||
当前系统仍然以 `prefix` 作为几乎唯一的分析身份，例如 `Multi.SPL`、`Lc.SPL`、`At.SPL`。这在单物种串行分析中能工作，但在多物种、target/comparative/reference 混合分析中不够：
|
||
|
||
- `prefix` 只能表达一个字符串，不能表达本次分析使用了哪些 target、reference、comparative。
|
||
- 多个不同选择范围会写入同一批 `final/{prefix}.*` 文件。
|
||
- 下游阶段多数只读 `final/{prefix}.final_members.*`、`final/{prefix}.tree.nwk`，不知道这些文件对应哪个物种组合。
|
||
- 部分阶段会从“最新 manifest”或固定中间目录取文件，容易串用旧结果。
|
||
- 前端弹窗只在 `identify` 和 `interspecies_synteny` 上部分支持物种选择，其他阶段没有统一选择范围。
|
||
|
||
因此需要引入统一的 `scope` 作为一等概念。`prefix` 保留为兼容输出前缀，`scope` 表达真实分析范围。
|
||
|
||
建议 scope 字段：
|
||
|
||
```json
|
||
{
|
||
"family": "SPL",
|
||
"prefix": "Multi.SPL",
|
||
"scope_id": "SPL.target-Lc-Dl.ref-At.comp-Os",
|
||
"target_species": ["Lc", "Dl"],
|
||
"reference_species": ["At"],
|
||
"comparative_species": ["Os"],
|
||
"selected_species": ["Lc", "Dl", "At", "Os"],
|
||
"stage_scope": "target_plus_reference_plus_comparative"
|
||
}
|
||
```
|
||
|
||
## 2. 代码链路通读记录
|
||
|
||
### 2.1 前端入口与弹窗
|
||
|
||
主要文件：`apps/web/vue3/index.html`
|
||
|
||
当前直接逻辑：
|
||
|
||
- 参数确认弹窗在 `index.html:1718` 附近定义。
|
||
- basic parameters 仍展示 `confirmParams.prefix`，`index.html:1730`。
|
||
- `identify` 独有 target/comparative/reference 展示在 `index.html:1803` 附近，但实际选择都写入 `confirmParams.targetSpecies`。
|
||
- `reference_classification` 只提供 `referenceTargetPrefix` 和 `referencePrefix`，`index.html:1889` 附近，没有 target/ref 物种勾选。
|
||
- 批量执行弹窗的 identify 物种选择在 `index.html:2448` 附近，仍只针对 identify。
|
||
- `showConfirm()` 在 `index.html:5149` 附近将 `confirmParams.prefix = normalizedPrefix(params.prefix)`。
|
||
- 只有 `interspecies_synteny` 调用 `loadInterspeciesSpeciesOptions()`，`index.html:5125` 附近。
|
||
- 单阶段执行时，`executeSingleStageWithParams()` 只对 `identify` 传 `targetSpecies`，只对 `interspecies_synteny` 传 `targetSpecies/refSpeciesList`，`index.html:5809-5877`。
|
||
- 普通执行 `doExecute()` 会把大量参数统一发送，包括 `targetSpecies/refSpeciesList`，`index.html:6343-6404`，但后端和 CLI 不一定消费。
|
||
- agent plan 展示仍打印 `Prefix`，`index.html:7668`。
|
||
- direct execution 只对 `identify` 和 `reference_classification` 做少量参数分支，`index.html:7768-7794`。
|
||
- 自动倒计时默认 15 秒，`index.html:5191`，批量倒计时也单独维护。
|
||
|
||
关联代码：
|
||
|
||
- `/api/execute` 请求进入 `apps/web/vue3/main.py`。
|
||
- `/api/agent/plan` 的返回被 `runPlanForIntent()` 消费，再进入批量执行。
|
||
- `/api/stage-status` 只按 `job_id/stage/prefix` 查状态，没有 scope。
|
||
|
||
当前风险：
|
||
|
||
- 用户选择 target/ref/comparative 后，只有少数阶段收到选择。
|
||
- `reference_classification/motif/domain/structure/combo` 仍按 prefix 读全局文件。
|
||
- 前端让用户理解和填写 prefix，用户负担过高。
|
||
- 倒计时太短且没有统一暂停机制。
|
||
|
||
### 2.2 Web API 与任务执行
|
||
|
||
主要文件：`apps/web/vue3/main.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- `ExecuteRequest` 定义 `prefix` 默认 `"Test"`，并有 `targetSpecies/refSpeciesList`，`main.py:1353-1378`。
|
||
- `run_stage_sync()` 解析 effective prefix 后调用 `pipeline_backend.run_stage()`，`main.py:1604-1711`。
|
||
- `targetSpecies` 只作为参数传给 backend；是否生效由 backend 的 stage 分支决定。
|
||
- `_load_identify_species_selection()` 从 config/input 推断 identify 可选物种，`main.py:230`。
|
||
- `_infer_family_from_config_or_final()` 和 `_apply_inferred_family_to_intent()` 尝试从 config/final 推断 family 和 prefix，`main.py:157-227`。
|
||
- `/api/execute` 会先查 stage status，仍然以 `prefix` 为粒度，`main.py:2505-2534`。
|
||
- `/api/stage-status` 调用 `get_unified_stage_status(job_id,prefix,stage)`，`main.py:3060-3066`。
|
||
- `parse_natural_language()` 仍从文本里抽 prefix 或构造 `species.family`，`main.py:3344-3526`。
|
||
- agent chat 中 explicit species 会强制改写 `intent["prefix"] = f"{species}.{family}"`，`main.py:3769`，多物种会变成 `Multi.family`。
|
||
- agent plan 阶段只在含 `identify` 且没有 targetSpecies 时补 `identifySpeciesSelection`，`main.py:4535-4562`。
|
||
|
||
关联代码：
|
||
|
||
- 实际 CLI 调度在 `apps/web/pipeline_backend.py`。
|
||
- 状态判断在 `gfa_core/mas/tools/stage_status.py`。
|
||
- 多智能体 stage 工具在 `gfa_core/mas/tools/stage_tools.py`。
|
||
|
||
当前风险：
|
||
|
||
- API 层没有统一 `scope` 请求模型。
|
||
- effective prefix 会从历史 manifest/final 自动推断，可能掩盖用户本次选择。
|
||
- status 只看 prefix，无法判断同一个 prefix 下不同物种组合是否已经完成。
|
||
|
||
### 2.3 后端 CLI 调度
|
||
|
||
主要文件：`apps/web/pipeline_backend.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- `run_stage()` 是所有 stage 的核心后端分发函数。
|
||
- `identify` 分支调用 `run_identify.py`，并传 `--target-species`，`pipeline_backend.py:1149` 附近。
|
||
- `phylogeny` 分支优先使用 `final/{prefix}.final_members.pep.fa`，并会去除蛋白末尾 `*`，`pipeline_backend.py:1257-1339`。
|
||
- `reference_classification` 分支固定读 `final/{prefix}.tree.nwk`，并用 `reference_target_prefix or prefix.split(".")[0]`，`pipeline_backend.py:1366-1400`。
|
||
- `structure` 分支固定检查 `final/{prefix}.tree.nwk`，并调用 `run_structure.py --prefix {prefix}`，`pipeline_backend.py:1422-1444`。
|
||
- `domain` 分支传 `--proteins final/{prefix}.final_members.pep.fa`，`pipeline_backend.py:1450-1483`。
|
||
- `motif` 分支传 `--fasta final/{prefix}.final_members.pep.fa`，`pipeline_backend.py:1511-1534`。
|
||
- `pep/promoter/synteny/karyotype/expression/ppi/kaks/combo` 也基本围绕 `{prefix}` 拼文件。
|
||
- `combo` 分支只检查 `final/{prefix}.{stage}.manifest.json` 是否存在，但 `run_combo.py` 内部仍可能找最新 manifest，`pipeline_backend.py:3030-3059`。
|
||
- `genome_prepare` 分支可以跑多物种准备，`pipeline_backend.py:3706` 之后。
|
||
|
||
关联代码：
|
||
|
||
- 传入参数来自 `main.py:run_stage_sync()` 和前端 `index.html`。
|
||
- 每个 CLI 自己还会再次自动发现文件和 manifest。
|
||
|
||
当前风险：
|
||
|
||
- backend 是 scope 断开的主要位置：参数收到了，但大多数 stage 没有用。
|
||
- 多数 stage 默认读 canonical 文件，导致同 prefix 不同分析范围互相覆盖。
|
||
- `reference_classification` 用字符串 target prefix，不足以表达 Lc+Dl 双目标。
|
||
|
||
### 2.4 意图识别、stage catalog、多智能体工具层
|
||
|
||
主要文件：
|
||
|
||
- `gfa_core/mas/core/fast_intent.py`
|
||
- `gfa_core/mas/stage_catalog.py`
|
||
- `gfa_core/mas/tools/stage_tools.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- `fast_intent.py` 用 `ANALYSIS_KEYWORDS` 判定阶段，`fast_intent.py:9`。
|
||
- 物种识别是简单 species code list，`fast_intent.py` 中 `SPECIES_KEYWORDS`。
|
||
- 返回字段主要是 `stage/stages/gene_family/species/prefix`，`fast_intent.py:86-362`。
|
||
- `stage_catalog.py` 定义 `STAGE_DEPENDENCIES`，如 `combo` 依赖 `identify/phylogeny/motif/structure/domain`，`stage_catalog.py:60`。
|
||
- `complete_dependencies()` 只按 stage 补依赖，不关心 scope，`stage_catalog.py:382`。
|
||
- `StageToolExecutor.execute()` 调用 `pipeline_backend.run_stage()`，并只传 `targetSpecies/refSpeciesList` 等已有参数，`stage_tools.py:139-245`。
|
||
|
||
关联代码：
|
||
|
||
- agent plan 使用这些 intent 和 stage catalog 输出计划。
|
||
- frontend `runPlanForIntent()` 消费 plan 并发起 batch execute。
|
||
|
||
当前风险：
|
||
|
||
- 多智能体计划没有 scope 字段，无法要求下游复用同一物种选择。
|
||
- 依赖完成判断无法区分不同 scope。
|
||
- repair/diagnosis agent 即使诊断出问题，也没有“修复 scope 输入/重建 stage input”的规范动作。
|
||
|
||
### 2.5 状态、manifest 校验和复现
|
||
|
||
主要文件：
|
||
|
||
- `gfa_core/mas/tools/stage_status.py`
|
||
- `apps/cli/validate_manifest.py`
|
||
- `gfa_core/reproducibility_v2.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- `manifest_path_for()` 固定返回 `final/{prefix}.{stage}.manifest.json`，`stage_status.py:51`。
|
||
- `get_stage_status()` 读取该 manifest 并校验 artifact 是否存在，`stage_status.py:142`。
|
||
- `list_completed_stages()` glob `f"{prefix}.*.manifest.json"`，`stage_status.py:233-259`。
|
||
- `validate_manifest.py` 校验 JSON schema 和 artifact 文件存在，`validate_manifest.py:30-104`。
|
||
- `ReproducibilityManager` 只以 `job_id/prefix` 建立复现目录和命令记录，`reproducibility_v2.py:57-344`。
|
||
|
||
关联代码：
|
||
|
||
- 前端 `/api/stage-status`、agent completed stage、batch 跳过逻辑依赖 stage_status。
|
||
- 每个 CLI 产出的 manifest 被 validate/status/repro 后续使用。
|
||
|
||
当前风险：
|
||
|
||
- 有 manifest 不代表本次选择范围正确。
|
||
- artifact 存在不代表来自正确 target/ref/comparative。
|
||
- 不同 scope 下的复现命令会混在同一个 prefix 目录。
|
||
|
||
### 2.6 契约与技能
|
||
|
||
主要目录：
|
||
|
||
- `gfa_core/schemas/stages/`
|
||
- `skills/`
|
||
|
||
当前直接逻辑：
|
||
|
||
- identify contract 只要求 `final/{prefix}.final_members.id/fa`，没有 scope。
|
||
- phylogeny contract 固定输入 `final/{prefix}.final_members.pep.fa` 和 identify manifest。
|
||
- motif/domain/structure/combo contract 也固定 `prefix` 文件和固定中间路径。
|
||
- combo contract 明确写死 `motif/analysis/meme_out/meme.xml` 和 `domain/analysis/results.domtblout`。
|
||
- skills 文档和 run.py 多数仍以 `prefix` 作为外部接口。
|
||
|
||
当前风险：
|
||
|
||
- 即使 CLI 修了，契约不改也会把后续开发拉回 prefix-only。
|
||
- combo contract 的固定中间路径会继续制造串结果。
|
||
|
||
## 3. 各分析阶段现状与关联
|
||
|
||
### 3.1 Identify
|
||
|
||
主要文件：`apps/cli/run_identify.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- 从 prefix 解析 family，或从 `config.xlsx` Analysis sheet 读取 family，`run_identify.py:147`、`552`、`803`。
|
||
- `find_pep_files()` 能读取 config 并按 `target_species` 过滤，`run_identify.py:678-743`。
|
||
- CLI 参数支持 `--target-species`，`run_identify.py:1926`。
|
||
- main 中读取 `selected_target_species` 并传给 `find_pep_files()`，`run_identify.py:1995-1998`。
|
||
- 逐物种跑 HMM/BLAST 后合并成 `final/{effective_prefix}.final_members.pep.fa/id`，`run_identify.py:2235-2239`。
|
||
- manifest 中已有 `target_species` 和 per-species artifacts，`run_identify.py:2244-2304`。
|
||
- InterProScan 是非阻塞 domain check，不是最终成员筛选器，`run_identify.py:2239` 后。
|
||
|
||
关联代码：
|
||
|
||
- backend `identify` 分支传 `--target-species`。
|
||
- phylogeny/motif/domain/structure 读取 identify 的 final members。
|
||
- identify_refine 会改写 final members，但没有 scope。
|
||
|
||
主要风险：
|
||
|
||
- identify 已支持 target，但没有 reference/comparative 角色定义。
|
||
- merged final members 覆盖同 prefix 的上一次选择。
|
||
- manifest 没有 scope_id 和输入 hash，不足以被下游校验。
|
||
|
||
### 3.2 Phylogeny
|
||
|
||
主要文件：`apps/cli/run_phylogeny.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- 默认读取 `final/{prefix}.identify.manifest.json`，`run_phylogeny.py:177-215`。
|
||
- 从 identify manifest 中找 `final_proteins/final_members/proteins`。
|
||
- 输出 `final/{prefix}.pep.msa.fa`、`final/{prefix}.tree.nwk`、manifest，`run_phylogeny.py:902-973`。
|
||
- CLI 有 `--proteins` override，`run_phylogeny.py:1003-1005`。
|
||
- tree plotting 的 subfamily 查找与 reference_classification 输出不完全统一。
|
||
|
||
关联代码：
|
||
|
||
- reference_classification、motif、domain、structure、combo 都读 tree。
|
||
|
||
主要风险：
|
||
|
||
- 没有 stage-level selection，无法表达 target-only、target+ref、target+comp 等不同树。
|
||
- 同 prefix 不同树会覆盖 `final/{prefix}.tree.nwk`。
|
||
|
||
### 3.3 Reference Classification
|
||
|
||
主要文件：`apps/cli/run_reference_classification.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- 目标树默认 `final/{prefix}.tree.nwk`。
|
||
- 如树中没有参考 tip，会用 target proteins + reference proteins 构建 helper joint tree。
|
||
- target 判断依赖 `--target-prefix`，reference 判断依赖 `--reference-prefix`。
|
||
- 输出 `final/{prefix}.reference_classification.*` 和兼容 `{prefix}.geneID.subfamily`。
|
||
|
||
关联代码：
|
||
|
||
- motif/domain/structure 优先读取 `reference_classification.subfamily.tsv`。
|
||
- combo 应使用 reference_classification 的 subfamily。
|
||
|
||
主要风险：
|
||
|
||
- 多目标物种 `Lc,Dl` 无法用单个 `target_prefix` 准确表达。
|
||
- 选择一个物种做分类时，现有逻辑可能仍用全体 tree。
|
||
- 没有 scope 校验，可能用不含 At 的树或旧 helper tree。
|
||
|
||
### 3.4 Motif
|
||
|
||
主要文件：`apps/cli/run_motif.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- `prepare_directories()` 固定 `motif/analysis/meme_out`，`run_motif.py:57-61`。
|
||
- 默认从 identify manifest 或 `final/{prefix}.final_members.pep.fa` 取蛋白，`run_motif.py:128-167`。
|
||
- `run_meme()` 检查 `meme_out/meme.xml` 存在就复用，`run_motif.py:229-322`。
|
||
- 输出 final motif 图、seqlogo、report、manifest。
|
||
|
||
关联代码：
|
||
|
||
- combo 固定找 `motif/analysis/meme_out/meme.xml`。
|
||
|
||
主要风险：
|
||
|
||
- 不同 scope 复用同一个 meme.xml。
|
||
- 不同参数或 FASTA 改变时没有严格 hash 判断。
|
||
|
||
### 3.5 Domain
|
||
|
||
主要文件：`apps/cli/run_domain.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- `find_latest_manifest(final_dir, stage)` 只按 `*.{stage}.manifest.json` 找最新，不带 prefix，`run_domain.py:108-114`。
|
||
- 默认从 identify/phylogeny latest manifest 或 `final/{prefix}.*` 取输入。
|
||
- `run_hmmscan()` 固定写 `domain/analysis/results.domtblout`，`run_domain.py:239-286`。
|
||
- manifest artifacts 记录 domtblout 路径，`run_domain.py:856-864`。
|
||
|
||
关联代码：
|
||
|
||
- combo 从 domain manifest 读取 domtblout。
|
||
|
||
主要风险：
|
||
|
||
- latest manifest 可能跨 prefix。
|
||
- domtblout 固定路径会被覆盖。
|
||
- 没有 scope 一致性。
|
||
|
||
### 3.6 Structure
|
||
|
||
主要文件：`apps/cli/run_structure.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- 默认 gene ID 文件为 `final/{prefix}.final_members.id`，`run_structure.py:606`。
|
||
- 默认 tree 为 `final/{prefix}.tree.nwk`。
|
||
- GFF3 自动解析为 `{species_short}_final.gff3`，其中 species_short 来自 prefix，`run_structure.py:76`。
|
||
- 优先读取 `final/{prefix}.reference_classification.subfamily.tsv`，`run_structure.py:141-168`。
|
||
- 输出 `structure/analysis/{prefix}.structure.filtered.gff3`、`{prefix}.structure.tsv` 和 final 图。
|
||
|
||
关联代码：
|
||
|
||
- combo 读取 structure manifest 中的 filtered gff3。
|
||
|
||
主要风险：
|
||
|
||
- `Multi.SPL` 无法映射到 `Lc_final.gff3` 和 `Dl_final.gff3`。
|
||
- 多物种结构需要 per-species GFF 过滤再合并或分图。
|
||
- tree tip、gene ID、GFF ID 没有统一校验。
|
||
|
||
### 3.7 Combo
|
||
|
||
主要文件：`apps/cli/run_combo.py`
|
||
|
||
当前直接逻辑：
|
||
|
||
- `find_meme_xml()` 优先固定 `motif/analysis/meme_out/meme.xml`，`run_combo.py:36-39`。
|
||
- `find_latest_manifest(workspace, stage, prefix)` 实际 glob `*{stage}*.manifest.json`，忽略 prefix，`run_combo.py:56-62`。
|
||
- phylogeny/structure/domain 都通过 latest manifest 获取，`run_combo.py:166-224`。
|
||
- domtblout 从 domain manifest artifact 获取。
|
||
- 输出 `final/{prefix}.combo.manifest.json`。
|
||
|
||
关联代码：
|
||
|
||
- backend combo 只检查 prefix manifest 存在，然后调用 CLI。
|
||
|
||
主要风险：
|
||
|
||
- 这是当前最容易串结果的阶段。
|
||
- 即使 backend 检查 prefix manifest，CLI 仍可能读取别的最新 manifest。
|
||
- meme XML 仍是固定目录。
|
||
|
||
### 3.8 下游阶段概览
|
||
|
||
需要后续逐个迁移：
|
||
|
||
- `pep`: 读 `final/{prefix}.final_members.pep.fa`，应改为 scope FASTA。
|
||
- `promoter`: 读单物种 genome/GFF/gene ID，scope 多物种时需 per-species。
|
||
- `synteny`: 偏单物种，backend 当前 gene ID 查找不够稳。
|
||
- `interspecies_synteny`: 已经有较好的 target/ref 物种选择和 config groups，可作为迁移模板。
|
||
- `karyotype`: 已支持按 config 物种循环，但 family ID 仍来自全局 final members。
|
||
- `expression`: 目前从 input/final glob gene ID，容易拿错；表达应明确单物种或显式选择表达物种。
|
||
- `ppi/kaks/ortholog`: 当前多依赖 prefix 和 interspecies outputs，需要等 scope 底座后再改。
|
||
|
||
## 4. 真实测试数据现状
|
||
|
||
已确认可用文件：
|
||
|
||
- `input/job_1779115921147_d4e0a7/config.xlsx`
|
||
- `input/job_1779115921147_d4e0a7/Lc.pep.fasta`
|
||
- `input/job_1779115921147_d4e0a7/Dl.pep.fasta`
|
||
- `input/job_1779115921147_d4e0a7/Os.pep.fasta`
|
||
- `input/job_1779115921147_d4e0a7/At.SPL.pep.fa`
|
||
- `input/job_1779115921147_d4e0a7/At.SPL.subfamily`
|
||
- `input/job_1779115921147_d4e0a7/Lc_final.gff3`
|
||
- `input/job_1779115921147_d4e0a7/Dl_final.gff3`
|
||
- `input/job_1779115921147_d4e0a7/Os_final.gff3`
|
||
- `input/job_1779115921147_d4e0a7/Ref_combined.SPL.pep.fasta`
|
||
- `workspace/job_1779022508209_52539d/final/At.SPL.ncbi_fetch.protein.fa`
|
||
|
||
已有历史结果：
|
||
|
||
- `workspace/job_1779115921147_d4e0a7/final/Multi.SPL.genome_prepare.manifest.json`
|
||
- `workspace/job_1779115921147_d4e0a7/final/Target.SPL.identify.manifest.json`
|
||
- `workspace/job_1779115921147_d4e0a7/final/Lc.identify.manifest.json`
|
||
|
||
后续真实测试不能只检查编译，必须检查：
|
||
|
||
- CLI exit code。
|
||
- manifest `status == success`。
|
||
- manifest `scope_id` 与本次选择一致。
|
||
- 输入 FASTA、tree tips、gene IDs、GFF IDs 的集合一致。
|
||
- target/reference/comparative 是否按本阶段定义参与。
|
||
- 不同 scope 的中间目录不互相覆盖。
|
||
- 故意制造 scope 不一致时，stage 必须拒绝继续。
|
||
|
||
## 5. 修订总体原则
|
||
|
||
1. `prefix` 只作为兼容输出前缀，不再承载完整分析范围。
|
||
2. 新增 `scope` 作为 stage 输入和 manifest 必填字段。
|
||
3. 每个阶段只读同一 scope 的上游 manifest。
|
||
4. 中间目录必须 scope 隔离。
|
||
5. 兼容旧文件名，但新逻辑优先读 scope manifest。
|
||
6. 一个阶段改完并通过真实数据测试后，再改下一个阶段。
|
||
7. 前端使用原有 Element Plus 弹窗技术，不换新框架。
|
||
8. config.xlsx 的 `Analysis!Family` 是 family 的优先来源；没有则让用户填写。
|
||
9. 默认示例使用 `At.WRKY`，不再全局默认 `Ft.SPL` 或 `Multi.SPL`。
|
||
|
||
## 6. 分阶段修订计划
|
||
|
||
### Phase 0: Scope 底座
|
||
|
||
目标：
|
||
|
||
- 建立统一 scope 解析、写入、校验能力。
|
||
- 不改变阶段生物学结果，只先让所有阶段能接收/记录 scope。
|
||
|
||
拟修改文件：
|
||
|
||
- 新增 `gfa_core/gene_family_scope.py`
|
||
- 修改 `apps/web/vue3/main.py`
|
||
- 修改 `apps/web/pipeline_backend.py`
|
||
- 修改 `gfa_core/mas/tools/stage_status.py`
|
||
- 修改 `gfa_core/reproducibility_v2.py`
|
||
- 修改 `gfa_core/schemas/core/workspace_contract_v1.md`
|
||
|
||
核心能力：
|
||
|
||
- 从 `config.xlsx` 读取 family 和物种角色。
|
||
- 生成 `scope_id`。
|
||
- 生成 stage input manifest。
|
||
- 校验 upstream manifest 的 scope 是否匹配。
|
||
- 保留 canonical manifest，同时写 scope manifest。
|
||
|
||
真实测试：
|
||
|
||
- 在 `job_1779115921147_d4e0a7` 解析出 family `SPL`。
|
||
- 解析出 target `Lc,Dl`、reference `At`、comparative `Os`。
|
||
- 生成 scope JSON 后，检查所有文件存在且 hash 可计算。
|
||
|
||
通过标准：
|
||
|
||
- scope JSON 非空。
|
||
- species roles 正确。
|
||
- `prefix` 自动生成为 `Multi.SPL` 或由兼容逻辑给出，但不要求用户理解 prefix。
|
||
|
||
### Phase 1: Identify
|
||
|
||
目标：
|
||
|
||
- identify 只鉴定用户选择的 target species。
|
||
- manifest 写入 scope。
|
||
- 输出 per-species artifacts 和 scope artifacts。
|
||
|
||
拟修改文件：
|
||
|
||
- `apps/cli/run_identify.py`
|
||
- `apps/web/pipeline_backend.py`
|
||
- `apps/web/vue3/index.html`
|
||
- `gfa_core/schemas/stages/identify_contract_v1.md`
|
||
- `skills/gene-identify/SKILL.md`
|
||
|
||
真实测试命令形态：
|
||
|
||
```bash
|
||
/home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_identify.py \
|
||
--input-dir input/job_1779115921147_d4e0a7 \
|
||
--workspace workspace/<test_job> \
|
||
--prefix Multi.SPL \
|
||
--target-species Lc,Dl \
|
||
--required-pfams PF03110 \
|
||
--cpu 4
|
||
```
|
||
|
||
通过标准：
|
||
|
||
- final members 只包含 `Lc,Dl`。
|
||
- 不包含 `At` 和 `Os`。
|
||
- `PF03110` 被记录到 parameters。
|
||
- manifest 含 `scope_id`、`target_species=["Lc","Dl"]`。
|
||
|
||
### Phase 2: Phylogeny
|
||
|
||
目标：
|
||
|
||
- 支持用户选择建树范围。
|
||
- tree/msa 文件和 manifest 记录 scope。
|
||
- 不同建树范围不互相覆盖。
|
||
|
||
拟修改文件：
|
||
|
||
- `apps/cli/run_phylogeny.py`
|
||
- `apps/web/pipeline_backend.py`
|
||
- `apps/web/vue3/index.html`
|
||
- `gfa_core/schemas/stages/phylogeny_contract_v1.md`
|
||
- `skills/phylogeny/SKILL.md`
|
||
|
||
测试矩阵：
|
||
|
||
- target only: `Lc,Dl`
|
||
- target + reference: `Lc,Dl,At`
|
||
- target + comparative + reference: `Lc,Dl,Os,At`
|
||
|
||
通过标准：
|
||
|
||
- tree tips 数量等于 scope FASTA 记录数。
|
||
- manifest 的 upstream identify scope 匹配。
|
||
- 不同 scope 输出在不同中间目录。
|
||
|
||
### Phase 3: Reference Classification
|
||
|
||
目标：
|
||
|
||
- 由 selected target species + reference species 驱动分类。
|
||
- 支持 `Lc,Dl` 多目标物种。
|
||
- At 只作为参考，不写入目标 assignment。
|
||
|
||
拟修改文件：
|
||
|
||
- `apps/cli/run_reference_classification.py`
|
||
- `apps/web/pipeline_backend.py`
|
||
- `apps/web/vue3/index.html`
|
||
- `gfa_core/schemas/stages/reference_classification_contract_v1.md`
|
||
|
||
通过标准：
|
||
|
||
- assignment/subfamily 目标行只包含 `Lc,Dl`。
|
||
- reference evidence 来自 `At.SPL.subfamily`。
|
||
- 如果 tree scope 不含 At 且需要 At，必须构建 helper tree 并记录。
|
||
|
||
### Phase 4: Motif
|
||
|
||
目标：
|
||
|
||
- MEME 输出目录 scope 隔离。
|
||
- 复用 MEME 前校验 FASTA hash、参数、scope_id。
|
||
|
||
拟修改文件：
|
||
|
||
- `apps/cli/run_motif.py`
|
||
- `apps/cli/run_combo.py`
|
||
- `gfa_core/schemas/stages/motif_contract_v1.md`
|
||
- `skills/motif-analysis/SKILL.md`
|
||
|
||
通过标准：
|
||
|
||
- `Lc,Dl` 和 `Lc` 两次运行生成不同 meme_out。
|
||
- 修改 scope 后不会复用旧 `meme.xml`。
|
||
|
||
### Phase 5: Domain
|
||
|
||
目标：
|
||
|
||
- hmmscan 输出 scope 隔离。
|
||
- 去除 latest manifest 跨 prefix 风险。
|
||
|
||
拟修改文件：
|
||
|
||
- `apps/cli/run_domain.py`
|
||
- `apps/cli/run_combo.py`
|
||
- `gfa_core/schemas/stages/domain_contract_v1.md`
|
||
- `skills/domain-analysis/SKILL.md`
|
||
|
||
通过标准：
|
||
|
||
- `domain/analysis/{scope_id}/results.domtblout` 存在。
|
||
- manifest domtblout path 指向 scope 目录。
|
||
- 故意放入其他 prefix manifest 时不会被读取。
|
||
|
||
### Phase 6: Structure
|
||
|
||
目标：
|
||
|
||
- 多目标物种时按物种 GFF 过滤，再合并绘图输入或输出分物种结构图。
|
||
- tree tips、gene ID、GFF gene ID 做一致性检查。
|
||
|
||
拟修改文件：
|
||
|
||
- `apps/cli/run_structure.py`
|
||
- `apps/web/pipeline_backend.py`
|
||
- `gfa_core/schemas/stages/structure_contract_v1.md`
|
||
- `skills/structure/SKILL.md`
|
||
|
||
通过标准：
|
||
|
||
- Lc/Dl 的结构分别从 `Lc_final.gff3`、`Dl_final.gff3` 解析。
|
||
- 缺失 ID 报告到 metrics，不静默失败。
|
||
|
||
### Phase 7: Combo
|
||
|
||
目标：
|
||
|
||
- 只合并同一 scope 的 phylogeny/motif/domain/structure。
|
||
- 不再从固定 `meme_out` 或 latest manifest 取文件。
|
||
|
||
拟修改文件：
|
||
|
||
- `apps/cli/run_combo.py`
|
||
- `apps/web/pipeline_backend.py`
|
||
- `gfa_core/schemas/stages/combo_contract_v1.md`
|
||
|
||
通过标准：
|
||
|
||
- 同 scope 正常出组合图。
|
||
- scope 不一致时明确失败并列出不一致 stage。
|
||
|
||
### Phase 8: 下游阶段
|
||
|
||
迁移顺序：
|
||
|
||
1. `pep`
|
||
2. `promoter`
|
||
3. `synteny`
|
||
4. `karyotype`
|
||
5. `interspecies_synteny`
|
||
6. `expression`
|
||
7. `ortholog`
|
||
8. `ppi`
|
||
9. `kaks`
|
||
10. `summary`
|
||
|
||
每个阶段独立改、独立真实测试。
|
||
|
||
## 7. 前端统一方案
|
||
|
||
目标：
|
||
|
||
- prefix 不再作为主输入。
|
||
- 弹窗使用原有 Element Plus dialog/form/checkbox 技术。
|
||
- 所有基因家族阶段都显示统一的 species role selector。
|
||
- family 从 config.xlsx 自动填；没有则留空让用户填。
|
||
- 倒计时全局 40 秒，并增加暂停按钮。
|
||
|
||
前端字段：
|
||
|
||
- `confirmParams.family`
|
||
- `confirmParams.scopeMode`
|
||
- `confirmParams.targetSpecies`
|
||
- `confirmParams.referenceSpecies`
|
||
- `confirmParams.comparativeSpecies`
|
||
- `confirmParams.prefixAdvanced`
|
||
- `confirmParams.countdownPaused`
|
||
|
||
scopeMode 建议选项：
|
||
|
||
- `target_only`
|
||
- `target_plus_reference`
|
||
- `target_plus_comparative`
|
||
- `target_plus_reference_plus_comparative`
|
||
- `custom`
|
||
|
||
## 8. 诊断/修复 agent 后续定位
|
||
|
||
当前诊断 agent 的主要问题：
|
||
|
||
- 它看 log 和 manifest，但没有 stage-specific 可执行修复策略。
|
||
- 它不能主动修代码。
|
||
- 它没有把这次修订逻辑固化成规则。
|
||
|
||
后续需要给诊断 agent 增加“可修复动作库”：
|
||
|
||
- `scope_mismatch`: 建议重新生成 stage scope input，或阻止下游执行。
|
||
- `wrong_manifest_scope`: 指向正确 manifest 或要求重跑上游。
|
||
- `fixed_intermediate_collision`: 清理/隔离旧中间目录后重跑。
|
||
- `missing_reference_family_pep`: 从 NCBI fetch 结果或 `At.SPL.pep.fa` 建立兼容软链接。
|
||
- `protein_stop_marker`: 调用已有 clean fasta 逻辑。
|
||
- `id_mismatch`: 运行 ID normalize/映射检查，不直接猜测替换。
|
||
|
||
这些动作要写入代码和文档，不能只停留在聊天说明。
|
||
|
||
## 9. 逐项执行记录
|
||
|
||
后续每完成一个阶段，在这里追加：
|
||
|
||
```text
|
||
Phase:
|
||
Modified files:
|
||
Real test command:
|
||
Real test data:
|
||
Result:
|
||
Manifest:
|
||
Artifacts:
|
||
Known residual risks:
|
||
Next phase gate:
|
||
```
|
||
|
||
当前状态：
|
||
|
||
- 文档创建完成。
|
||
- Phase 0 scope 解析模块已开始落地。
|
||
|
||
```text
|
||
Phase: 0 Scope 底座
|
||
Modified files:
|
||
- gfa_core/gene_family_scope.py
|
||
- apps/cli/resolve_scope.py
|
||
- docs/gene_family_scope_refactor_plan.md
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/gene_family_scope.py apps/cli/resolve_scope.py
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/resolve_scope.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/job_1779115921147_d4e0a7 --prefix Multi.SPL --family SPL --target-species Lc,Dl --reference-species At --comparative-species Os --scope-mode target_plus_reference_plus_comparative
|
||
Real test data:
|
||
- input/job_1779115921147_d4e0a7/config.xlsx
|
||
- input/job_1779115921147_d4e0a7/At.SPL.pep.fa
|
||
- Lc/Dl/Os pep/gff/genome rows from config.xlsx
|
||
Result:
|
||
- scope_id = SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference_plus_comparative
|
||
- family = SPL
|
||
- target_species = Lc,Dl
|
||
- reference_species = At
|
||
- comparative_species = Os
|
||
- selected_species = Lc,Dl,At,Os
|
||
Artifacts:
|
||
- workspace/job_1779115921147_d4e0a7/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference_plus_comparative.scope.json
|
||
Known residual risks:
|
||
- At whole-genome pep/cds rows did not resolve in this job, but At.SPL family_pep resolved and is sufficient for reference classification.
|
||
- Existing stages do not yet consume scope JSON.
|
||
Next phase gate:
|
||
- Identify must accept --scope-file, default target species from scope, and write scope fields into manifest.
|
||
```
|
||
|
||
```text
|
||
Phase: 1 Identify scope 接入
|
||
Modified files:
|
||
- apps/cli/run_identify.py
|
||
- apps/web/pipeline_backend.py
|
||
- apps/web/vue3/main.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- docs/gene_family_scope_refactor_plan.md
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_identify.py apps/web/pipeline_backend.py apps/web/vue3/main.py gfa_core/mas/tools/stage_tools.py
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/resolve_scope.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/codex_scope_identify_test2 --prefix Multi.SPL --family SPL --target-species Lc,Dl --reference-species At --comparative-species Os --scope-mode target_only --output workspace/codex_scope_identify_test2/final/Multi.SPL.identify_scope.json
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_identify.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/codex_scope_identify_test2 --prefix Multi.SPL --scope-file workspace/codex_scope_identify_test2/final/Multi.SPL.identify_scope.json --required-pfams PF03110 --cpu 4 --skip-ref-warning
|
||
- Direct backend run_stage('identify') with scope_file in workspace/codex_scope_backend_identify_test.
|
||
Real test data:
|
||
- target: Lc,Dl
|
||
- reference: At
|
||
- comparative: Os
|
||
- family: SPL / PF03110
|
||
Result:
|
||
- CLI identify status = success
|
||
- CLI target_species = Lc,Dl
|
||
- CLI selected_species = Lc,Dl
|
||
- CLI final members = 55 total; Lc=33, Dl=22, At-like=0, Os-like=0
|
||
- CLI InterProScan status = success
|
||
- Backend identify status = success
|
||
- Backend final members = 59 total; Lc=37, Dl=22, At-like=0, Os-like=0
|
||
- Backend/CLI count difference is explained by parameter difference: backend default blast_evalue is 1e-5, CLI test used 1e-7.
|
||
Artifacts:
|
||
- workspace/codex_scope_identify_test2/final/Multi.SPL.identify.manifest.json
|
||
- workspace/codex_scope_identify_test2/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_only.identify.manifest.json
|
||
- workspace/codex_scope_identify_test2/final/Multi.SPL.final_members.id
|
||
- workspace/codex_scope_identify_test2/final/Multi.SPL.final_members.pep.fa
|
||
- workspace/codex_scope_backend_identify_test/final/Multi.SPL.identify.manifest.json
|
||
Known residual risks:
|
||
- Frontend does not yet generate scope JSON automatically.
|
||
- stage_status still checks prefix manifest only; scope-aware completed checks are not done.
|
||
- Downstream stages still ignore scope.
|
||
Next phase gate:
|
||
- Phylogeny must accept scope/stage FASTA and ensure tree tips match selected species.
|
||
```
|
||
|
||
```text
|
||
Phase: 2 Phylogeny scope 接入
|
||
Modified files:
|
||
- apps/cli/run_phylogeny.py
|
||
- apps/web/pipeline_backend.py
|
||
- docs/gene_family_scope_refactor_plan.md
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_phylogeny.py apps/web/pipeline_backend.py
|
||
- target-only CLI:
|
||
/home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_phylogeny.py --workspace workspace/codex_scope_phy_target_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_target_fast_test/final/Multi.SPL.phylogeny_target_scope.json --cpu 4 --iqtree-params '-m LG+G -nt 4' --tree-layout rectangular
|
||
- target+reference CLI:
|
||
/home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_phylogeny.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --cpu 4 --iqtree-params '-m LG+G -nt 4' --tree-layout rectangular
|
||
- backend run_stage('phylogeny') with scope_file:
|
||
workspace/codex_scope_backend_phy_test
|
||
Real test data:
|
||
- target-only: Lc,Dl SPL members from real identify output
|
||
- target+reference: Lc,Dl SPL members + At.SPL.pep.fa
|
||
Result:
|
||
- target-only CLI status = success
|
||
- target-only CLI scope counts = Lc 33, Dl 22
|
||
- target-only CLI scope_sequence_count = 55
|
||
- target-only CLI MSA count = 55
|
||
- target-only CLI tree tips = 55
|
||
- target+reference CLI status = success
|
||
- target+reference CLI scope counts = Lc 33, Dl 22, At 17
|
||
- target+reference CLI scope_sequence_count = 72
|
||
- target+reference CLI MSA count = 72
|
||
- target+reference CLI tree tips = 72
|
||
- backend phylogeny status = success
|
||
- backend phylogeny scope counts = Lc 33, Dl 22
|
||
- backend phylogeny MSA/tree tips = 55/55
|
||
Artifacts:
|
||
- workspace/codex_scope_phy_target_fast_test/final/Multi.SPL.phylogeny.manifest.json
|
||
- workspace/codex_scope_phy_target_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_only.phylogeny.manifest.json
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny.manifest.json
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.phylogeny.manifest.json
|
||
- workspace/codex_scope_backend_phy_test/final/Multi.SPL.phylogeny.manifest.json
|
||
Known residual risks:
|
||
- Frontend does not yet generate scope JSON automatically.
|
||
- target+comparative+reference requires Os family members. Current behavior should fail clearly unless Os has already been identified or Species.Family.pep.fa is provided.
|
||
- The backend still performs stop-marker cleanup on canonical final_members, while scope FASTA currently uses per-species artifacts directly. A later shared FASTA sanitizer should be applied to scope-built FASTA before all sequence stages.
|
||
- MFP model selection on target+reference took too long in a validation run and was stopped; fixed-model LG+G was used for scope verification. The production default remains configurable.
|
||
Next phase gate:
|
||
- Reference classification must consume selected target/reference species from scope and must not classify At as target.
|
||
```
|
||
|
||
```text
|
||
Phase: 3 Reference classification scope 接入
|
||
Modified files:
|
||
- apps/cli/run_reference_classification.py
|
||
- apps/web/pipeline_backend.py
|
||
- docs/gene_family_scope_refactor_plan.md
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_reference_classification.py apps/web/pipeline_backend.py
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_reference_classification.py --workspace workspace/codex_scope_phy_ref_fast_test --input-dir input/job_1779115921147_d4e0a7 --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --reference-subfamily input/job_1779115921147_d4e0a7/At.SPL.subfamily --reference-prefix At --overwrite-gene-subfamily
|
||
- Direct backend run_stage('reference_classification') with the same scope_file/reference_subfamily_file.
|
||
Real test data:
|
||
- Joint tree from Phase 2 target+reference: Lc,Dl + At, 72 tips
|
||
- Reference subfamily table: input/job_1779115921147_d4e0a7/At.SPL.subfamily
|
||
Result:
|
||
- CLI reference_classification status = success
|
||
- Backend reference_classification status = success
|
||
- assignment rows = 55
|
||
- assignment target composition = Lc 33, Dl 22, At 0
|
||
- subfamily rows = 72, retaining At 17 reference rows plus Lc/Dl rows for joint-tree plotting
|
||
- manifest scope_id = SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference
|
||
- manifest target_species = Lc,Dl; reference_species = At; comparative_species = Os
|
||
- scoped manifest exists.
|
||
Artifacts:
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.assignment.tsv
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.subfamily.tsv
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.confidence.tsv
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.manifest.json
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.reference_classification.manifest.json
|
||
Known residual risks:
|
||
- The internal tree script still accepts a single target_prefix and reference_prefix; the wrapper now filters assignment rows by scope target IDs, but a future cleanup should move scope awareness into scripts/reference_classification_from_tree.py directly.
|
||
- The compatibility file final/Multi.SPL.geneID.subfamily intentionally includes At reference rows when the classification scope includes reference species. Downstream plotting stages must use scope-aware filtering if they need target-only plots.
|
||
- Frontend still needs to generate/pass scope JSON from the target/comparative/reference checklist automatically.
|
||
Next phase gate:
|
||
- Structure/motif/domain/combo must consume scope/subfamily consistently and reuse the scoped reference classification outputs without rerunning unrelated species.
|
||
```
|
||
|
||
```text
|
||
Phase: 4 Structure scope 接入
|
||
Modified files:
|
||
- apps/cli/run_structure.py
|
||
- apps/web/pipeline_backend.py
|
||
- docs/gene_family_scope_refactor_plan.md
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_structure.py apps/web/pipeline_backend.py
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_structure.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --subfamily workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.subfamily.tsv --cpu 4
|
||
- Direct backend run_stage('structure') with the same scope_file.
|
||
Real test data:
|
||
- target GFF3 from scope: Lc=Lchinesis_genome.Chr.gff3.gz, Dl=Longan.v20190918.gff3.gz
|
||
- target IDs: final/Multi.SPL.final_members.id, 55 IDs
|
||
- tree/subfamily: Lc,Dl + At reference classification from Phase 2/3
|
||
Result:
|
||
- CLI structure status = success
|
||
- Backend structure status = success
|
||
- gzip GFF3 input is supported
|
||
- scanned GFF3 lines = 1,260,473
|
||
- filtered GFF3 rows = 353
|
||
- genes_with_structure = 55/55
|
||
- structure TSV ID check: final_ids=55, tsv_ids=55, missing=0, extra=0
|
||
- species composition in structure TSV: Lc 33, Dl 22
|
||
- structure PNG/PDF generated.
|
||
Artifacts:
|
||
- workspace/codex_scope_phy_ref_fast_test/structure/analysis/Multi.SPL.structure.filtered.gff3
|
||
- workspace/codex_scope_phy_ref_fast_test/structure/analysis/Multi.SPL.structure.tsv
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.structure.png
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.structure.pdf
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.structure.manifest.json
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.structure.manifest.json
|
||
Known residual risks:
|
||
- The structure plot uses a joint 72-tip tree but structure features only exist for target species Lc/Dl. This is intentional for target structure visualization after reference-guided classification, but downstream combo needs explicit scope-aware panel rules.
|
||
- run_structure.py still writes the first GFF3 path in the legacy cmd field for compatibility; the authoritative source list is parameters.gff3_sources.
|
||
Next phase gate:
|
||
- Motif must use scoped target proteins and scoped subfamily consistently, and must not silently include reference/comparative proteins unless the selected scope requests them.
|
||
```
|
||
|
||
```text
|
||
Phase: 5 Motif scope 接入
|
||
Modified files:
|
||
- apps/cli/run_motif.py
|
||
- apps/web/pipeline_backend.py
|
||
- docs/gene_family_scope_refactor_plan.md
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_motif.py apps/web/pipeline_backend.py
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_motif.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --subfamily workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.subfamily.tsv --nmotifs 3 --minw 6 --maxw 100 --cpu 4
|
||
- Direct backend run_stage('motif') with the same scope_file and motif parameters.
|
||
Real test data:
|
||
- scoped protein FASTA from phylogeny: phylogeny/analysis/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.input.pep.fa
|
||
- sequence composition: Lc,Dl target + At reference, 72 proteins
|
||
- subfamily file: final/Multi.SPL.reference_classification.subfamily.tsv
|
||
Result:
|
||
- CLI motif status = success
|
||
- Backend motif status = success
|
||
- MEME ran on 72 scoped proteins, not target-only final_members
|
||
- motif_count = 3
|
||
- total_sites = 109
|
||
- motif plot PNG/PDF generated
|
||
- scoped manifest exists
|
||
- MEME now records input FASTA hash and motif parameters in motif/analysis/meme_out/run_signature.json; existing results are reused only when the signature matches.
|
||
Artifacts:
|
||
- workspace/codex_scope_phy_ref_fast_test/motif/analysis/meme_out/meme.xml
|
||
- workspace/codex_scope_phy_ref_fast_test/motif/analysis/meme_out/run_signature.json
|
||
- workspace/codex_scope_phy_ref_fast_test/motif/analysis/Multi_SPL.motif_prot.txt
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.motif.png
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.motif.pdf
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.motif.manifest.json
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.motif.manifest.json
|
||
Known residual risks:
|
||
- The motif analysis directory is still the legacy shared path motif/analysis. The new MEME signature prevents stale MEME reuse, but future multi-scope parallel execution should use per-scope analysis subdirectories.
|
||
- If MEME reruns, the summary file is now regenerated; older outputs from pre-signature runs may be replaced during the first post-fix execution.
|
||
Next phase gate:
|
||
- Domain must use the same scoped protein FASTA/subfamily rules as motif and should avoid stale hmmscan outputs when scope or thresholds change.
|
||
```
|
||
|
||
```text
|
||
Phase: 6 Domain scope 接入
|
||
Modified files:
|
||
- apps/cli/run_domain.py
|
||
- apps/web/pipeline_backend.py
|
||
- docs/gene_family_scope_refactor_plan.md
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_domain.py apps/web/pipeline_backend.py
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_domain.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --subfamily workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.subfamily.tsv --evalue 1e-5 --cpu 4
|
||
- Direct backend run_stage('domain') with the same scope_file.
|
||
Real test data:
|
||
- scoped protein FASTA from phylogeny: 72 proteins, Lc/Dl target + At reference
|
||
- Pfam database: /home/xia-lab/00.share_data/hmm/Pfam-A.hmm
|
||
Result:
|
||
- CLI domain status = success
|
||
- Backend domain status = success
|
||
- hmmscan elapsed = 38.3 s in CLI test
|
||
- proteins_total = 72
|
||
- proteins_with_domains = 63
|
||
- significant_domain_hits = 84
|
||
- unique_domain_types = 8
|
||
- top domain = SBP, 50 hits
|
||
- domain PNG/PDF generated
|
||
- scoped manifest exists.
|
||
Artifacts:
|
||
- workspace/codex_scope_phy_ref_fast_test/domain/analysis/results.domtblout
|
||
- workspace/codex_scope_phy_ref_fast_test/domain/analysis/results.tblout
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.domain.png
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.domain.pdf
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.domain.manifest.json
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.domain.manifest.json
|
||
Known residual risks:
|
||
- hmmscan outputs still use the legacy shared domain/analysis/results.* filenames. This is safe for serial execution but not for two scopes running in parallel in the same workspace.
|
||
- Domain stage does not cache by input signature; it reruns hmmscan, which avoids stale output but costs time.
|
||
Next phase gate:
|
||
- Combo must read scoped manifests/artifacts from phylogeny, motif, structure and domain instead of relying on the latest manifest glob alone.
|
||
```
|
||
|
||
```text
|
||
Phase: 7 Combo scope 接入
|
||
Modified files:
|
||
- apps/cli/run_combo.py
|
||
- apps/web/pipeline_backend.py
|
||
- docs/gene_family_scope_refactor_plan.md
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_combo.py apps/web/pipeline_backend.py
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_combo.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --subfamily workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.reference_classification.subfamily.tsv --evalue 1e-5
|
||
- Direct backend run_stage('combo') with the same scope_file.
|
||
Real test data:
|
||
- scoped phylogeny, motif, structure and domain manifests from Phases 2, 5, 4 and 6
|
||
- subfamily file from Phase 3
|
||
Result:
|
||
- CLI combo status = success
|
||
- Backend combo status = success
|
||
- dependency check uses scoped manifests when scope_file is supplied
|
||
- combo PDF/PNG generated
|
||
- scoped combo manifest exists.
|
||
Artifacts:
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.combo_tree_motif_structure_domain.pdf
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.combo_tree_motif_structure_domain.png
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.combo.manifest.json
|
||
- workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.combo.manifest.json
|
||
Known residual risks:
|
||
- run_combo.py still finds meme.xml by legacy workspace search because motif manifest does not expose meme.xml as an artifact. The current test is correct for serial execution; a later cleanup should add meme_xml to motif artifacts and make combo consume it from the scoped motif manifest.
|
||
- Combo output filename remains legacy prefix-based, so concurrent scopes in one workspace can overwrite each other. Scoped manifest records which scope produced the current plot.
|
||
Next phase gate:
|
||
- Frontend/router needs to generate/pass scope_file automatically from target/reference/comparative choices; status/diagnostic/repair should become scope-aware.
|
||
```
|
||
|
||
```text
|
||
Phase: 8 Frontend/API scope 自动传递与状态检查
|
||
Modified files:
|
||
- apps/web/vue3/main.py
|
||
- apps/web/vue3/index.html
|
||
- gfa_core/mas/tools/stage_status.py
|
||
- tasks.py
|
||
- docs/gene_family_scope_refactor_plan.md
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/web/vue3/main.py gfa_core/mas/tools/stage_status.py tasks.py
|
||
- Direct API helper:
|
||
get_gene_family_scope_options(job_id='job_1779115921147_d4e0a7', prefix='Multi.SPL')
|
||
- Direct API helper:
|
||
_ensure_gene_family_scope_file(job_id='job_1779115921147_d4e0a7', prefix='Multi.SPL', stage='motif', target=Lc,Dl, reference=At, comparative=Os, scopeMode=target_plus_reference)
|
||
- Direct /api/execute function call:
|
||
execute_stage(ExecuteRequest(stage='phylogeny', job_id='codex_scope_phy_ref_fast_test', prefix='Multi.SPL', scopeFile='workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json'))
|
||
- Direct /api/stage-status function call:
|
||
get_stage_status(job_id='codex_scope_phy_ref_fast_test', stage='combo', prefix='Multi.SPL', scope_id='SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference')
|
||
Real test data:
|
||
- input/job_1779115921147_d4e0a7/config.xlsx
|
||
- workspace/codex_scope_phy_ref_fast_test scoped manifests from Phases 2-7
|
||
Result:
|
||
- /api/gene-family/scope-options returns family=SPL, target_species=Lc,Dl, reference_species=At, comparative_species from config.
|
||
- _ensure_gene_family_scope_file writes final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.scope.json.
|
||
- /api/execute with an existing scopeFile checks scoped phylogeny manifest and skips as already_completed instead of using the unscoped manifest.
|
||
- /api/stage-status with scope_id returns completed=True for scoped combo.
|
||
- Element Plus confirmation dialogs now expose analysis range as Target / Target+Reference / Target+Comparative / All, with checkboxes for target/reference/comparative species.
|
||
- Countdown defaults are now 40 s and include a "暂停倒计时" button for normal, batch and per-stage confirmation dialogs.
|
||
Known residual risks:
|
||
- The browser UI was validated by code-path inspection and backend function tests, not by an automated Playwright click-through in a live browser.
|
||
- Comparative species are listed from config; they are not auto-selected unless the selected scope mode includes comparative or upstream stage params specify them.
|
||
- Natural-language router still needs deeper prompt/rule cleanup so it consistently sets scopeMode when the user says target/comparative/reference in free text.
|
||
Next phase gate:
|
||
- Diagnostic/repair agent should get explicit gene-family repair playbooks for ID mismatches, stop codons, missing/ambiguous GFF/PEP/HMM inputs, and scoped manifest/final mismatch.
|
||
```
|
||
|
||
```text
|
||
Phase: 9 Diagnostic/repair agent scope 接入与 stop-marker 真实修复
|
||
Modified files:
|
||
- gfa_core/mas/agents/diagnosis.py
|
||
- apps/cli/run_phylogeny.py
|
||
- apps/cli/run_motif.py
|
||
- apps/cli/run_domain.py
|
||
- docs/gene_family_scope_refactor_plan.md
|
||
|
||
Code relationship summary:
|
||
- tasks.py calls DiagnosisAgent.diagnose_stage_gate() before and after StageToolRegistry execution.
|
||
- diagnose_stage_gate() previously searched only final/{prefix}.{stage}.manifest.json or latest unscoped manifests, then checked legacy final_members files.
|
||
- Scoped stages now pass params.scopeFile from frontend/API/backend; DiagnosisAgent reads scope_id from scopeFile and loads final/{prefix}.{scope_id}.{stage}.manifest.json first.
|
||
- Protein-consuming stages now resolve protein input from scoped manifest parameters: protein_fasta, protein_source and scope_input_fasta.
|
||
- GFF/member stages now read target GFF paths from scopeFile when available instead of guessing Multi_final.gff3 or the first final/*.gff3.
|
||
- structure/combo no longer trigger protein stop-marker checks because they do not directly consume protein FASTA.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_motif.py apps/cli/run_domain.py apps/cli/run_phylogeny.py gfa_core/mas/agents/diagnosis.py
|
||
- Diagnosis preflight repair:
|
||
DiagnosisAgent.diagnose_stage_gate('codex_scope_phy_ref_fast_test','motif','Multi.SPL', gate='preflight', params={'scopeFile': 'workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json'})
|
||
DiagnosisAgent.apply_runtime_repairs(...)
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_domain.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --cpu 4
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_motif.py --workspace workspace/codex_scope_phy_ref_fast_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --nmotifs 3 --minw 6 --maxw 100 --cpu 4
|
||
- Postflight gate for motif/domain/structure/combo with the same scopeFile.
|
||
|
||
Real test data:
|
||
- scope: SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference
|
||
- protein input before repair: workspace/codex_scope_phy_ref_fast_test/phylogeny/analysis/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.input.pep.fa
|
||
- protein input after repair: workspace/codex_scope_phy_ref_fast_test/phylogeny/analysis/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.input.pep.clean.fa
|
||
|
||
Result:
|
||
- Diagnosis preflight detected protein_fasta_stop_markers on the real scoped FASTA.
|
||
- apply_runtime_repairs generated the clean scoped FASTA and a second preflight returned pass with no runtime repair candidates.
|
||
- domain CLI used input.pep.clean.fa, completed hmmscan successfully, and produced 72 proteins / 84 significant hits / 8 unique domains.
|
||
- motif CLI used input.pep.clean.fa, reran MEME because the input hash changed, produced 3 motifs / 109 sites, and generated PNG/PDF plus scoped manifest.
|
||
- postflight diagnosis results:
|
||
motif pass ok []
|
||
domain pass ok []
|
||
structure pass ok []
|
||
combo pass ok []
|
||
|
||
Behavioral fixes:
|
||
- Scoped manifest/final mismatch no longer causes diagnosis to inspect stale unscoped manifests first.
|
||
- Stop-marker repair is now job-local and actually affects downstream CLI inputs.
|
||
- ID/GFF low-overlap is treated as a reportable job-local diagnosis; the agent can write a diagnosis report, but it does not silently rewrite biological IDs.
|
||
- Plot-only combo no longer warns about empty metrics.
|
||
|
||
Known residual risks:
|
||
- The natural-language router still needs a separate pass so free text consistently maps to scopeMode and selected species.
|
||
- ID mismatch repair remains intentionally conservative: automatic semantic ID rewriting needs an explicit mapping rule or user confirmation.
|
||
- Motif/domain still write legacy shared analysis directories; serial real-data execution is validated, but parallel same-workspace multi-scope runs can still overwrite intermediate files.
|
||
Next phase gate:
|
||
- Router/plan/skill/contract cleanup should make scope explicit in natural-language planning, stage contracts and reproducibility records.
|
||
```
|
||
|
||
```text
|
||
Phase: 10 Router/plan/skill/contract/repro scope 清理
|
||
Modified files:
|
||
- gfa_core/mas/utils/intent_parser.py
|
||
- gfa_core/mas/core/fast_intent.py
|
||
- apps/web/vue3/main.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/reproducibility_v2.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/schemas/stages/identify_contract_v1.md
|
||
- gfa_core/schemas/stages/phylogeny_contract_v1.md
|
||
- gfa_core/schemas/stages/motif_contract_v1.md
|
||
- gfa_core/schemas/stages/domain_contract_v1.md
|
||
- gfa_core/schemas/stages/structure_contract_v1.md
|
||
- gfa_core/schemas/stages/combo_contract_v1.md
|
||
- skills/gene-identify/SKILL.md
|
||
- skills/phylogeny/SKILL.md
|
||
- skills/motif-analysis/SKILL.md
|
||
- skills/domain-analysis/SKILL.md
|
||
- gfa_core/mas/core/orchestrator.py
|
||
- gfa_core/mas/agents/discovery.py
|
||
- apps/web/vue3/static/conversation-sidebar.js
|
||
- gfa_core/pipeline_api_integration.py
|
||
|
||
Code relationship summary:
|
||
- Natural-language parsing now emits targetSpecies/referenceSpecies/comparativeSpecies/scopeMode instead of only prefix/species.
|
||
- /api/chat, /api/agent/chat and /api/agent/plan preserve these scope fields and put them into params for /api/execute.
|
||
- _ensure_gene_family_scope_file can now use the parsed natural-language scope fields directly to write a scope JSON from config.xlsx.
|
||
- Stage tool metadata now describes the real scoped inputs; structure uses final_members.id + target GFF3, not protein FASTA.
|
||
- ReproducibilityManager now records scope_id, scope_file and scope content in progress.json, commands.sh and provenance.json.
|
||
- pipeline_backend passes scope_file/scope_id into ReproducibilityManager.
|
||
- Stage contracts and skills now state that scope, not prefix alone, defines the analysis range.
|
||
|
||
Behavioral fixes:
|
||
- Removed runtime-path Ft.SPL defaults from orchestrator/discovery fallbacks. Generic RNA-seq fallback prefix is now "analysis"; gene-family fallback leaves family/species/prefix blank until config/user input supplies them.
|
||
- Natural-language example "目标物种Lc和Dl，参考物种At，比较物种Os，SPL基因家族成员鉴定和进化树" parses to:
|
||
targetSpecies = Lc,Dl
|
||
referenceSpecies = At
|
||
comparativeSpecies = Os
|
||
scopeMode = target_plus_reference_plus_comparative
|
||
prefix = Multi.SPL
|
||
- Lowercase family names such as "spl" are normalized to SPL in the fast keyword intent path.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/mas/utils/intent_parser.py gfa_core/mas/core/fast_intent.py apps/web/vue3/main.py gfa_core/mas/tools/stage_tools.py gfa_core/reproducibility_v2.py apps/web/pipeline_backend.py gfa_core/mas/core/orchestrator.py gfa_core/mas/agents/discovery.py gfa_core/pipeline_api_integration.py gfa_core/mas/agents/result_parser.py gfa_core/mas/agents/result_parser_cumulative.py
|
||
- parse_gene_family_scope_selection / parse_intent_from_query / fast_intent_recognition on:
|
||
"请用目标物种Lc和Dl，参考物种At，比较物种Os，进行SPL基因家族成员鉴定和进化树分析"
|
||
"帮我执行Lc和Dl的spl基因家族鉴定"
|
||
- apps.web.vue3.main.parse_natural_language + _ensure_gene_family_scope_file with real job config:
|
||
job_1779115921147_d4e0a7
|
||
|
||
Real test result:
|
||
- parse_natural_language returned prefix Multi.SPL and params containing targetSpecies Lc,Dl; referenceSpecies At; comparativeSpecies Os; scopeMode target_plus_reference_plus_comparative; geneFamily SPL.
|
||
- _ensure_gene_family_scope_file wrote:
|
||
workspace/job_1779115921147_d4e0a7/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference_plus_comparative.scope.json
|
||
- ReproducibilityManager loaded the existing real scope file and resolved:
|
||
scope_id = SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference
|
||
target_species = Lc,Dl
|
||
reference_species = At
|
||
comparative_species = Os
|
||
|
||
Known residual risks:
|
||
- Historical docs, backups and test fixtures still contain Ft.SPL as historical data. Active runtime fallbacks were cleaned; historical records were not rewritten.
|
||
- Full browser click-through of /api/agent/plan -> confirmation dialog -> /api/execute is still not automated by Playwright.
|
||
- Some older result interpretation text still uses prefix as prose context; it no longer controls execution scope, but future interpretation cleanup should display scope_id/species sets to users.
|
||
Next phase gate:
|
||
- Update result interpretation/reporting so summaries display family + scope species sets rather than relying on prefix wording.
|
||
```
|
||
|
||
```text
|
||
Phase: 11 Result interpretation/report scope 化
|
||
Modified files:
|
||
- gfa_core/mas/agents/result_parser.py
|
||
- gfa_core/mas/agents/interpretation_evidence.py
|
||
- gfa_core/mas/agents/result_parser_cumulative.py
|
||
|
||
Code relationship summary:
|
||
- ResultParserAgent now extracts scope context from manifest top-level fields, parameters or scope blocks.
|
||
- Single-stage reports now include an explicit "分析范围" section with family, scope_id, scope_mode, target/reference/comparative/selected species.
|
||
- Prompt metric bundles and LLM digests now carry scope, so downstream interpretation can cite the real analysis range.
|
||
- Cumulative reports now accept optional scope_id, auto-discover a scope_id from matching manifests when omitted, and prefer scoped manifests before legacy manifests.
|
||
- Evidence-first cumulative interpretation now filters manifests by scope_id and uses the scope label in report title/biological story.
|
||
- The fallback cumulative mixin was updated to keep the same scope fields, avoiding a future import path reverting to prefix-only wording.
|
||
|
||
Behavioral fixes:
|
||
- "Multi.SPL" is no longer presented as if it were the biological species range in reports.
|
||
- Real scope "SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference" reports as:
|
||
family = SPL
|
||
target_species = Lc,Dl
|
||
reference_species = At
|
||
comparative_species = Os
|
||
selected_species = Lc,Dl,At
|
||
- Cumulative interpretation can now distinguish scoped manifests from legacy manifests for the same prefix.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/mas/agents/result_parser.py gfa_core/mas/agents/result_parser_cumulative.py gfa_core/mas/agents/interpretation_evidence.py
|
||
- ResultParserAgent._parse_result and _generate_report using:
|
||
workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.domain.manifest.json
|
||
- ResultParserAgent.generate_cumulative_report using real completed stages:
|
||
phylogeny,motif,domain,structure,combo
|
||
|
||
Real test result:
|
||
- Parsed scope label:
|
||
SPL家族；目标物种=Lc,Dl；参考物种=At；比较物种=Os
|
||
- Single-stage markdown includes:
|
||
基因家族 SPL, Scope ID SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference, 目标物种 Lc/Dl, 参考物种 At, 比较物种 Os, 实际参与分析物种 Lc/Dl/At.
|
||
- Cumulative report auto-discovered scope_id when scope_id was omitted and loaded the scoped manifests.
|
||
|
||
Known residual risks:
|
||
- Evidence helper sibling-manifest lookup still falls back to legacy sibling names for some cross-stage raw-file parsing; scoped report selection works, but sibling evidence enrichment can be made fully scope-aware later.
|
||
Next phase gate:
|
||
- Continue with the next analysis stage or reporting polish pass; prioritize any active user-facing confusion before broad refactors.
|
||
```
|
||
|
||
```text
|
||
Phase: 11b Evidence report hardcoded-prefix cleanup
|
||
Modified files:
|
||
- gfa_core/mas/agents/interpretation_evidence.py
|
||
- gfa_core/mas/agents/result_parser.py
|
||
|
||
Behavioral fixes:
|
||
- Removed active result-report hardcoded Ft.SPL/Ath SPL wording from synteny, interspecies_synteny, PPI and publication-claim templates.
|
||
- Evidence report text now uses the current scope label/family label for target/reference family statements.
|
||
- Identify reference-count fallback no longer only recognizes SPL_Ath.fasta; it accepts reference-role, known-family or family-named inputs from manifest.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile gfa_core/mas/agents/result_parser.py gfa_core/mas/agents/result_parser_cumulative.py gfa_core/mas/agents/interpretation_evidence.py
|
||
- ResultParserAgent.generate_cumulative_report with real scope job codex_scope_phy_ref_fast_test.
|
||
|
||
Real test result:
|
||
- Cumulative report scope label is:
|
||
SPL家族；目标物种=Lc,Dl；参考物种=At；比较物种=Os
|
||
- Serialized cumulative report no longer contains active "Ft.SPL" or "Ath SPL" wording.
|
||
```
|
||
|
||
```text
|
||
Phase: 12 Promoter scope guard and single-target resolution
|
||
Modified files:
|
||
- apps/cli/run_promoter.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/schemas/stages/promoter_contract_v1.md
|
||
|
||
Code relationship summary:
|
||
- run_promoter.py now accepts --scope-file.
|
||
- In scope mode, promoter resolves GFF/GFF3 and genome FASTA from the single selected target species instead of guessing from prefix.
|
||
- Promoter refuses multi-target scope because promoter extraction cannot safely merge different genomes/GFFs in one run.
|
||
- Success and failure manifests now record scope_id, scope_mode, scope_file, target/reference/comparative/selected species and actual gff3/genome/gene_id files.
|
||
- pipeline_backend passes scope_file to run_promoter.py and skips legacy "first GFF/first genome" selection when scope is available.
|
||
- Stage metadata and contract now state that promoter is a single-target stage.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_promoter.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py gfa_core/mas/agents/interpretation_evidence.py
|
||
- Multi-target guard test:
|
||
/home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_promoter.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/codex_promoter_scope_guard_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --plantcare-tab workspace/codex_scope_phy_ref_fast_test/promoter/analysis/nonexistent.tab
|
||
- Single-target real scope resolution:
|
||
build_gene_family_scope(... target_species=['Lc'] ...) + resolve_promoter_scope_inputs()
|
||
|
||
Real test result:
|
||
- Multi-target Lc,Dl scope failed early with clear message:
|
||
Promoter 阶段一次只能分析一个目标物种；当前 scope target_species=['Lc', 'Dl']。请在弹窗中只选择一个 target，或分别运行每个目标物种。
|
||
- Failure manifest was written to:
|
||
workspace/codex_promoter_scope_guard_test/final/Multi.SPL.promoter.manifest.json
|
||
and contains scope_id + target_species.
|
||
- Single-target Lc scope resolved real files:
|
||
/home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.Chr.gff3.gz
|
||
/home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.Chr.fasta.gz
|
||
|
||
Known residual risks:
|
||
- Full PlantCARE execution was not run because it depends on external web/email service. Scope input resolution and multi-target guard were validated with real project data.
|
||
- Promoter still writes legacy output names for compatibility; scoped manifest is written in addition when execution reaches success.
|
||
Next phase gate:
|
||
- Move to synteny/interspecies_synteny, which need careful target/comparative/reference separation and should not infer species from Multi.<family>.
|
||
```
|
||
|
||
```text
|
||
Phase: 13 Synteny scope guard and full-proteome target resolution
|
||
Modified files:
|
||
- apps/cli/run_synteny.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/schemas/stages/synteny_contract_v1.md
|
||
|
||
Code relationship summary:
|
||
- run_synteny.py now accepts --scope-file.
|
||
- In scope mode, synteny resolves target pep/GFF/genome from the single selected target species.
|
||
- Synteny refuses multi-target scope because it is an intraspecies MCScanX stage and cannot safely combine multiple genomes in one run.
|
||
- validate_inputs now prioritizes full target proteome over identify final family proteins; identify proteins are only a legacy fallback.
|
||
- final/{prefix}.final_members.id is now used as the preferred family gene list for Circos filtering.
|
||
- Success and failure manifests now record scope fields and write a scoped manifest copy when scope_id is present.
|
||
- pipeline_backend passes scope_file to run_synteny.py and skips legacy "first GFF/first genome" selection when scope is available.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_synteny.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py
|
||
- Multi-target guard test:
|
||
/home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/run_synteny.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/codex_synteny_scope_guard_test --prefix Multi.SPL --scope-file workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json --cpu 1
|
||
- Single-target real scope resolution:
|
||
resolve_synteny_scope_inputs(workspace/codex_promoter_scope_guard_test/final/Multi.SPL.Lc.promoter.scope.json)
|
||
|
||
Real test result:
|
||
- Multi-target Lc,Dl scope failed early with clear message:
|
||
Synteny 阶段是种内共线性，一次只能分析一个目标物种；当前 scope target_species=['Lc', 'Dl']。请在弹窗中只选择一个 target，或分别运行每个目标物种。
|
||
- Failure manifest was written to:
|
||
workspace/codex_synteny_scope_guard_test/final/Multi.SPL.synteny.manifest.json
|
||
and contains scope_id + target_species.
|
||
- Single-target Lc scope resolved real files:
|
||
/home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.pep.sim.fa.gz
|
||
/home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.Chr.gff3.gz
|
||
/home/xia-lab/wutx/plantAnalysisAgent/ref_data/lizhi_longyan/Lchinesis_genome.Chr.fasta.gz
|
||
|
||
Known residual risks:
|
||
- Full MCScanX execution was not run in this pass because it can be long and depends on local tool availability/data scale. Scope input resolution and multi-target guard were validated with real project data.
|
||
- Some legacy behavior still allows identify final proteins as fallback when no full target proteome is found; this is retained for compatibility but is not the recommended path.
|
||
Next phase gate:
|
||
- Move to interspecies_synteny. It should use target/reference/comparative roles explicitly rather than scanning arbitrary protein files.
|
||
```
|
||
|
||
```text
|
||
Phase: 14 Interspecies synteny scope selection and gzip-safe real data discovery
|
||
Modified files:
|
||
- apps/cli/run_interspecies_synteny.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/schemas/stages/interspecies_synteny_contract_v1.md
|
||
|
||
Code relationship summary:
|
||
- run_interspecies_synteny.py now accepts --scope-file.
|
||
- Scope selection is applied before discovery:
|
||
target_species -> target side
|
||
selected reference/comparative species -> ref side
|
||
- In scope mode, species files are resolved from scope files first.
|
||
- In scope mode, full proteome can only come from scope `pep`; family_pep is not treated as all_pep.
|
||
- reference/comparative family_pep can still be used for the interspecies family tree.
|
||
- gzip FASTA/GFF files are now read through open_text(), which supports .gz ref_data paths.
|
||
- Success/failure manifests now record scope_id, scope_mode, scope_file, target/reference/comparative/selected species and write a scoped manifest copy when scope_id is present.
|
||
- pipeline_backend passes scope_file to run_interspecies_synteny.py and skips legacy input pep precheck when scope is available.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_interspecies_synteny.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py
|
||
- Real scope discovery with:
|
||
workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json
|
||
- Real target+reference+comparative discovery with:
|
||
workspace/job_1779115921147_d4e0a7/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference_plus_comparative.scope.json
|
||
- Failure manifest write test with real scope metadata.
|
||
|
||
Real test result:
|
||
- target_plus_reference scope selected:
|
||
targets = Lc,Dl
|
||
refs = At
|
||
target_all_peps = Lc,Dl
|
||
ref_all_peps = empty because At all-pep path is missing in scope
|
||
ref_family_protein = generated selected_ref.family.pep.fa
|
||
- target_plus_reference_plus_comparative scope selected:
|
||
targets = Lc,Dl
|
||
refs = At,Os
|
||
target_all_peps = Lc,Dl
|
||
ref_all_peps = Os only
|
||
ref_gff = At,Os
|
||
- Failure manifest records:
|
||
scope_id = SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference
|
||
target_species = Lc,Dl
|
||
reference_species = At
|
||
comparative_species = Os
|
||
|
||
Known residual risks:
|
||
- Full MAFFT/IQ-TREE/MCScanX execution was not run in this pass because it can be long. The corrected scope selection, gzip-safe file discovery and manifest behavior were validated with real project data.
|
||
- At whole-proteome path is absent in the current scope file, so interspecies MCScanX for At should remain partial/skipped unless the At all-pep file is added or genome_prepare provides it. This is correct behavior; the system must not substitute At.SPL.pep.fa as whole proteome.
|
||
Next phase gate:
|
||
- Move to karyotype, which is another target-side stage and must not run all targets together unless explicitly designed for multi-target output.
|
||
```
|
||
|
||
```text
|
||
Phase: 15 Karyotype target-scope selection and gzip GFF support
|
||
Modified files:
|
||
- apps/cli/run_karyotype.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/schemas/stages/karyotype_contract_v1.md
|
||
|
||
Code relationship summary:
|
||
- run_karyotype.py now accepts --scope-file.
|
||
- In scope mode, karyotype uses only `scope_file.target_species` and resolves each target species' GFF/genome from scope.
|
||
- Multi-target scope remains supported, but output is per target species (`{Species}.{Family}.karyotype.*`), not one merged cross-species karyotype.
|
||
- gzip GFF files are now read through open_text(), supporting real ref_data *.gff3.gz inputs.
|
||
- Success/failure manifests now record scope_id, scope_mode, scope_file, target/reference/comparative/selected species and write a scoped manifest copy when scope_id is present.
|
||
- pipeline_backend passes scope_file to run_karyotype.py.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_karyotype.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py
|
||
- discover_species_inputs() using real scope:
|
||
workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json
|
||
- build_simplified_gff() against real gzip GFFs for Dl and Lc using real final_members IDs.
|
||
- write_manifest() scope-copy test.
|
||
|
||
Real test result:
|
||
- Scope discovery selected only target species:
|
||
Lc, Dl
|
||
- Resolved real files:
|
||
Lc: Lchinesis_genome.Chr.gff3.gz + Lchinesis_genome.Chr.fasta.gz
|
||
Dl: Longan.v20190918.gff3.gz + Longan_Chr.v20190918.fasta.gz
|
||
- GFF matching succeeded on gzip inputs:
|
||
Dl sample: 5/5 matched rows
|
||
Lc sample: 5/5 matched rows
|
||
- Manifest writes both:
|
||
|
||
```text
|
||
Phase: 16 PEP scoped protein-set input
|
||
Modified files:
|
||
- apps/cli/run_pep.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/schemas/stages/pep_contract_v1.md
|
||
|
||
Code relationship summary:
|
||
- run_pep.py now accepts --scope-file and --fasta.
|
||
- In scope mode, PEP resolves the same protein FASTA used by the matching phylogeny scope manifest (`parameters.scope_input_fasta`), then removes terminal/internal `*` into a job-local clean FASTA when needed.
|
||
- target_only scope may fall back to `final/{prefix}.final_members.pep.fa`; multi-species/reference/comparative scopes must use the phylogeny-scoped merged FASTA unless --fasta is explicitly provided.
|
||
- Success manifests record scope_id, scope_mode, scope_file, target/reference/comparative/selected species, family, and write a scoped manifest copy.
|
||
- pipeline_backend passes the scoped FASTA into the existing PEP skill path and annotates the copied manifest with scope metadata, so the current faster skill implementation remains compatible.
|
||
|
||
Real test command:
|
||
- Compile run_pep.py / pipeline_backend.py / stage_tools.py.
|
||
- Resolve PEP input from the real target+reference scope and verify it points to the phylogeny-scoped clean FASTA.
|
||
- Run the PEP skill on that real scoped FASTA and verify final manifest contains scope metadata and the expected protein count.
|
||
|
||
Real test result:
|
||
- py_compile passed.
|
||
- CLI resolver:
|
||
`/home/xia-lab/wutx/00test/bbav5_0502_Yang_Tihuan.7880/workspace/codex_scope_phy_ref_fast_test/phylogeny/analysis/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.input.pep.clean.fa`
|
||
- Backend resolver uses the same scoped clean FASTA instead of `final/{prefix}.final_members.pep.clean.fa`, preventing cross-scope overwrite.
|
||
- PEP skill loaded 72 real sequences and produced:
|
||
`workspace/codex_pep_scope_fast_test/final/Multi.SPL.pep.manifest.json`
|
||
`workspace/codex_pep_scope_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.pep.manifest.json`
|
||
- Manifest contains `scope_id=SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference` in both top-level fields and `parameters`.
|
||
```
|
||
|
||
```text
|
||
Phase: 17 Expression single-target scope selection
|
||
Modified files:
|
||
- apps/cli/run_expression.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/schemas/stages/expression_contract_v1.md
|
||
|
||
Code relationship summary:
|
||
- Expression is treated as a single-target-species stage because expression matrices are species-specific.
|
||
- pipeline_backend now reads scope_file for expression and rejects multi-target scopes before execution.
|
||
- In scope mode, backend resolves member IDs from `identify/{Species}/final/{Species}.final_gfmembers.id` instead of globbing any `*geneID*` or multi-species `final/{prefix}.final_members.id`.
|
||
- Removed the duplicated expression/gene/sample discovery block that could overwrite the first resolved inputs.
|
||
- run_expression.py accepts `--scope-file`, validates that exactly one target species is selected, allows the scoped per-species gene ID file to satisfy upstream validation, records scope fields in manifest, and writes a scoped manifest copy.
|
||
- comparative_summary.target_species now uses the scoped target species instead of parsing `Multi` from `Multi.SPL`.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_expression.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py
|
||
- Resolver test against real Lc/Dl identify outputs from `workspace/codex_scope_phy_ref_fast_test/identify/{Species}/final/{Species}.final_gfmembers.id`.
|
||
- Full expression run with a small real Lc member subset expression matrix and `rnaR` Rscript.
|
||
- Multi-target guard run with target_species=[Lc,Dl].
|
||
|
||
Real test result:
|
||
- py_compile passed.
|
||
- Resolver selected:
|
||
`workspace/codex_scope_phy_ref_fast_test/identify/Lc/final/Lc.final_gfmembers.id`
|
||
`workspace/codex_scope_phy_ref_fast_test/identify/Dl/final/Dl.final_gfmembers.id`
|
||
- Single-target Lc expression run succeeded in `workspace/codex_expression_scope_fast_test`, using:
|
||
`workspace/codex_expression_scope_fast_test/identify/Lc/final/Lc.final_gfmembers.id`
|
||
- R generated heatmap/correlation/barplot PDF+PNG and expression/correlation txt outputs.
|
||
- Manifest contains `scope_id=SPL.target-Lc.ref-At.comp-Os.target_only`, `gene_count=33`, and comparative_summary.target_species=`Lc`.
|
||
- Multi-target scope failed early with the intended message:
|
||
`Expression 阶段一次只能分析一个目标物种；当前 scope target_species=['Lc', 'Dl']。请在弹窗中只选择一个 target，或分别运行每个目标物种。`
|
||
```
|
||
|
||
```text
|
||
Phase: 18 Ortholog target-to-reference scope mapping
|
||
Modified files:
|
||
- apps/cli/run_ortholog.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/schemas/stages/ortholog_contract_v1.md
|
||
|
||
Code relationship summary:
|
||
- run_ortholog.py had the broken inline CONDA_BIN statement repaired.
|
||
- pipeline_backend ortholog stage now reads scope_file when provided.
|
||
- In scope mode, target family proteins are resolved per species from:
|
||
`identify/{Species}/final/{Species}.final_gfmembers.pep.fa`, then compatible `{Species}.{Family}.pep.fa` fallbacks.
|
||
- Reference protein resolves from scope `family_pep` first, then compatible `{Reference}.{Family}.pep.fa`; it no longer depends only on hardcoded Ath/At filenames.
|
||
- Multi-target scope runs one BLASTP job per target species against the first reference species, then merges mappings.
|
||
- Output mapping now includes source/target species columns:
|
||
`Source_ID Target_ID Identity(%) Source_Species Target_Species`
|
||
- Manifest records scope fields and writes a scoped manifest copy.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_ortholog.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py
|
||
- run_stage('ortholog') with real scope:
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json`
|
||
target: Lc,Dl; reference: At; family: SPL.
|
||
|
||
Real test result:
|
||
- py_compile passed.
|
||
- BLASTP 2.17.0+ was available.
|
||
- Ortholog ran two real BLASTP jobs:
|
||
Lc final family proteins vs `input/job_1779115921147_d4e0a7/At.SPL.pep.fa`
|
||
Dl final family proteins vs `input/job_1779115921147_d4e0a7/At.SPL.pep.fa`
|
||
- Produced 54 mapped pairs.
|
||
- Output:
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.ortholog_pairs.txt`
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.orthologs.txt`
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.ortholog_pairs.raw.txt`
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.ortholog_blastp.txt`
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.ortholog.manifest.json`
|
||
- Example mapping:
|
||
`LITCHI014255.m1 At2g47070 49.81 Lc At`
|
||
```
|
||
|
||
```text
|
||
Phase: 19 PPI scoped ortholog mapping input
|
||
Modified files:
|
||
- apps/cli/run_ppi.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/schemas/stages/ppi_contract_v1.md
|
||
|
||
Code relationship summary:
|
||
- run_ppi.py had the broken inline CONDA_BIN statement repaired.
|
||
- pipeline_backend PPI stage now reads scope_file when provided.
|
||
- In scope mode, reference_species is converted to the STRING species code (`At -> ath`, `Os -> osa`).
|
||
- If an ortholog mapping exists, PPI builds a scoped source gene ID file from `{prefix}.orthologs.txt` / `{prefix}.ortholog_pairs.txt` and passes the mapping to the existing PPI skill.
|
||
- Multi-target scope now requires Ortholog first; PPI will not send mixed local IDs from multiple species directly to STRING.
|
||
- PPI manifest annotation uses the shared scope manifest helper and writes a scoped manifest copy after a real PPI run.
|
||
|
||
Real-data validation command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_ppi.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py
|
||
- build_ppi_gene_ids_from_ortholog_mapping() with real ortholog output:
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.orthologs.txt`
|
||
- skills.ppi.run.parse_ortholog_mapping() on the same real mapping.
|
||
|
||
Real-data validation result:
|
||
- py_compile passed.
|
||
- From real Lc/Dl -> At ortholog mapping, PPI source ID extraction produced 54 source genes:
|
||
`workspace/codex_scope_phy_ref_fast_test/ppi/analysis/codex_ppi_source_ids.txt`
|
||
- STRING species code resolution:
|
||
`At -> ath`, `Os -> osa`
|
||
- Existing PPI skill parsed 54 source→At mappings, e.g.:
|
||
`LITCHI014255.m1 -> At2g47070`
|
||
|
||
Known residual risk:
|
||
- Full STRING network querying was not executed in this pass because it depends on external network service availability. The scoped input/mapping path was validated with real ortholog data; a complete online PPI run should be verified separately when network access is allowed.
|
||
```
|
||
|
||
```text
|
||
Phase: 20 Ka/Ks scoped target-reference runs and YN00 parser correction
|
||
Modified files:
|
||
- apps/cli/run_kaks_v2.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/schemas/stages/kaks_contract_v1.md
|
||
|
||
Code relationship summary:
|
||
- run_kaks_v2.py now has a valid CONDA_BIN/PATH block.
|
||
- run_kaks_v2.py can read gzipped FASTA inputs.
|
||
- run_kaks_v2.py now matches IDs case-insensitively after removing transcript/version suffixes, so `At2g47070` can match `AT2G47070.1`.
|
||
- run_kaks_v2.py now detects pal2nal error output instead of passing an empty/error codon alignment into yn00.
|
||
- YN00 parsing was corrected. The old parser incorrectly treated columns `S/N/t` as `Ka/Ks/KaKs`; the corrected parser reads `dN/dS/omega`.
|
||
- pipeline_backend KaKs stage now reads scope_file.
|
||
- In scope mode, KaKs splits multi-target scopes into per-target runs:
|
||
`{prefix}.{Target}_to_{Reference}`
|
||
- Target CDS/PEP and reference CDS/PEP are resolved per species. Reference PEP is intentionally resolved from the same whole-genome annotation source as reference CDS, not from `{Reference}.{Family}.pep.fa`, because Ka/Ks requires CDS and protein from the same transcript set.
|
||
- Aggregate and scoped manifests are written:
|
||
`{prefix}.kaks.manifest.json`
|
||
`{prefix}.{scope_id}.kaks.manifest.json`
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_kaks_v2.py apps/web/pipeline_backend.py gfa_core/mas/tools/stage_tools.py
|
||
- Resolve real scope files:
|
||
Lc CDS: Lchinesis_genome.sim.cds.fa.gz
|
||
Dl CDS: longan.v20190918.sim.CDS.gz
|
||
At CDS: Arabidopsis_thaliana.TAIR10.cds.all.fa
|
||
At PEP: Arabidopsis_thaliana.TAIR10.pep.all.fa
|
||
- Split real ortholog mapping:
|
||
Lc->At: 33 pairs
|
||
Dl->At: 21 pairs
|
||
- Run full scoped KaKs through run_stage('kaks').
|
||
|
||
Real test result:
|
||
- py_compile passed.
|
||
- Initial test with `At.SPL.pep.fa` failed pal2nal consistency checks, proving that NCBI/downloaded family PEP cannot be mixed with TAIR10 CDS for Ka/Ks.
|
||
- After switching reference PEP to TAIR10 whole PEP and fixing YN00 parser, full run succeeded:
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.Lc_to_At.kaks_results.csv`
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.Dl_to_At.kaks_results.csv`
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.kaks.manifest.json`
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference.kaks.manifest.json`
|
||
- Successful pairs:
|
||
Lc->At: 33/33
|
||
Dl->At: 21/21
|
||
- Corrected example values:
|
||
`LITCHI014255.m1, At2g47070, Ka=0.3281, Ks=1.3925, Ka/Ks=0.2356`
|
||
`Dil.01g033640.1.t1, At5g18830, Ka=0.3490, Ks=1.5298, Ka/Ks=0.2282`
|
||
- Aggregate manifest records:
|
||
`species_runs=2`, `input_pairs=54`, `scope_id=SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference`.
|
||
```
|
||
|
||
```text
|
||
Phase: 21 Summary scoped objective result index
|
||
Modified files:
|
||
- apps/cli/run_summary.py
|
||
- apps/web/pipeline_backend.py
|
||
- gfa_core/mas/tools/stage_tools.py
|
||
- gfa_core/schemas/stages/summary_contract_v1.md
|
||
|
||
Code relationship summary:
|
||
- run_summary.py was replaced with a self-contained manifest aggregator. The old file imported a missing summary_agent module and could not run as a reliable CLI stage.
|
||
- Summary is defined as an objective result index and QC report, not deep biological interpretation.
|
||
- Summary scans `final/*.manifest.json`, excludes summary/minimal self manifests, filters child manifests whose prefix does not match the requested prefix, groups results by `scope_id`, and records stage status, key metrics, artifacts and consistency checks.
|
||
- Summary supports gene-family and RNA-seq style manifests because it only depends on the shared manifest schema fields.
|
||
- In scope mode, Summary writes both a global summary and a scoped summary:
|
||
`{prefix}.summary_report.md`
|
||
`{prefix}.summary.files.tsv`
|
||
`{prefix}.summary.manifest.json`
|
||
`{prefix}.{scope_id}.summary_report.md`
|
||
`{prefix}.{scope_id}.summary.files.tsv`
|
||
`{prefix}.{scope_id}.summary.manifest.json`
|
||
- pipeline_backend now runs Summary through `apps/cli/run_summary.py` and no longer requires an identify manifest as the seed.
|
||
|
||
Real test command:
|
||
- /home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_summary.py
|
||
- Run CLI summary with real scope:
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.phylogeny_ref_scope.json`
|
||
- Run backend `run_stage('summary')` on the same real workspace.
|
||
- Run CLI summary on an existing real RNA-seq manifest workspace:
|
||
`workspace/job_1778662928678_06813e`, prefix `Os`.
|
||
|
||
Real test result:
|
||
- py_compile passed.
|
||
- CLI and backend Summary both succeeded.
|
||
- Summary grouped the real Lc,Dl target / At reference / Os comparative SPL run into:
|
||
`SPL.target-Lc-Dl.ref-At.comp-Os.target_only`: identify
|
||
`SPL.target-Lc-Dl.ref-At.comp-Os.target_plus_reference`: combo, domain, kaks, motif, ortholog, phylogeny, reference_classification, structure
|
||
- Output report:
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.summary_report.md`
|
||
- Output artifact index:
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.summary.files.tsv`
|
||
- Output manifest:
|
||
`workspace/codex_scope_phy_ref_fast_test/final/Multi.SPL.summary.manifest.json`
|
||
- Aggregate manifest records:
|
||
`manifest_count=9`, `scope_count=2`, `artifact_count=103`.
|
||
- RNA-seq manifest aggregation also succeeded:
|
||
`workspace/job_1778662928678_06813e/final/Os.summary_report.md`
|
||
`workspace/job_1778662928678_06813e/final/Os.summary.files.tsv`
|
||
`workspace/job_1778662928678_06813e/final/Os.summary.manifest.json`
|
||
- RNA-seq summary grouped 13 unscoped stage manifests, including:
|
||
annotation, config_autofill, rnaseq_align, rnaseq_candidate_family, rnaseq_de, rnaseq_fgsea, rnaseq_go_enrich, rnaseq_kegg_enrich, rnaseq_qc, rnaseq_quant, rnaseq_sample_qc, rnaseq_wgcna, samples_autofill.
|
||
- RNA-seq aggregate manifest records:
|
||
`manifest_count=13`, `scope_count=1`, `artifact_count=305`.
|
||
|
||
Known residual risks:
|
||
- Summary currently reports objective artifacts/metrics/status only. It intentionally does not infer biological conclusions or recommend subfamilies; that should be a separate deep interpretation stage built on top of these indexed outputs.
|
||
```
|
||
|
||
```text
|
||
Phase: 22 Full real-data end-to-end validation and fixes
|
||
Modified files:
|
||
- apps/web/pipeline_backend.py
|
||
- apps/cli/run_reference_classification.py
|
||
- apps/cli/run_promoter.py
|
||
- apps/cli/run_synteny.py
|
||
|
||
Validation workspace:
|
||
- workspace/codex_full_flow_real_20260519_235947
|
||
|
||
Real data:
|
||
- target species: Lc,Dl
|
||
- reference species: At
|
||
- comparative species: Os
|
||
- family: SPL
|
||
- input dir: input/job_1779115921147_d4e0a7
|
||
|
||
Issues found and fixed:
|
||
- Phylogeny advanced IQ-TREE params beginning with `-` were passed as a separate argv value, so argparse treated the value as another option. pipeline_backend now passes these as `--iqtree-params=<value>` and `--mafft-params=<value>`.
|
||
- reference_classification auto-selected `Target.SPL.geneID.subfamily`, which already contained target labels, instead of the pure At reference file. It now prefers `{reference_prefix}.{family}.subfamily`, for example `At.SPL.subfamily`.
|
||
- promoter failed on non-UTF-8 stderr from external tools. The command runner now decodes subprocess output with replacement instead of raising UnicodeDecodeError.
|
||
- synteny failed or fell back to very slow BLASTP because whole-proteome FASTA files contained terminal `*`/`.` stop markers and DIAMOND v2 uses `--threads`, not `--thread`. Synteny now writes a BLAST-safe cleaned protein FASTA and uses the correct DIAMOND option.
|
||
|
||
Final validation result:
|
||
- Final summary report:
|
||
`workspace/codex_full_flow_real_20260519_235947/final/Multi.SPL.summary_report.md`
|
||
- Final summary manifest:
|
||
`workspace/codex_full_flow_real_20260519_235947/final/Multi.SPL.summary.manifest.json`
|
||
- Final summary artifact index:
|
||
`workspace/codex_full_flow_real_20260519_235947/final/Multi.SPL.summary.files.tsv`
|
||
- Summary recorded:
|
||
`manifest_count=18`, `scope_count=3`, `artifact_count=181`
|
||
- Final consistency check:
|
||
`No manifest-level consistency issues detected`
|
||
|
||
Successful final stages:
|
||
- target_plus_reference scope:
|
||
identify, phylogeny, reference_classification, structure, domain, motif, pep, combo, karyotype, interspecies_synteny, ortholog, ppi, kaks
|
||
- Lc target_only scope:
|
||
expression, promoter, synteny
|
||
- Dl target_only scope:
|
||
promoter, synteny
|
||
|
||
Key final metrics:
|
||
- Identify/phylogeny protein set: 76 sequences = 59 target + 17 At reference.
|
||
- Reference classification: target_gene_count=59, reference_gene_count=17.
|
||
- Domain: proteins_total=76, proteins_with_domains=67, domain_coverage_rate=0.8816.
|
||
- Motif: sequence_count=76.
|
||
- Ortholog: ortholog_pairs=58.
|
||
- Ka/Ks: species_runs=2, input_pairs=58.
|
||
- Karyotype: species_count=2, success_count=2.
|
||
- Interspecies synteny: status=success, Lc.At and Dl.At collinearity files produced.
|
||
- Promoter:
|
||
Lc and Dl target_only runs both succeeded.
|
||
- Synteny:
|
||
Lc gene_count=31896, Dl gene_count=37142.
|
||
|
||
Known residual risks:
|
||
- The validation used a generated full_clean scope for interspecies_synteny because the original scope had empty At whole-proteome paths. The next front-end/config revision should ensure whole-proteome reference paths are filled in the generated scope by default.
|
||
- Full phylogeny with ModelFinder is slow for validation. A future UI/plan option should expose fast verification mode such as `-m LG+G4 -nt <cpu> -fast`.
|
||
```
|
||
|
||
```text
|
||
Phase: 23 Automatic scope artifact and reference-path repair
|
||
Modified files:
|
||
- apps/cli/run_promoter.py
|
||
- apps/cli/run_synteny.py
|
||
- gfa_core/gene_family_scope.py
|
||
|
||
Purpose:
|
||
- Remove manual intervention for two remaining scope problems:
|
||
1. Single-target stages wrote scope manifests but their artifacts still pointed to generic files, so Lc/Dl runs could overwrite each other.
|
||
2. Generated scopes could leave reference whole-proteome paths empty when config.xlsx recorded files stored in local ref_data instead of input/final.
|
||
|
||
Fixes:
|
||
- promoter and synteny now materialize scope-specific artifact copies when `scope_id` is present. Generic legacy outputs are still written for compatibility, but scoped manifests point to stable paths containing the sanitized scope ID.
|
||
- promoter failure handling now writes both generic and scoped failure manifests when a scope file is provided.
|
||
- promoter PATH initialization was repaired so the test4py311 tool directory is actually prepended; this fixed a real `gffread` not found failure.
|
||
- gene_family_scope now searches local reference-data roots, including `/home/xia-lab/wutx/plantAnalysisAgent/ref_data`, by the config original file name when input/workspace/final lookup fails.
|
||
|
||
Real-data validation:
|
||
- Scope generation:
|
||
`/tmp/codex_scope_autofill_test.json`
|
||
resolved At whole-genome gff/cds/genome/pep paths from local ref_data, including:
|
||
`/home/xia-lab/wutx/plantAnalysisAgent/ref_data/ath/Arabidopsis_thaliana.TAIR10.pep.all.fa`
|
||
- Synteny rerun with real Lc and Dl target_only scopes:
|
||
`Multi.SPL.SPL.target-Lc.ref-At.comp-Os.target_only.synteny.manifest.json`
|
||
`Multi.SPL.SPL.target-Dl.ref-At.comp-Os.target_only.synteny.manifest.json`
|
||
Both scoped manifests point to scope-specific blast/circos/report artifacts and `missing=0` for registered artifact paths.
|
||
- Promoter rerun with real Lc and Dl target_only scopes using existing local PlantCARE tab:
|
||
`Multi.SPL.SPL.target-Lc.ref-At.comp-Os.target_only.promoter.manifest.json`
|
||
`Multi.SPL.SPL.target-Dl.ref-At.comp-Os.target_only.promoter.manifest.json`
|
||
Both scoped manifests point to scope-specific report and PlantCARE/table artifacts and `missing=0` for registered artifact paths.
|
||
|
||
Validation commands:
|
||
- `/home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile apps/cli/run_promoter.py apps/cli/run_synteny.py gfa_core/gene_family_scope.py`
|
||
- `/home/xia-lab/miniconda3/envs/test4py311/bin/python apps/cli/resolve_scope.py --input-dir input/job_1779115921147_d4e0a7 --workspace workspace/codex_full_flow_real_20260519_235947 --prefix Multi.SPL --family SPL --target-species Lc,Dl --reference-species At --comparative-species Os --scope-mode target_plus_reference --output /tmp/codex_scope_autofill_test.json`
|
||
- Real Lc/Dl synteny and promoter reruns in `workspace/codex_full_flow_real_20260519_235947`.
|
||
|
||
Known residual risks:
|
||
- The promoter validation intentionally used an existing local PlantCARE tab to avoid submitting to the external PlantCARE service. That tab was empty after filtering, so R promoter visualizations reported "no lines available in input"; the stage still completed and scoped artifact/manifest behavior was validated.
|
||
```
|
||
|
||
## 2026-05-26 追加记录：Stage Requirements 与 Agent 职责边界
|
||
|
||
KaKs/PPI/scope 讨论暴露出 PlanningAgent、DiagnosisAgent、ResultParserAgent 与执行层之间存在职责混杂：静态依赖、动态需求、artifact 解析、完成状态、修复策略在多个文件重复实现。
|
||
|
||
详细问题归类、非破坏迁移计划和验收矩阵见：[`docs/stage_requirements_agent_refactor_plan_2026-05-26.md`](stage_requirements_agent_refactor_plan_2026-05-26.md)。
|
||
|