Compare commits
27
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b8be4e4831 | ||
|
|
c59f346037 | ||
|
|
0428a0d0ed | ||
|
|
1824f2d14b | ||
|
|
90033214b4 | ||
|
|
23f265f2b1 | ||
|
|
509f5ff11b | ||
|
|
da8001ab2a | ||
|
|
a2e8cbae86 | ||
|
|
77011a21b7 | ||
|
|
4a80793b4a | ||
|
|
8b61269d3f | ||
|
|
7b7e334213 | ||
|
|
b725a9c125 | ||
|
|
7228b586c8 | ||
|
|
fe5f7b9d6e | ||
|
|
87c158efe5 | ||
|
|
963e4ddc73 | ||
|
|
9dbd87696b | ||
|
|
39a2c969de | ||
|
|
9b9c88b18e | ||
|
|
b8dcb63fa1 | ||
|
|
1252c41661 | ||
|
|
1c2cfc5b6e | ||
|
|
0f81b5140c | ||
|
|
55338be930 | ||
|
|
9c38f32d52 |
@@ -30,3 +30,7 @@ Rplots.pdf
|
||||
*.pyc
|
||||
*.log
|
||||
.gitignore
|
||||
.worktrees/
|
||||
extension_stages_lightweight/
|
||||
services/*/logs/
|
||||
*.bak_*
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
# Plan Review Log: PPI/KaKs temporary frontend circuit breaker
|
||||
|
||||
Act 1 complete: the user selected a global frontend-only circuit breaker, retained plan visibility, paused batch and manual UI execution, preserved backend/API/CLI behavior, and required independent Ortholog execution to remain available.
|
||||
|
||||
MAX_ROUNDS=5
|
||||
PLAN_FILE=.planning/ppi-kaks-temporary-frontend-circuit-breaker/PLAN.md
|
||||
|
||||
## Local read-only audit — frontend control flow
|
||||
|
||||
- Confirmed the global gate precedes normal stage scheduling.
|
||||
- Required coverage of foreground batch, background batch, manual entry, shared submit, and stale download-monitor resume.
|
||||
- Rejected reuse of `ppiKaksPauseStageIds()` because it pauses every planned Ortholog.
|
||||
- Required the circuit breaker to activate before preflight can rewrite an existing Ortholog as PPI-owned.
|
||||
|
||||
## Local read-only audit — layer and contract boundaries
|
||||
|
||||
- Confirmed that PlanningAgent, backend, API, CLI, contracts, Manifest, stage-status, reproducibility, diagnosis, and result mapping should remain unchanged.
|
||||
- Found that dependency-expanded KaKs Ortholog may lack frontend owner fields; required preserving original intent stage provenance in frontend batch state.
|
||||
- Unknown Ortholog provenance must default to preserved/runnable.
|
||||
|
||||
## Local read-only audit — tests
|
||||
|
||||
- Required Node-executed behavior tests using the repository's `_source_block()` and `_run_node()` patterns, not constant/string-only proxy tests.
|
||||
- Required positive controls proving unrelated stages still submit.
|
||||
- Required explicit background-batch and stale-monitor tests.
|
||||
- Recorded an unrelated pre-existing `openPpiResetDialog()` contract failure so it is not hidden or misattributed.
|
||||
|
||||
## External Codex review
|
||||
|
||||
The requested read-only Codex CLI review was not run. The escalation was rejected because it would disclose the local plan and potentially repository context to an external service without explicit user authorization for that disclosure. No workaround was attempted, and no `VERDICT: APPROVED` is claimed.
|
||||
@@ -0,0 +1,190 @@
|
||||
# Plan: Temporarily pause PPI/KaKs without blocking other analyses
|
||||
|
||||
_Locked via user decisions on 2026-08-05; implementation is not authorized by this plan alone._
|
||||
|
||||
## Goal
|
||||
|
||||
Keep `ppi` and `kaks` visible in every generated plan while both features are under revision, but make the web UI treat them as globally and explicitly paused. The UI must not run PPI/KaKs reference-species preflight, show reference-download dialogs, submit reference-download requests, or submit PPI/KaKs execution requests. Other planned stages and their confirmation dialogs must continue normally. Backend APIs, CLI programs, stage contracts, manifests, and biological methods remain unchanged so developers can still exercise those paths directly.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
1. Plans returned by the backend still contain `ppi` and `kaks`; no PlanningAgent pruning is introduced.
|
||||
2. Before dependency preflight, the batch UI records every planned `ppi`/`kaks` stage as `paused` with the user-visible reason `功能修订中,本次自动跳过`.
|
||||
3. With the temporary circuit breaker enabled, the UI does not call `/api/ppi/species-resolution` or `/api/reference-bundle/download` for those paused owners.
|
||||
4. The UI never calls `/api/execute` with `stage` equal to `ppi` or `kaks` through normal batch or manual UI flows.
|
||||
5. A PPI/KaKs-owned auto-inserted `ortholog` stage is paused only when its parameters identify that ownership via `autoInsertedFor`, `dependencyOwner`, or a PPI/KaKs dependency `referencePolicySource`. An independently requested, unowned `ortholog` stage remains runnable.
|
||||
6. Non-PPI/KaKs stages continue through the existing confirmation queue and dependency scheduler after the paused stages are recorded.
|
||||
7. Paused stages are reported as paused, not failed or completed, and no success Manifest is generated for them.
|
||||
8. Turning off or removing the single temporary circuit-breaker declaration restores the existing PPI/KaKs preflight behavior without backend or contract migration.
|
||||
9. Foreground batches, background batches, manual confirmation, stale open dialogs, and shared submit paths enforce the same pause behavior; none can bypass the circuit breaker or misreport a pause as failure.
|
||||
|
||||
## Root cause
|
||||
|
||||
`startBatchExecute()` currently awaits `ensureOrthologProducerForPpiKaks()` before it initializes and runs the ordinary stage scheduler. `ensureOrthologProducerForPpiKaks()` stops on the first owner for which `ensurePpiKaksDependencyOwner()` returns false. `startBatchExecute()` then returns before `executeStagesWithDependencies()`, so a PPI/KaKs dependency problem prevents every later confirmation dialog and otherwise independent analysis.
|
||||
|
||||
The observed job log contains repeated successful HTTP calls to `/api/ppi/species-resolution` but no `/api/reference-bundle/download` call for the failing attempt. That localizes the blocking behavior to frontend dependency/preflight control flow rather than an executed NCBI bundle download.
|
||||
|
||||
## CodeGraph and source findings
|
||||
|
||||
- `.codegraph/` exists, so `codegraph explore` was attempted first for the PPI/KaKs dialog and scheduling call path. It returned no usable output; targeted source reads were therefore used.
|
||||
- `apps/web/vue3/index.html:14058` — `runPpiSpeciesResolutionPreflight()` owns the species-resolution request and blocking alerts.
|
||||
- `apps/web/vue3/index.html:14354` — `ensurePpiKaksDependencyOwner()` owns per-owner reference decisions and download behavior.
|
||||
- `apps/web/vue3/index.html:14491` — `ensureOrthologProducerForPpiKaks()` globally gates the batch.
|
||||
- `apps/web/vue3/index.html:14980` — `startBatchExecute()` returns early when that gate is false.
|
||||
- `apps/web/vue3/index.html:15037` — stored pause states are applied before dependency execution.
|
||||
- `apps/web/vue3/index.html:15600-15900` — the shared stage confirmation queue and confirmation handlers control subsequent dialogs.
|
||||
- `apps/web/vue3/index.html:12881` and `apps/web/vue3/index.html:18927` — manual confirmation and execution entry points must be guarded before scope loading, timers, or submission.
|
||||
- `apps/web/vue3/index.html:21789-21882` — background batch execution is a separate scheduling path and must preserve `paused` rather than converting the circuit breaker into `failed`.
|
||||
- `apps/web/vue3/index.html:13713` — an already-running legacy reference-download monitor can otherwise resume temporarily disabled owners.
|
||||
- Auto-inserted Ortholog ownership is already recorded with `autoInsertedFor`, `dependencyOwner`, and `referencePolicySource` near `apps/web/vue3/index.html:14243-14339`.
|
||||
- PlanningAgent dependency expansion can add the formal KaKs Ortholog without those frontend-only owner fields. The plan response exposes the original intent stages near `apps/web/vue3/main.py:22079-22089`, but the current frontend plan handoff near `apps/web/vue3/index.html:22617-22625` does not preserve them. The frontend must retain that evidence to distinguish an explicit independent Ortholog from a KaKs-only dependency.
|
||||
|
||||
## Allowed existing APIs and patterns
|
||||
|
||||
- Use the existing Vue reactive stage state: `stageExecutionStatus`, `stagePauseReasons`, `recordStagePause()`, and the batch pause summary.
|
||||
- Use existing stage identity helper `getStageId()` and existing stage parameter objects; do not invent backend request fields.
|
||||
- Follow the existing test pattern that extracts real JavaScript source blocks and executes them with Node in `tests/unit/test_ppi_owner_scheduling_task5.py`.
|
||||
- Keep `/api/ppi/species-resolution`, `/api/reference-bundle/download`, `/api/execute`, CLI entry points, and Manifest schemas unchanged.
|
||||
|
||||
## Approach
|
||||
|
||||
### Phase 1 — RED: executable regression tests
|
||||
|
||||
Modify only frontend scheduling tests first.
|
||||
|
||||
1. Add a behavior-level test for a small circuit-breaker helper extracted from the real HTML source. Given a plan containing ordinary stages, `ppi`, `kaks`, a PPI-owned Ortholog, and an independent Ortholog, assert that only `ppi`, `kaks`, and the owned Ortholog are selected for temporary pause.
|
||||
- Cover PPI-owned, KaKs-owned, jointly owned, unrelated-auto, explicit-independent, and unknown-source Ortholog examples.
|
||||
- Preserve the stage list itself; only statuses change.
|
||||
2. Add a batch-start control-flow test or source-extracted harness proving that, while disabled, the existing `ensureOrthologProducerForPpiKaks()` callback is not invoked and scheduling proceeds.
|
||||
3. Add a manual-confirmation guard test proving that PPI/KaKs resolve as paused and do not reach the execute-submit callback.
|
||||
4. Add a shared-submit guard test covering stale open dialogs or alternate callers.
|
||||
5. Add a background-batch test proving the owners and owned Ortholog remain paused rather than failed while unrelated background stages continue.
|
||||
6. Add a stale-monitor test proving `resumePpiKaksPausedStages()` cannot restore disabled owners to pending.
|
||||
7. Add negative assertions that the disabled branch does not invoke species-resolution, reference-bundle download, or PPI/KaKs `/api/execute` submission.
|
||||
8. Run the focused tests and confirm they fail because the circuit-breaker behavior does not exist yet.
|
||||
|
||||
Documentation references:
|
||||
|
||||
- `tests/unit/test_ppi_owner_scheduling_task5.py` — `_source_block()` and Node execution pattern.
|
||||
- `tests/unit/test_ppi_frontend_display_contract.py` — frontend contract assertions.
|
||||
|
||||
Anti-pattern guards:
|
||||
|
||||
- Do not add tests that only assert a constant string exists.
|
||||
- Do not mock the helper under test.
|
||||
- Do not weaken existing PPI owner-isolation tests.
|
||||
|
||||
### Phase 2 — GREEN: one frontend circuit breaker
|
||||
|
||||
Modify `apps/web/vue3/index.html` only.
|
||||
|
||||
1. Add one plainly named temporary feature declaration near batch execution state, for example `const PPI_KAKS_TEMPORARILY_DISABLED = true`, with a removal condition comment rather than a configurable framework.
|
||||
2. Add one pure ownership predicate for an Ortholog stage. Its canonical condition is that `autoInsertedFor` contains `ppi` or `kaks`; `dependencyOwner` and the two existing PPI/KaKs `referencePolicySource` values may be accepted only as compatibility evidence for already-built stage objects. Never infer ownership merely because an Ortholog is present in the same batch.
|
||||
- When the plan contains an untagged Ortholog added by dependency expansion, compare it with the preserved original intent stages: explicit `ortholog` intent means independent; absent `ortholog` intent combined with requested `kaks` means KaKs-owned.
|
||||
- If intent provenance is absent or ambiguous, preserve Ortholog rather than guessing.
|
||||
3. Add one helper that records the temporary pause reason for planned PPI/KaKs and owned Ortholog stages. It must not reuse `ppiKaksPauseStageIds()` because that helper currently adds any planned Ortholog and would violate independent-Ortholog preservation.
|
||||
4. In `startBatchExecute()`, branch once:
|
||||
- disabled: record the temporary pause set and continue;
|
||||
- enabled: execute the existing `ensureOrthologProducerForPpiKaks()` path unchanged.
|
||||
5. Preserve the existing status initialization and `applyStoredPpiKaksPauseStatuses()` flow, or introduce a narrowly named equivalent if the existing pause object cannot express precise stage IDs.
|
||||
6. Add an early guard in `showConfirm()` before scope loading, dialog display, and the automatic countdown. Add a second guard at the start of `doExecute()` for a stale dialog opened before deployment/page-state refresh.
|
||||
7. Add a final shared-submit guard before PPI/KaKs preflight and `/api/execute`. It is defense in depth for alternate callers, but its result must be interpreted as `paused`, never as a failed stage.
|
||||
8. Apply the same pause selection to `startBackgroundBatchExecution()` so background execution excludes disabled owners and owned Ortholog, continues unrelated stages, and reports paused counts.
|
||||
9. Prevent `resumePpiKaksPausedStages()` from changing temporarily disabled owners back to pending when an older reference-download monitor completes.
|
||||
10. Do not change backend stage availability or returned plans.
|
||||
11. At the frontend plan handoff, preserve the existing plan API's original intent stage list in batch state solely as ownership provenance; do not change the plan API schema or PlanningAgent.
|
||||
|
||||
Anti-pattern guards:
|
||||
|
||||
- No edits to `gfa_core/mas/agents/planning_v2.py`, `apps/web/pipeline_backend.py`, `apps/web/vue3/main.py`, CLI scripts, contracts, or Manifest writers.
|
||||
- No job-ID special case.
|
||||
- No broad `ortholog` pause by stage name alone.
|
||||
- No fake completed state, empty success Manifest, or swallowed failure reported as success.
|
||||
- No path that represents the temporary circuit breaker as an exception and then maps it to `failed`.
|
||||
- No refactor of the large Vue file beyond the helpers required for this behavior.
|
||||
|
||||
### Phase 3 — GREEN verification and focused regression
|
||||
|
||||
1. Run the new focused tests.
|
||||
2. Run all existing PPI frontend/owner tests.
|
||||
3. Run the repository's HTML/JavaScript validation command used for `apps/web/vue3/index.html`.
|
||||
4. Inspect the diff and verify every production change maps to an acceptance criterion.
|
||||
|
||||
Expected focused commands:
|
||||
|
||||
```bash
|
||||
/home/xia-lab/miniconda3/envs/test4py311/bin/python -m pytest -q \
|
||||
tests/unit/test_ppi_owner_scheduling_task5.py \
|
||||
tests/unit/test_ppi_frontend_display_contract.py
|
||||
/usr/bin/node --version
|
||||
python validate_js.py
|
||||
```
|
||||
|
||||
If the test environment provides a narrower JS validator command, use the repository-documented command rather than inventing one.
|
||||
|
||||
Known baseline note: the audit observed `test_ppi_frontend_display_contract.py` at 17 passed / 1 unrelated existing failure in the `openPpiResetDialog()` string contract. Record the exact fresh baseline before RED, and do not attribute that failure to this revision or weaken it to obtain green.
|
||||
|
||||
### Phase 4 — real UI/API validation
|
||||
|
||||
Use a non-destructive test job or a justified copied/subset job; do not mutate the user's completed scientific artifacts.
|
||||
|
||||
1. Request a plan containing `ppi`, `kaks`, and at least one unrelated stage; verify the plan still displays all stages.
|
||||
2. Start the batch in the UI.
|
||||
3. Verify PPI and KaKs immediately show paused with the exact temporary reason.
|
||||
4. Verify no reference-species confirmation/download modal appears.
|
||||
5. Verify the next unrelated stage confirmation appears and can be submitted.
|
||||
6. Verify browser/network and server logs contain no new `/api/ppi/species-resolution`, `/api/reference-bundle/download`, or `/api/execute` PPI/KaKs request for this run.
|
||||
7. Verify no new PPI/KaKs success Manifest exists.
|
||||
8. Verify an explicitly requested independent Ortholog is not paused; separately verify an owned auto-inserted Ortholog is paused.
|
||||
9. Exercise the background-batch path and verify disabled stages remain paused while an unrelated background stage proceeds.
|
||||
10. Simulate completion of a pre-existing reference-download monitor and verify it cannot resume disabled owners.
|
||||
|
||||
This is workflow validation, not a biological full-data validation, because no PPI/KaKs method or output is being changed.
|
||||
|
||||
## Affected layers and synchronization
|
||||
|
||||
| Layer | Change |
|
||||
|---|---|
|
||||
| Frontend scheduling | Temporary availability declaration, precise pause selection, batch gate bypass, manual UI guard |
|
||||
| Frontend status/result summary | Reuse paused status and pause reason; ensure it is not counted as failed/completed |
|
||||
| Frontend tests | Add behavior-level red/green regression coverage |
|
||||
| PlanningAgentV2 | No change; stages remain visible |
|
||||
| Backend/API | No change |
|
||||
| CLI/skills | No change |
|
||||
| Stage contracts and identity | No change |
|
||||
| Manifest/reproducibility | No change; paused stages produce no success Manifest |
|
||||
| Diagnosis | No backend diagnostic change; the UI pause reason is explicit |
|
||||
|
||||
## Biological-method assumptions and fallbacks
|
||||
|
||||
- This circuit breaker makes no biological claim and does not alter PPI, Ka/Ks, STRING species selection, Ortholog mapping, or reference bundle methodology.
|
||||
- No fallback species is selected automatically.
|
||||
- Existing backend/CLI paths remain available only for deliberate development use.
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
1. **Owned Ortholog is not tagged in an older plan.** Preserve it rather than guessing. It may run independently, but an explicit stage must not be silently disabled. Add logging/diagnostic text if encountered.
|
||||
2. **A later UI path submits directly to `/api/execute`.** Search all in-file submission paths for PPI/KaKs and cover the shared submission boundary with a regression test.
|
||||
3. **Pause state is overwritten during batch initialization.** Verify the pause is stored before initialization and re-applied after stage statuses are initialized.
|
||||
4. **Background execution maps the guard to failure.** Apply pause filtering before its execute/catch path and assert paused accounting explicitly.
|
||||
5. **Background download monitor survives from an earlier attempt.** This change prevents new monitors and prevents their callbacks from resuming disabled owners; it does not cancel an already submitted external task. Document that existing tasks are unaffected.
|
||||
6. **Temporary flag becomes permanent.** Use one declaration and a clear removal condition; do not distribute hard-coded booleans.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Repairing PPI, Ka/Ks, Ortholog, STRING selection, or NCBI downloads.
|
||||
- Changing dependency graphs or PlanningAgent output.
|
||||
- Cancelling already running reference-bundle tasks.
|
||||
- Backfilling or rewriting existing manifests.
|
||||
- Changing API/CLI access for developers.
|
||||
|
||||
## Implementation handoff checklist
|
||||
|
||||
- [ ] User explicitly authorizes implementation after reviewing this plan.
|
||||
- [ ] Focused regression tests are written and observed failing before production edits.
|
||||
- [ ] Only `apps/web/vue3/index.html` and focused frontend test files are changed.
|
||||
- [ ] Independent Ortholog preservation is covered by an executable test.
|
||||
- [ ] Focused tests pass and existing PPI frontend tests introduce no new failures; any pre-existing baseline failure is reported separately.
|
||||
- [ ] JavaScript validation passes.
|
||||
- [ ] UI/network/log validation confirms unrelated analysis continues.
|
||||
- [ ] Service restart requirement is reported; Celery restart should not be required for a frontend-only change unless deployment packaging dictates otherwise.
|
||||
@@ -0,0 +1,78 @@
|
||||
# Findings
|
||||
|
||||
## Historical hypotheses to re-check
|
||||
|
||||
- PPI Gm LIS/SoyBase identifiers may not resolve to STRING identifiers.
|
||||
- PPI failed Manifest may lose the skill-level issue and ID-resolution artifact.
|
||||
- PPI technical reference may leak into biological scope/planning.
|
||||
- KaKs may contain user-specific absolute tool paths.
|
||||
- KaKs project skill may be a non-executing placeholder.
|
||||
- KaKs method/provenance may disagree with the actual PAML yn00/MUSCLE commands.
|
||||
- KaKs legacy execution may silently default to Arabidopsis.
|
||||
|
||||
## Current evidence
|
||||
|
||||
- Current main HEAD is `7399b0bf1`; WTX worktree is independently ahead at `68d305f5d`, so WTX history cannot be assumed to equal current main.
|
||||
- Current main `apps/cli/run_kaks_v2.py` still hard-codes `/home/xzq/miniconda3/envs/test4py311` in the shebang, PATH/LIB setup, MUSCLE, pal2nal, yn00, and Rscript paths.
|
||||
- Current main `skills/kaks/run.py:83-84` still logs completion while explicitly marked TODO placeholder.
|
||||
- Current KaKs CLI defaults `neutral_rate=7.08e-9`, contains Ath-specific data comments/diagnostics, and records yn00 as `KaKs_Calculator (YN model)`.
|
||||
- Current legacy backend KaKs path still defaults missing reference to `Ath` and invokes hard-coded `/home/xzq/miniconda3/bin/conda`.
|
||||
- Current PlanningAgentV2 still maps explicit PPI species through a five-item static map and may write it into shared `referenceSpecies/refSpeciesList` with `scopeMode=target_plus_reference`.
|
||||
- Current stage tool description still says PPI requires ortholog mapping and defaults PPI `species=ath`, `score=200`.
|
||||
- Current PPI normalization and failed-manifest ownership still require detailed source tracing; search confirms the same issue codes and layers exist.
|
||||
- PPI planning root cause is current and explicit: `_generate_default_stages()` mutates the shared intent params before per-stage params are built. An explicit PPI species in a five-item static map becomes shared `referenceSpecies/refSpeciesList` and may set `scopeMode=target_plus_reference`, so technical STRING ownership can alter biological scope for every later stage.
|
||||
- `normalize_gene_ids_for_string()` currently contains only the rice transcript-to-gene rule. The PPI skill can additionally consume an evidence TSV through `ReferenceAliasIndex`, so the current Gm question depends on whether backend supplies an appropriate job-local `--id-map`; absence of a Gm mapping rule alone is not yet enough to conclude the production path must fail.
|
||||
- Stage tool/catalog drift is confirmed: PPI metadata still describes ortholog mapping as required and defaults `species=ath`, inconsistent with the newer snapshot/direct-target execution path.
|
||||
- PPI skill failed manifests are capable of preserving exact `issue_code`, metrics, artifacts, and provenance. The suspected information loss must therefore occur after the skill exits, in backend finalization/merge logic.
|
||||
- Backend projected PPI requires an ID-map and passes `--id-map`, but direct STRING-supported targets only pass it opportunistically if `<input>/<ppi_storage_key>/<storage_key>.pep_cds_idmap.tsv` happens to exist. For direct Gm, `ppi_storage_key` falls back to the STRING species token (`gma`/taxon), while job identifiers use LIS/SoyBase namespace; there is no current Gm transform in policy. This reproduces the structural precondition for 0/56 mapping unless a correctly named job-local ID map is present.
|
||||
- PPI failure ownership defect is confirmed in current source: the skill writes the exact failed manifest under `workspace/ppi/<prefix>.manifest.json`, but backend copies skill artifacts only when exit code is 0. On nonzero exit, it never reads/copies/merges that manifest; it creates or mutates `final/<prefix>.ppi.manifest.json` with generic error/provenance and can leave `artifacts={}`. This discards the skill-level ID-resolution issue/metrics/artifact and can also mutate a pre-existing final manifest.
|
||||
- Existing tests cover successful PPI artifact synchronization and owner isolation, but the initial search found no test asserting skill-failed-manifest preservation through backend finalization.
|
||||
- KaKs has two materially different backend paths. The scope path is fail-closed for missing target/reference species and files and uses `sys.executable`; the legacy no-scope path silently defaults the reference to `Ath`, searches Ft/Ath-specific filenames, and invokes `/home/xzq/miniconda3/bin/conda`.
|
||||
- Both KaKs paths call the same CLI, whose shebang/environment and MUSCLE/pal2nal/yn00/Rscript executables are hard-coded under `/home/xzq`. Therefore scope correctness does not solve deployment portability on a `/home/xia-lab` installation.
|
||||
- `run_kaks_v2.py` does generate a failed manifest and exits nonzero when no pair succeeds, but the backend returns a generic stage manifest on CLI failure and does not explicitly surface/merge the CLI failed manifest into status/diagnosis.
|
||||
- KaKs execution method is PAML `yn00` after MUSCLE protein alignment and pal2nal codon projection, but CLI Manifest names the calculator `KaKs_Calculator (YN model)`. The project skill/config additionally advertise CodeML M0/M1/M7/M8 and MUSCLE 5.1 while the authoritative backend never invokes the placeholder skill and the CLI uses MUSCLE 3-style `-in/-out` syntax.
|
||||
- The default neutral rate `7.08e-9` is labeled Arabidopsis in the project skill but is passed to every target/reference run unless overridden; the manifest records the number without species applicability/source provenance.
|
||||
- The current test inventory is rich for PPI/KaKs preflight isolation and frontend dependency ownership, but the search found no dedicated tests that execute `run_kaks_v2.py`, validate its tool resolver, assert nonzero failed-manifest propagation, or audit method/version truthfulness.
|
||||
- The normal frontend execution path is intentionally disabled for both stages: `ppiKaksTemporaryCircuitBreakerEnabled = true`. Batch execution marks PPI and KaKs paused, blocks their `/api/execute` submission, and also pauses an Ortholog stage that was auto-inserted solely for unavailable PPI/KaKs owners. Manual execution is blocked with “功能修订中,当前暂不可用”.
|
||||
- Therefore the most immediate current user-visible state is not “PPI/KaKs sometimes fail”; it is “the frontend never submits either stage”. Backend/CLI defects remain latent and can still affect direct API or future re-enablement.
|
||||
- The circuit breaker has dedicated unit tests proving later non-PPI/KaKs stages continue after pause; it is deliberate current behavior, not an accidental deadlock.
|
||||
- Environment classification: the mounted SJ/WTX-style worktree has broken absolute `input` and `workspace` symlinks into `/home/xzq`, while `data` is valid. On the actual 5090 production host, every hard-coded KaKs executable checked (MUSCLE, pal2nal, yn00, Rscript, conda) exists. Thus hard-coded paths are a confirmed portability/design defect and break the `/home/xia-lab` worktree, but they do not by themselves prove current 5090 production execution failure.
|
||||
- Existing Gm/TCP evidence exactly confirms cross-layer PPI loss: skill manifest has `ppi_string_id_resolution_failed`, 0/56 mappings, metrics and an ID-resolution TSV; final manifest has no issue code/metrics and empty artifacts. Both carry a misleading `resource_error` provenance and unused `reference_species=At` for a direct Gm query.
|
||||
- Existing Os/PEBP KaKs real run proves the core production computation can succeed on 5090: 17/17 pairs, result table and plots. It simultaneously confirms methodology drift: actual MUSCLE 3.8 + PAML yn00 is recorded as `KaKs_Calculator`, yn00 version is `unknown`, and the universal neutral-rate value is used without source/applicability metadata.
|
||||
- Current full PPI/KaKs-named unit suite result on production: 151 collected, 150 passed, 1 failed. The failure is the natural-language PPI Reset entry point.
|
||||
- PPI Reset root cause is current: `isPpiResetIntent()` is defined and detects the intended phrases, and the dialog/API implementation exists, but the predicate is never called by `handleNlInput()` and there is no `openPpiResetDialog()` call. The button path `openPpiResetDialog(stage)` still exists, so this is an orphaned natural-language route rather than a broken reset backend.
|
||||
- Current SJ main and WTX contain byte-identical PPI/KaKs core files: KaKs CLI/skill/config/docs, PPI policy/skill/provenance, backend, PlanningAgentV2, and stage tools. Only `apps/web/vue3/index.html` differs among the checked files. Therefore WTX does not contain a hidden backend/CLI PPI/KaKs fix that is merely awaiting merge.
|
||||
- The circuit breaker was introduced by commit `2623d3114` on 2026-08-06, although that commit is titled `feat: update header and hide extension`. This is a traceability problem: a material PPI/KaKs runtime behavior change was bundled under an unrelated frontend commit title. WTX does not contain the circuit breaker; SJ main does.
|
||||
- Diagnosis drift is confirmed. PPI diagnostics derive their code primarily from `ppi_provenance.diagnosis_code`; the real Gm final provenance says `ppi_string_resource_error`, so DiagnosisAgent cannot recover the exact skill issue after backend discarded it. Its declared PPI failure list includes `ppi_string_id_mapping_low` but not the skill's actual `ppi_string_id_resolution_failed` code.
|
||||
- KaKs DiagnosisAgent correctly detects missing reference scope, but its recommendation says to use a project default reference when unspecified. That conflicts with the KaKs contract's fail-closed rule (“不得自动换用 At”), so diagnosis can recommend the same unsafe legacy behavior the contract forbids.
|
||||
- Minimal current PlanningAgentV2 reproduction confirms the PPI scope bug, not just static suspicion. Input target=`Gm`, `scopeMode=target_only`, `ppiSpecies=gma`, stages=`motif,ppi` produces both stages with `referenceSpecies=['Gm']`, `scopeMode=target_plus_reference`, `analysisRoute=multi_species`, and scope/member-set IDs containing `ref-Gm`. Thus a PPI technical selector changes unrelated motif biological identity and even makes the target its own reference.
|
||||
- The failed Gm job contains `input/Gm.id_alias.tsv` (168,133 lines), but it maps LIS raw IDs to essentially the same LIS canonical IDs; it is not a STRING alias map. Backend also does not discover this file, because PPI looks for `<input>/<ppi_storage_key>/<ppi_storage_key>.pep_cds_idmap.tsv`. Resolution rows show only raw/versionless LIS candidates and all 56 unresolved. Root cause is not “STRING lacks Gm”; it is the absence of a validated LIS/SoyBase→STRING namespace mapping plus no compatible evidence-discovery contract.
|
||||
- Live 9010 `/api/stage-status` confirms the user-visible Gm PPI status is generic/wrong: failed, zero artifacts, error only “Manifest has no artifacts”, empty diagnosis report, and provenance `ppi_string_resource_error` with stale `reference_species=At`. The exact 0/56 ID-resolution root cause and TSV do not reach status.
|
||||
- Live 9010 status for historical `Sv.HKL` confirms PPI computation is not globally broken: the stage is completed with 6/6 IDs mapped, 15 interactions, six artifacts, target-only biological scope, and STRING taxon 4555 as technical ownership.
|
||||
- Live 9010 full-identity status for historical `Os.PEBP` confirms KaKs computation is not globally broken: the stage is completed with 17/17 pairs and three aggregate artifacts under the expected target/reference scope and member-set identity.
|
||||
|
||||
## Final classification
|
||||
|
||||
### Confirmed current defects
|
||||
|
||||
1. PPI/KaKs are globally blocked in the normal frontend execution path by the enabled temporary circuit breaker; this is deliberate pause behavior, not a backend exception.
|
||||
2. PPI planning mutates shared biological scope from a technical STRING selector, contaminating unrelated stages and allowing target=reference scope.
|
||||
3. PPI nonzero-exit finalization discards the skill's exact issue, metrics, and resolution artifact, causing generic and sometimes stale user-facing diagnosis.
|
||||
4. Direct Gm PPI lacks a validated LIS/SoyBase-to-STRING mapping/evidence-discovery contract; the available job alias table is neither compatible nor consumed.
|
||||
5. KaKs has divergent scope and legacy backend semantics; the legacy path silently defaults to Ath contrary to the fail-closed contract.
|
||||
6. The project KaKs skill is a placeholder that reports completion without running KaKs, while production uses a separate CLI path.
|
||||
7. KaKs executable/environment paths are deployment-specific and its declared methodology/provenance do not truthfully match MUSCLE 3 + pal2nal + PAML yn00 execution.
|
||||
8. Natural-language PPI Reset is orphaned because its intent predicate is never invoked, despite the dialog/API and button path existing.
|
||||
|
||||
### Confirmed but currently latent or environment-dependent
|
||||
|
||||
- PPI/KaKs backend defects are hidden from the normal UI while the frontend circuit breaker remains enabled, but remain reachable by direct API or future re-enablement.
|
||||
- KaKs hard-coded `/home/xzq` paths are a portability defect and break the mounted `/home/xia-lab` worktree; those exact tools currently exist on the 5090 production host, so this alone is not evidence of a current production computation failure.
|
||||
- KaKs failed-manifest loss is strongly indicated by current source structure, but no real failed KaKs run was used to confirm the final user-visible outcome in this diagnosis.
|
||||
|
||||
### Explicitly disproved overbroad conclusions
|
||||
|
||||
- PPI is not globally nonfunctional: a validated Sv projected run completed successfully.
|
||||
- KaKs is not globally nonfunctional: an Os-to-At real run completed 17/17 pairs.
|
||||
- STRING support for Gm is not the root cause of the Gm failure; identifier namespace/evidence wiring is.
|
||||
- WTX does not contain an unsynchronized hidden core fix for these paths; checked core files are byte-identical to SJ main.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Progress
|
||||
|
||||
- 2026-08-08: Read the 2026-07-22 production PPI record and 2026-08-06 WTX audit record.
|
||||
- 2026-08-08: Started current-state diagnosis under Karpathy, systematic-debugging, and CropAI constraints.
|
||||
- 2026-08-08: Attempted CodeGraph-first PPI and KaKs trace; both queries returned no output, so diagnosis will use targeted source search.
|
||||
- 2026-08-08: Established Git baseline and searched current PPI/KaKs planning, backend, CLI, skill, provenance, and diagnosis symbols.
|
||||
- 2026-08-08: Traced current PPI planning mutation, direct/projected ID-map wiring, and nonzero-exit manifest handling to concrete source lines.
|
||||
- 2026-08-08: Confirmed the current frontend circuit breaker blocks all manual and batch PPI/KaKs submissions while allowing unrelated stages to continue.
|
||||
- 2026-08-08: Compared mounted-path health with production tool availability and inspected the Gm PPI and Os/At KaKs real manifests.
|
||||
- 2026-08-08: Ran all 11 PPI/KaKs-named unit files on production (150 passed, 1 failed) and traced the failure to an unused PPI Reset intent predicate.
|
||||
- 2026-08-08: Compared current SJ main and WTX key files; core PPI/KaKs layers are identical, frontend index differs.
|
||||
- 2026-08-08: Traced the SJ-only circuit breaker to commit `2623d3114` and confirmed it was bundled under an unrelated commit title.
|
||||
- 2026-08-08: Audited stage-status/DiagnosisAgent issue handling and found PPI issue-code loss plus a KaKs fail-closed recommendation contradiction.
|
||||
- 2026-08-08: Reproduced PlanningAgentV2 PPI scope contamination with a minimal no-LLM call and confirmed the Gm job has an unwired alias file.
|
||||
- 2026-08-08: Queried live 9010 Gm PPI stage-status and confirmed exact root-cause/artifact loss reaches the user-facing status layer.
|
||||
- 2026-08-08: Queried live 9010 successful Sv PPI and Os/At KaKs statuses, proving both computation cores can succeed under supported identities.
|
||||
- 2026-08-08: Completed issue classification without modifying product code or restarting services.
|
||||
@@ -0,0 +1,32 @@
|
||||
# PPI/KaKs Current-State Diagnosis
|
||||
|
||||
## Goal
|
||||
|
||||
Re-evaluate the historical PPI/KaKs findings against the current main-project source, tests, deployment paths, and existing real-job evidence. Diagnose only; do not edit product code.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- Every reported issue has current source or runtime evidence.
|
||||
- Historical-only or already-fixed findings are explicitly separated.
|
||||
- Each issue is traced to its owning layer and root cause.
|
||||
- PPI and KaKs are assessed independently so one stage is not used as evidence for the other.
|
||||
|
||||
## Phases
|
||||
|
||||
- [complete] Phase 1: Establish current Git/code/deployment baseline.
|
||||
- [complete] Phase 2: Trace PPI planning -> preflight -> backend -> skill -> manifest/status.
|
||||
- [complete] Phase 3: Trace KaKs planning -> preflight -> backend -> CLI/skill -> manifest/status.
|
||||
- [complete] Phase 4: Run focused tests and inspect existing real-job evidence.
|
||||
- [complete] Phase 5: Classify confirmed, fixed, environmental, and unverified findings.
|
||||
|
||||
## Constraints
|
||||
|
||||
- No production code edits.
|
||||
- No service restart or new real analysis task unless separately authorized.
|
||||
- Do not infer current defects solely from the July/August records.
|
||||
|
||||
## Errors Encountered
|
||||
|
||||
| Error | Attempt | Resolution |
|
||||
|---|---:|---|
|
||||
| `codegraph explore` returned no output for both PPI and KaKs traces | 1 | Treat the mounted index as unavailable; fall back to targeted `rg` and symbol reads without rebuilding it. |
|
||||
@@ -0,0 +1,95 @@
|
||||
# PPI/KaKs 最小可用修订计划(Karpathy 审核版)
|
||||
|
||||
## 唯一目标
|
||||
|
||||
让正常家族分析中的 PPI 和 KaKs 能从前端真实提交并完成;任一阶段失败时,后续无关阶段仍继续调度。
|
||||
|
||||
## 明确假设
|
||||
|
||||
1. 本轮生产环境是 5090;现有 `/home/xzq` KaKs 工具路径在该环境可用,因此不处理可移植性。
|
||||
2. KaKs 的 scope + Ortholog + `run_kaks_v2.py` 路径已经有 Os→At 17/17 成功证据,除非标准前端请求没有进入这条路径,否则不改 KaKs 后端或算法。
|
||||
3. 当前必须支持用户实际失败的 Gm/TCP PPI;Gm ID 映射只能使用 STRING 官方接口/别名证据,不接受字符串猜测。
|
||||
4. “不阻塞后续”只指调度隔离,不要求本轮重构诊断、Manifest 或状态体系。
|
||||
|
||||
## 最终验收标准
|
||||
|
||||
1. Gm/TCP 的 56 个真实 ID 能完成 PPI,而不是 0/56。
|
||||
2. Os→At PEBP KaKs 仍完成 17/17 pairs。
|
||||
3. 前端手动与批量执行都会向 `/api/execute` 提交 PPI/KaKs,不再标记临时 `paused`。
|
||||
4. `ppiSpecies` 不再改变 motif 等无关阶段的 biological scope。
|
||||
5. 分别模拟 PPI 和 KaKs 执行失败,后续无关阶段仍被提交。
|
||||
|
||||
## 第 0 步:只读确认两个执行前提
|
||||
|
||||
### PPI 映射前提
|
||||
|
||||
- 用现有 Gm/TCP 56 个 ID 调用当前 STRING 官方映射入口/检查官方 alias 数据。
|
||||
- 记录原始 ID、提交 ID、STRING ID 和来源。
|
||||
- 只有确认一条可重复的官方映射路径后才写代码;如果 STRING 官方证据不能映射这些 ID,停止实施并报告数据前提,不制造启发式规则。
|
||||
|
||||
### KaKs 路由前提
|
||||
|
||||
- 从正常前端请求体追到 `/api/execute` 和 `pipeline_backend.py`,确认是否携带 scope、reference species 和 Ortholog mapping。
|
||||
- 若已进入现有成功 scope 路径,KaKs 代码零修改。
|
||||
- 若只缺一个现有参数,只修该参数传递点;不改 CLI 和分析方法。
|
||||
|
||||
## 第 1 步:修复 PPI scope 污染
|
||||
|
||||
### 必要修改
|
||||
|
||||
- `gfa_core/mas/agents/planning_v2.py`
|
||||
- 移除把 `ppiSpecies` 写入共享 `referenceSpecies/refSpeciesList/scopeMode` 的逻辑。
|
||||
- 保留 PPI stage 自己的 `ppiSpecies` 参数。
|
||||
- 不改主 Agent、IntentAgent、其他 stage 规划和 scope 生成。
|
||||
|
||||
### 验证
|
||||
|
||||
- 回归用例:target=`Gm`、scopeMode=`target_only`、ppiSpecies=`gma`、stages=`motif,ppi`。
|
||||
- motif 保持 target-only;PPI 收到 `ppiSpecies=gma`。
|
||||
- 只增加这一条回归测试,并运行相关现有规划测试。
|
||||
|
||||
## 第 2 步:接入已证实的 Gm→STRING 映射
|
||||
|
||||
### 必要修改
|
||||
|
||||
- 优先只改 `apps/web/pipeline_backend.py`,在现有 PPI 命令构造点把第 0 步确认的映射证据传给现有 resolver。
|
||||
- 只有现有 resolver 无法读取该证据格式时,才最小修改 `gfa_core/ppi_string_policy.py`。
|
||||
- 不新增通用映射框架,不扩展其他物种,不做 ID 猜测规则。
|
||||
|
||||
### 验证
|
||||
|
||||
- 完整运行 `job_1785919737565_421af1` 的 56 个 Gm/TCP ID。
|
||||
- 验证映射数不再是 0/56、PPI stage 成功且产生网络产物。
|
||||
- 回归 Sv/HKL 已成功路线,防止破坏现有 projected PPI。
|
||||
|
||||
## 第 3 步:解除熔断并验证调度隔离
|
||||
|
||||
### 必要修改
|
||||
|
||||
- `apps/web/vue3/index.html`
|
||||
- 将 PPI/KaKs 临时熔断关闭;优先采用最小开关变更,不删除现有辅助代码。
|
||||
- 不在前端补规划,不改变阶段列表或下游 Agent。
|
||||
- KaKs 仅在第 0 步确认标准请求参数丢失时,修改那个唯一传递点。
|
||||
- 如果现有“单阶段失败后继续循环”逻辑已经通过测试,不修改调度实现。
|
||||
|
||||
### 验证
|
||||
|
||||
- 手动执行 PPI、KaKs:确认真实 `/api/execute` 请求。
|
||||
- 批量正常场景:Gm PPI 和 Os→At KaKs 分别通过真实数据完成。
|
||||
- 前端调度回归:分别让 PPI、KaKs 的执行请求失败,断言下一无关阶段仍被提交;若现有代码已满足,只保留测试、不改实现。
|
||||
- 最后检查相关单测、业务 diff,并重启 9010 后用实际 API 再验证一次。
|
||||
|
||||
## 本轮明确不做
|
||||
|
||||
- 不修 PPI Reset 自然语言入口。
|
||||
- 不改失败 Manifest 合并、DiagnosisAgent 或通用状态体系;它们不阻止阶段执行和后续调度。
|
||||
- 不实现 KaKs placeholder skill。
|
||||
- 不统一 `/home/xzq` 路径,不做跨机器可移植性重构。
|
||||
- 不修改 KaKs 算法、模型、图表、方法文档或 legacy 路径。
|
||||
- 不扩展更多 STRING 物种映射。
|
||||
- 不重构主 Agent、IntentAgent、下游 Agent 或前端规划。
|
||||
- 不清理无关技术债。
|
||||
|
||||
## 顺序约束
|
||||
|
||||
必须按 0 → 1 → 2 → 3 执行。只有 PPI scope 和 Gm 映射都通过真实数据后才关闭前端熔断。
|
||||
+282
-23
@@ -25,6 +25,7 @@ from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Sequence
|
||||
import hashlib
|
||||
import os
|
||||
import shlex
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
@@ -231,8 +232,17 @@ RNASEQ_BIN = str(rnaseq_bin())
|
||||
prepend_path(RNASEQ_BIN)
|
||||
|
||||
|
||||
MAX_TOOL_THREADS = 8
|
||||
|
||||
|
||||
def resolve_thread_plan(cpu: int, sample_parallel_jobs: int = 1, threads_per_sample: Optional[int] = None) -> Dict[str, int]:
|
||||
"""Resolve conservative RNA-seq sample/thread settings from CLI args."""
|
||||
"""Resolve conservative RNA-seq sample/thread settings from CLI args.
|
||||
|
||||
Per-tool thread count is capped at MAX_TOOL_THREADS because fastp (and most
|
||||
QC tools) are single-threaded for their core algorithm; oversized --thread
|
||||
values only add gzip-thread contention without speeding up filtering. Real
|
||||
parallelism is achieved via chunked splitting (see run_fastp).
|
||||
"""
|
||||
total_cpu = max(1, int(cpu or 1))
|
||||
jobs = max(1, int(sample_parallel_jobs or 1))
|
||||
if threads_per_sample is None or int(threads_per_sample) < 1:
|
||||
@@ -241,6 +251,7 @@ def resolve_thread_plan(cpu: int, sample_parallel_jobs: int = 1, threads_per_sam
|
||||
threads = max(1, int(threads_per_sample))
|
||||
if jobs * threads > total_cpu:
|
||||
threads = max(1, total_cpu // jobs)
|
||||
threads = min(threads, MAX_TOOL_THREADS)
|
||||
return {"cpu": total_cpu, "sample_parallel_jobs": jobs, "threads_per_sample": threads}
|
||||
|
||||
|
||||
@@ -378,6 +389,253 @@ def run_cutadapt(
|
||||
raise
|
||||
|
||||
|
||||
def _build_fastp_cmd(
|
||||
fastp_bin: Path,
|
||||
r1: Path,
|
||||
r2: Path,
|
||||
out_r1: Path,
|
||||
out_r2: Path,
|
||||
html_report: Path,
|
||||
json_report: Path,
|
||||
cpu: int,
|
||||
detect_adapter_for_pe: bool,
|
||||
cut_front: bool,
|
||||
cut_tail: bool,
|
||||
cut_window_size: int,
|
||||
cut_mean_quality: int,
|
||||
qualified_quality_phred: int,
|
||||
unqualified_percent_limit: int,
|
||||
n_base_limit: int,
|
||||
length_required: int,
|
||||
) -> List[str]:
|
||||
"""Build a fastp command line with a capped --thread value."""
|
||||
cmd = [
|
||||
str(fastp_bin),
|
||||
"-i", str(r1),
|
||||
"-I", str(r2),
|
||||
"-o", str(out_r1),
|
||||
"-O", str(out_r2),
|
||||
"--html", str(html_report),
|
||||
"--json", str(json_report),
|
||||
"--thread", str(max(1, min(MAX_TOOL_THREADS, int(cpu or 1)))),
|
||||
"--cut_window_size", str(cut_window_size),
|
||||
"--cut_mean_quality", str(cut_mean_quality),
|
||||
"--qualified_quality_phred", str(qualified_quality_phred),
|
||||
"--unqualified_percent_limit", str(unqualified_percent_limit),
|
||||
"--n_base_limit", str(n_base_limit),
|
||||
"--length_required", str(length_required),
|
||||
]
|
||||
if detect_adapter_for_pe:
|
||||
cmd.append("--detect_adapter_for_pe")
|
||||
if cut_front:
|
||||
cmd.append("--cut_front")
|
||||
if cut_tail:
|
||||
cmd.append("--cut_tail")
|
||||
return cmd
|
||||
|
||||
|
||||
def _count_fastq_reads(fastq_gz: Path) -> int:
|
||||
"""Count the number of reads (4 lines each) in a gzipped FASTQ using pigz."""
|
||||
proc = subprocess.run(
|
||||
["bash", "-c", f"pigz -dc {shlex.quote(str(fastq_gz))} | wc -l"],
|
||||
capture_output=True, text=True, check=True,
|
||||
)
|
||||
return int((proc.stdout or "0").strip() or 0) // 4
|
||||
|
||||
|
||||
def _split_fastq_gz(fastq_gz: Path, work_dir: Path, tag: str, n_chunks: int) -> List[Path]:
|
||||
"""Split a gzipped FASTQ into n_chunks contiguous, order-preserving gzipped chunks.
|
||||
|
||||
Uses pigz for parallel (de)compression. Read boundaries (4 lines per read) are
|
||||
preserved so the downstream merge stays in order and R1/R2 stay paired.
|
||||
"""
|
||||
work_dir.mkdir(parents=True, exist_ok=True)
|
||||
total_reads = _count_fastq_reads(fastq_gz)
|
||||
if total_reads <= 0:
|
||||
raise RuntimeError(f"fastq is empty or unreadable: {fastq_gz}")
|
||||
reads_per_chunk = -(-total_reads // n_chunks) # ceil division
|
||||
lines_per_chunk = max(4, reads_per_chunk * 4)
|
||||
raw_prefix = work_dir / f"{tag}.raw"
|
||||
subprocess.run(
|
||||
["bash", "-c",
|
||||
f"pigz -dc {shlex.quote(str(fastq_gz))} | split -l {lines_per_chunk} -d -a 4 - {shlex.quote(str(raw_prefix))}"],
|
||||
check=True,
|
||||
)
|
||||
raw_chunks = sorted(work_dir.glob(f"{tag}.raw[0-9]*"))
|
||||
chunk_paths: List[Path] = []
|
||||
for i, raw in enumerate(raw_chunks):
|
||||
out = work_dir / f"{tag}.part{i:04d}.fastq.gz"
|
||||
with open(out, "wb") as fout:
|
||||
subprocess.run(["pigz", "-p", "4", "-c", str(raw)], stdout=fout, check=True)
|
||||
raw.unlink(missing_ok=True)
|
||||
chunk_paths.append(out)
|
||||
if not chunk_paths:
|
||||
raise RuntimeError(f"failed to split fastq: {fastq_gz}")
|
||||
return chunk_paths
|
||||
|
||||
|
||||
def _merge_fastp_json(chunk_jsons: List[Path]) -> Dict[str, Any]:
|
||||
"""Merge per-chunk fastp JSON summaries into one combined summary.
|
||||
|
||||
Read/base counts are summed; rates, GC content and mean lengths are recomputed
|
||||
as (base|read)-weighted averages so merged stats match a single full run.
|
||||
"""
|
||||
summary: Dict[str, Any] = {"before_filtering": {}, "after_filtering": {}}
|
||||
for section in ("before_filtering", "after_filtering"):
|
||||
total_reads = 0
|
||||
total_bases = 0
|
||||
q20_bases = 0
|
||||
q30_bases = 0
|
||||
gc_sum = 0.0
|
||||
r1_len_sum = 0.0
|
||||
r2_len_sum = 0.0
|
||||
for jf in chunk_jsons:
|
||||
with open(jf, encoding="utf-8") as f:
|
||||
s = json.load(f).get("summary", {}).get(section, {})
|
||||
tr = int(s.get("total_reads", 0) or 0)
|
||||
tb = int(s.get("total_bases", 0) or 0)
|
||||
total_reads += tr
|
||||
total_bases += tb
|
||||
q20_bases += int(s.get("q20_bases", 0) or 0)
|
||||
q30_bases += int(s.get("q30_bases", 0) or 0)
|
||||
gc_sum += float(s.get("gc_content", 0) or 0) * tb
|
||||
r1_len_sum += float(s.get("read1_mean_length", 0) or 0) * tr
|
||||
r2_len_sum += float(s.get("read2_mean_length", 0) or 0) * tr
|
||||
summary[section] = {
|
||||
"total_reads": total_reads,
|
||||
"total_bases": total_bases,
|
||||
"q20_bases": q20_bases,
|
||||
"q30_bases": q30_bases,
|
||||
"q20_rate": (q20_bases / total_bases) if total_bases else 0.0,
|
||||
"q30_rate": (q30_bases / total_bases) if total_bases else 0.0,
|
||||
"gc_content": (gc_sum / total_bases) if total_bases else 0.0,
|
||||
"read1_mean_length": (r1_len_sum / total_reads) if total_reads else 0.0,
|
||||
"read2_mean_length": (r2_len_sum / total_reads) if total_reads else 0.0,
|
||||
}
|
||||
return {"summary": summary}
|
||||
|
||||
|
||||
def _write_fastp_summary_html(merged: Dict[str, Any], html_report: Path) -> None:
|
||||
"""Write a lightweight self-contained HTML report from the merged summary."""
|
||||
before = merged["summary"]["before_filtering"]
|
||||
after = merged["summary"]["after_filtering"]
|
||||
rows = [
|
||||
("Total reads", before.get("total_reads", 0), after.get("total_reads", 0)),
|
||||
("Q20 rate", f"{before.get('q20_rate', 0):.4f}", f"{after.get('q20_rate', 0):.4f}"),
|
||||
("Q30 rate", f"{before.get('q30_rate', 0):.4f}", f"{after.get('q30_rate', 0):.4f}"),
|
||||
("GC content", f"{before.get('gc_content', 0):.4f}", f"{after.get('gc_content', 0):.4f}"),
|
||||
]
|
||||
trs = "".join(f"<tr><td>{name}</td><td>{b}</td><td>{a}</td></tr>" for name, b, a in rows)
|
||||
html = (
|
||||
"<!DOCTYPE html><html><head><meta charset='utf-8'>"
|
||||
"<title>fastp QC Report (chunked)</title></head><body>"
|
||||
"<h1>fastp QC Report</h1><p>Generated from chunked parallel fastp runs.</p>"
|
||||
"<table border='1' cellpadding='6' cellspacing='0'>"
|
||||
"<tr><th>Metric</th><th>Before filtering</th><th>After filtering</th></tr>"
|
||||
f"{trs}</table></body></html>"
|
||||
)
|
||||
html_report.write_text(html, encoding="utf-8")
|
||||
|
||||
|
||||
def _run_fastp_chunked(
|
||||
fastp_bin: Path,
|
||||
r1: Path,
|
||||
r2: Path,
|
||||
out_r1: Path,
|
||||
out_r2: Path,
|
||||
html_report: Path,
|
||||
json_report: Path,
|
||||
log_file: Path,
|
||||
cpu: int,
|
||||
chunks: int,
|
||||
chunk_work_dir: Path,
|
||||
detect_adapter_for_pe: bool,
|
||||
cut_front: bool,
|
||||
cut_tail: bool,
|
||||
cut_window_size: int,
|
||||
cut_mean_quality: int,
|
||||
qualified_quality_phred: int,
|
||||
unqualified_percent_limit: int,
|
||||
n_base_limit: int,
|
||||
length_required: int,
|
||||
) -> Dict[str, Any]:
|
||||
"""Run fastp in chunked-parallel mode: split FASTQ, parallel fastp, merge outputs."""
|
||||
chunk_work_dir.mkdir(parents=True, exist_ok=True)
|
||||
print(f" [fastp] Splitting into {chunks} chunks (parallel mode)...", flush=True)
|
||||
start_ts = time.time()
|
||||
|
||||
r1_chunks = _split_fastq_gz(r1, chunk_work_dir, "r1", chunks)
|
||||
r2_chunks = _split_fastq_gz(r2, chunk_work_dir, "r2", chunks)
|
||||
n = max(len(r1_chunks), len(r2_chunks))
|
||||
if len(r1_chunks) != len(r2_chunks):
|
||||
print(
|
||||
f" [fastp] WARNING: R1/R2 chunk counts differ "
|
||||
f"({len(r1_chunks)} vs {len(r2_chunks)}); using {n}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
tasks = []
|
||||
for i in range(n):
|
||||
tasks.append({
|
||||
"idx": i,
|
||||
"r1": r1_chunks[i],
|
||||
"r2": r2_chunks[i],
|
||||
"o1": chunk_work_dir / f"out{i:04d}.r1.fastq.gz",
|
||||
"o2": chunk_work_dir / f"out{i:04d}.r2.fastq.gz",
|
||||
"html": chunk_work_dir / f"out{i:04d}.fastp.html",
|
||||
"json": chunk_work_dir / f"out{i:04d}.fastp.json",
|
||||
})
|
||||
|
||||
def _run_one(t: Dict[str, Any]):
|
||||
cmd = _build_fastp_cmd(
|
||||
fastp_bin, t["r1"], t["r2"], t["o1"], t["o2"], t["html"], t["json"], cpu,
|
||||
detect_adapter_for_pe, cut_front, cut_tail, cut_window_size, cut_mean_quality,
|
||||
qualified_quality_phred, unqualified_percent_limit, n_base_limit, length_required,
|
||||
)
|
||||
res = subprocess.run(cmd, capture_output=True, text=True, check=False)
|
||||
if res.returncode != 0:
|
||||
return t["idx"], None, res.stderr
|
||||
for p in (t["o1"], t["o2"], t["json"]):
|
||||
if not p.exists() or p.stat().st_size == 0:
|
||||
return t["idx"], None, f"empty output: {p}"
|
||||
return t["idx"], t, None
|
||||
|
||||
results: Dict[int, Dict[str, Any]] = {}
|
||||
errors: List[str] = []
|
||||
with ThreadPoolExecutor(max_workers=max(1, int(chunks))) as ex:
|
||||
futures = [ex.submit(_run_one, t) for t in tasks]
|
||||
for fut in as_completed(futures):
|
||||
idx, t, err = fut.result()
|
||||
if err:
|
||||
errors.append(f"chunk {idx}: {err}")
|
||||
else:
|
||||
results[idx] = t
|
||||
if errors:
|
||||
raise RuntimeError(f"fastp chunked failed: {'; '.join(errors[:3])}")
|
||||
|
||||
# Merge clean reads in chunk order (preserves read order/pairing)
|
||||
for out_path, key in ((out_r1, "o1"), (out_r2, "o2")):
|
||||
with open(out_path, "wb") as fout:
|
||||
for i in range(n):
|
||||
with open(results[i][key], "rb") as fin:
|
||||
shutil.copyfileobj(fin, fout)
|
||||
|
||||
# Merge JSON + generate HTML summary
|
||||
merged = _merge_fastp_json([results[i]["json"] for i in range(n)])
|
||||
json_report.write_text(json.dumps(merged), encoding="utf-8")
|
||||
_write_fastp_summary_html(merged, html_report)
|
||||
|
||||
elapsed = time.time() - start_ts
|
||||
log_file.write_text(
|
||||
f"fastp chunked parallel mode: {n} chunks x --thread {max(1, min(MAX_TOOL_THREADS, int(cpu or 1)))}\n"
|
||||
f"elapsed: {elapsed:.1f}s\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
shutil.rmtree(chunk_work_dir, ignore_errors=True)
|
||||
print(f" [fastp] Completed (chunked x{n}) in {elapsed:.1f}s", flush=True)
|
||||
return {"success": True, "time_sec": elapsed}
|
||||
|
||||
|
||||
def run_fastp(
|
||||
fastp_bin: Path,
|
||||
r1: Path,
|
||||
@@ -397,30 +655,22 @@ def run_fastp(
|
||||
unqualified_percent_limit: int = 30,
|
||||
n_base_limit: int = 5,
|
||||
length_required: int = 30,
|
||||
chunks: int = 1,
|
||||
chunk_work_dir: Optional[Path] = None,
|
||||
) -> Dict[str, Any]:
|
||||
cmd = [
|
||||
str(fastp_bin),
|
||||
"-i", str(r1),
|
||||
"-I", str(r2),
|
||||
"-o", str(out_r1),
|
||||
"-O", str(out_r2),
|
||||
"--html", str(html_report),
|
||||
"--json", str(json_report),
|
||||
"--thread", str(max(1, int(cpu or 1))),
|
||||
"--cut_window_size", str(cut_window_size),
|
||||
"--cut_mean_quality", str(cut_mean_quality),
|
||||
"--qualified_quality_phred", str(qualified_quality_phred),
|
||||
"--unqualified_percent_limit", str(unqualified_percent_limit),
|
||||
"--n_base_limit", str(n_base_limit),
|
||||
"--length_required", str(length_required),
|
||||
]
|
||||
if detect_adapter_for_pe:
|
||||
cmd.append("--detect_adapter_for_pe")
|
||||
if cut_front:
|
||||
cmd.append("--cut_front")
|
||||
if cut_tail:
|
||||
cmd.append("--cut_tail")
|
||||
if chunks and int(chunks) > 1 and chunk_work_dir is not None:
|
||||
return _run_fastp_chunked(
|
||||
fastp_bin, r1, r2, out_r1, out_r2, html_report, json_report, log_file,
|
||||
cpu, int(chunks), chunk_work_dir,
|
||||
detect_adapter_for_pe, cut_front, cut_tail, cut_window_size, cut_mean_quality,
|
||||
qualified_quality_phred, unqualified_percent_limit, n_base_limit, length_required,
|
||||
)
|
||||
|
||||
cmd = _build_fastp_cmd(
|
||||
fastp_bin, r1, r2, out_r1, out_r2, html_report, json_report, cpu,
|
||||
detect_adapter_for_pe, cut_front, cut_tail, cut_window_size, cut_mean_quality,
|
||||
qualified_quality_phred, unqualified_percent_limit, n_base_limit, length_required,
|
||||
)
|
||||
print(f" [fastp] Starting: {r1.name} + {r2.name}...", flush=True)
|
||||
start_ts = time.time()
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
|
||||
@@ -767,6 +1017,7 @@ def main():
|
||||
parser.add_argument("-t", "--cpu", type=int, default=4, help="Total CPU threads for this stage")
|
||||
parser.add_argument("--sample-parallel-jobs", type=int, default=1, help="Number of samples to process concurrently")
|
||||
parser.add_argument("--threads-per-sample", type=int, help="Threads used by each per-sample tool invocation")
|
||||
parser.add_argument("--chunks-per-sample", type=int, default=0, help="Split each sample into N chunks for parallel fastp. 0 = auto (cpu // threads-per-sample).")
|
||||
parser.add_argument("--resume-existing", action="store_true", help="Skip samples whose FastQC, clean reads, and cutadapt outputs already exist in the output directory")
|
||||
parser.add_argument("-p", "--prefix", default="", help="Output file prefix")
|
||||
parser.add_argument("--adapter-r1", default="AGATCGGAAGAGC", help="Adapter sequence for read1")
|
||||
@@ -786,6 +1037,12 @@ def main():
|
||||
args = parser.parse_args()
|
||||
thread_plan = resolve_thread_plan(args.cpu, args.sample_parallel_jobs, args.threads_per_sample)
|
||||
threads_per_sample = thread_plan["threads_per_sample"]
|
||||
|
||||
# Chunked parallelism: split each sample into N chunks and run fastp in
|
||||
# parallel (the real way to saturate many cores, since fastp is single-threaded).
|
||||
chunks_per_sample = int(args.chunks_per_sample or 0)
|
||||
if chunks_per_sample < 1:
|
||||
chunks_per_sample = max(1, int(thread_plan["cpu"]) // max(1, int(threads_per_sample)))
|
||||
|
||||
start_time = datetime.now()
|
||||
input_dir = Path(args.input)
|
||||
@@ -878,6 +1135,8 @@ def main():
|
||||
json_report=analysis_dir / f"{sample_name}_fastp.json",
|
||||
log_file=analysis_dir / f"{sample_name}_fastp.log",
|
||||
cpu=threads_per_sample,
|
||||
chunks=chunks_per_sample,
|
||||
chunk_work_dir=analysis_dir / f".{sample_name}.chunks",
|
||||
**fastp_params,
|
||||
)
|
||||
except RuntimeError as e:
|
||||
|
||||
Executable
+66
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate or open a formal RNA-seq evidence report."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).absolute().parents[2]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from gfa_core.rnaseq_report import NoSuccessfulRNASeqEvidence, generate_rnaseq_report
|
||||
|
||||
|
||||
def _csv(value: str) -> list[str] | None:
|
||||
values = [item.strip() for item in str(value or "").split(",") if item.strip()]
|
||||
return values or None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate a deterministic formal RNA-seq result report from successful manifests."
|
||||
)
|
||||
parser.add_argument("--workspace", required=True, help="RNA-seq job workspace containing final/")
|
||||
parser.add_argument("--prefix", required=True, help="RNA-seq output prefix")
|
||||
parser.add_argument("--job-id", default="")
|
||||
parser.add_argument("--rnaseq-scope-id", default="")
|
||||
parser.add_argument(
|
||||
"--expected-stages",
|
||||
default="",
|
||||
help="Comma-separated stages selected for the current RNA-seq batch",
|
||||
)
|
||||
parser.add_argument("--action", choices=("generate", "open"), default="generate")
|
||||
parser.add_argument("--no-model", action="store_true", help="Disable optional LLM language enhancement")
|
||||
parser.add_argument("--log-file", default="", help="Accepted for pipeline runner compatibility")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
result = generate_rnaseq_report(
|
||||
Path(args.workspace),
|
||||
args.prefix,
|
||||
rnaseq_scope_id=args.rnaseq_scope_id,
|
||||
expected_stages=_csv(args.expected_stages),
|
||||
use_model=not args.no_model,
|
||||
action=args.action,
|
||||
job_id=args.job_id,
|
||||
)
|
||||
except NoSuccessfulRNASeqEvidence as exc:
|
||||
print(str(exc), file=sys.stderr)
|
||||
return 2
|
||||
except Exception as exc:
|
||||
print(f"RNA-seq 结果报告生成失败:{type(exc).__name__}: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
reuse = "(复用最新报告)" if result.reused else ""
|
||||
print(
|
||||
f"RNA-seq 结果报告已就绪{reuse}: {result.report_path} "
|
||||
f"[coverage={result.coverage_status}, stages={','.join(result.included_stages)}]"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+87
-12
@@ -1,10 +1,10 @@
|
||||
#!/home/xzq/miniconda3/envs/test4py311/bin/python
|
||||
"""Scope-aware whole-job summary stage.
|
||||
"""Scope-aware whole-job summary and DeepSeek report stage.
|
||||
|
||||
Summary is an objective result index and quality-control layer. It reads all
|
||||
stage manifests in a job, groups them by scope_id, extracts key metrics and
|
||||
artifacts, and writes machine-readable and human-readable summaries. It does
|
||||
not perform deep biological interpretation.
|
||||
The deterministic layer first builds a traceable manifest/artifact index. By
|
||||
default that evidence is then passed to DeepSeek to produce the human-readable
|
||||
Chinese report. The evidence index remains a separate artifact so generated
|
||||
prose can always be audited against local results.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
@@ -16,6 +16,10 @@ from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Optional, Tuple
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
CONDA_BIN = "/home/xzq/miniconda3/envs/test4py311/bin"
|
||||
if CONDA_BIN not in os.environ.get("PATH", ""):
|
||||
os.environ["PATH"] = CONDA_BIN + ":" + os.environ.get("PATH", "")
|
||||
@@ -256,6 +260,10 @@ def main():
|
||||
parser.add_argument("--scope-id", help="optional explicit scope_id filter")
|
||||
parser.add_argument("--force", action="store_true", help="overwrite existing results")
|
||||
parser.add_argument("--log-file", help="log file path")
|
||||
parser.add_argument("--no-deepseek", action="store_true", help="write only the deterministic evidence index")
|
||||
parser.add_argument("--framework", help="optional Markdown framework for the DeepSeek report")
|
||||
parser.add_argument("--deepseek-model", default="", help="optional DeepSeek model override")
|
||||
parser.add_argument("--deepseek-max-tokens", type=int, default=16000)
|
||||
args = parser.parse_args()
|
||||
|
||||
manifest_path = Path(args.manifest).expanduser().resolve() if args.manifest else None
|
||||
@@ -284,11 +292,11 @@ def main():
|
||||
artifact_rows = [row for item in summaries for row in item["artifact_rows"]]
|
||||
checks = build_checks(summaries)
|
||||
|
||||
report = final_dir / f"{prefix}.summary_report.md"
|
||||
evidence_index = final_dir / f"{prefix}.summary_evidence_index.md"
|
||||
files_tsv = final_dir / f"{prefix}.summary.files.tsv"
|
||||
manifest_out = final_dir / f"{prefix}.summary.manifest.json"
|
||||
write_files_tsv(files_tsv, artifact_rows)
|
||||
write_report(report, prefix, summaries, checks, files_tsv)
|
||||
write_report(evidence_index, prefix, summaries, checks, files_tsv)
|
||||
|
||||
manifest_data = {
|
||||
"manifest_version": "1.0",
|
||||
@@ -306,22 +314,24 @@ def main():
|
||||
"summaries": summaries,
|
||||
"checks": checks,
|
||||
"artifacts": {
|
||||
"summary_report": {"path": f"final/{report.name}", "type": "markdown", "size": report.stat().st_size},
|
||||
"summary_evidence_index": {"path": f"final/{evidence_index.name}", "type": "markdown", "size": evidence_index.stat().st_size},
|
||||
"files_tsv": {"path": f"final/{files_tsv.name}", "type": "tsv", "size": files_tsv.stat().st_size},
|
||||
},
|
||||
}
|
||||
manifest_out.write_text(json.dumps(manifest_data, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
scoped_manifest = None
|
||||
scoped_data = None
|
||||
if scope_id:
|
||||
scoped_items = [x for x in summaries if x["scope_id"] == scope_id]
|
||||
scoped_rows = [row for x in scoped_items for row in x["artifact_rows"]]
|
||||
scoped_checks = [x for x in checks if x["scope_id"] in {scope_id, "all"}]
|
||||
token = sanitize_token(scope_id)
|
||||
scoped_report = final_dir / f"{prefix}.{token}.summary_report.md"
|
||||
scoped_evidence_index = final_dir / f"{prefix}.{token}.summary_evidence_index.md"
|
||||
scoped_files = final_dir / f"{prefix}.{token}.summary.files.tsv"
|
||||
scoped_manifest = final_dir / f"{prefix}.{token}.summary.manifest.json"
|
||||
write_files_tsv(scoped_files, scoped_rows)
|
||||
write_report(scoped_report, prefix, scoped_items, scoped_checks, scoped_files)
|
||||
write_report(scoped_evidence_index, prefix, scoped_items, scoped_checks, scoped_files)
|
||||
scoped_data = dict(manifest_data)
|
||||
scoped_data.update({
|
||||
"scope_id": scope_id,
|
||||
@@ -334,14 +344,79 @@ def main():
|
||||
"summaries": scoped_items,
|
||||
"checks": scoped_checks,
|
||||
"artifacts": {
|
||||
"summary_report": {"path": f"final/{scoped_report.name}", "type": "markdown", "size": scoped_report.stat().st_size},
|
||||
"summary_evidence_index": {"path": f"final/{scoped_evidence_index.name}", "type": "markdown", "size": scoped_evidence_index.stat().st_size},
|
||||
"files_tsv": {"path": f"final/{scoped_files.name}", "type": "tsv", "size": scoped_files.stat().st_size},
|
||||
},
|
||||
})
|
||||
scoped_manifest.write_text(json.dumps(scoped_data, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
if args.no_deepseek:
|
||||
fallback_report = final_dir / (
|
||||
f"{prefix}.{sanitize_token(scope_id)}.summary_report.md" if scope_id else f"{prefix}.summary_report.md"
|
||||
)
|
||||
source_index = scoped_evidence_index if scope_id else evidence_index
|
||||
fallback_report.write_text(source_index.read_text(encoding="utf-8"), encoding="utf-8")
|
||||
target_manifest = scoped_manifest if scope_id and scoped_manifest else manifest_out
|
||||
target_data = read_json(target_manifest) or manifest_data
|
||||
target_data.setdefault("artifacts", {})["summary_report"] = {
|
||||
"path": f"final/{fallback_report.name}",
|
||||
"type": "markdown",
|
||||
"size": fallback_report.stat().st_size,
|
||||
"generation": "deterministic_no_deepseek",
|
||||
}
|
||||
target_data["deepseek_generation"] = {"status": "skipped", "reason": "--no-deepseek"}
|
||||
target_manifest.write_text(json.dumps(target_data, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
else:
|
||||
try:
|
||||
from gfa_core.mas.agents.deepseek_summary import generate_deepseek_summary
|
||||
|
||||
result = generate_deepseek_summary(
|
||||
final_dir=final_dir,
|
||||
prefix=prefix,
|
||||
job_id=workspace.name,
|
||||
summaries=summaries,
|
||||
scope_id=scope_id,
|
||||
framework_path=Path(args.framework).expanduser() if args.framework else None,
|
||||
model=args.deepseek_model,
|
||||
max_tokens=max(4096, int(args.deepseek_max_tokens)),
|
||||
)
|
||||
effective_scope = str(result.get("scope_id") or "")
|
||||
target_manifest = scoped_manifest if scope_id and scoped_manifest else manifest_out
|
||||
target_data = read_json(target_manifest) or manifest_data
|
||||
report_path = Path(result["report_path"])
|
||||
evidence_path = Path(result["evidence_path"])
|
||||
metadata_path = Path(result["metadata_path"])
|
||||
target_data.setdefault("artifacts", {}).update({
|
||||
"summary_report": {
|
||||
"path": f"final/{report_path.name}", "type": "markdown", "size": report_path.stat().st_size,
|
||||
"generation": "deepseek_api",
|
||||
},
|
||||
"summary_evidence": {
|
||||
"path": f"final/{evidence_path.name}", "type": "json", "size": evidence_path.stat().st_size,
|
||||
},
|
||||
"deepseek_metadata": {
|
||||
"path": f"final/{metadata_path.name}", "type": "json", "size": metadata_path.stat().st_size,
|
||||
},
|
||||
})
|
||||
target_data["deepseek_generation"] = {
|
||||
"status": "success",
|
||||
"scope_id": effective_scope,
|
||||
"provider": result["metadata"].get("provider"),
|
||||
"model": result["metadata"].get("model"),
|
||||
"validation": result["metadata"].get("validation"),
|
||||
}
|
||||
target_manifest.write_text(json.dumps(target_data, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
except Exception as exc:
|
||||
target_manifest = scoped_manifest if scope_id and scoped_manifest else manifest_out
|
||||
target_data = read_json(target_manifest) or manifest_data
|
||||
target_data["status"] = "failed"
|
||||
target_data["deepseek_generation"] = {"status": "failed", "error": str(exc)[:2000]}
|
||||
target_manifest.write_text(json.dumps(target_data, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
print(f"Error: DeepSeek summary generation failed: {exc}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
print("Summary complete")
|
||||
print(f" - {report}")
|
||||
print(f" - {evidence_index}")
|
||||
print(f" - {files_tsv}")
|
||||
print(f" - {manifest_out}")
|
||||
|
||||
|
||||
+299
-118
@@ -1,6 +1,7 @@
|
||||
# web/pipeline_backend.py
|
||||
import csv
|
||||
import json
|
||||
import csv
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
@@ -48,6 +49,7 @@ from gfa_core.ppi_provenance import (
|
||||
record_ppi_preflight_decision,
|
||||
)
|
||||
from gfa_core.string_species_registry import (
|
||||
ARABIDOPSIS_TAXON_ID,
|
||||
StringSnapshotError,
|
||||
load_string_snapshot,
|
||||
resolve_ppi_string_species,
|
||||
@@ -1721,7 +1723,7 @@ def infer_string_species_from_ortholog_mapping(mapping_file: Path, fallback: str
|
||||
return fallback
|
||||
|
||||
|
||||
def infer_reference_species_from_ortholog_mapping(mapping_file: Path, fallback: str = "") -> str:
|
||||
def infer_reference_species_from_ortholog_mapping(mapping_file: Path, fallback: str = "") -> str:
|
||||
"""Infer the reference species label from the first data row of an ortholog mapping."""
|
||||
try:
|
||||
with mapping_file.open("r", encoding="utf-8", errors="ignore") as src:
|
||||
@@ -1735,10 +1737,34 @@ def infer_reference_species_from_ortholog_mapping(mapping_file: Path, fallback:
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
return fallback
|
||||
|
||||
|
||||
def _reference_species_from_manifest_data(data: dict[str, Any]) -> str:
|
||||
return fallback
|
||||
|
||||
|
||||
def ortholog_mapping_candidates(
|
||||
final_dir: Path,
|
||||
prefix: str,
|
||||
*,
|
||||
dependency_owner: str = "",
|
||||
scope_id: str = "",
|
||||
) -> list[Path]:
|
||||
from gfa_core.ortholog_materialize import ortholog_output_stem
|
||||
|
||||
output_stem = ortholog_output_stem(prefix, dependency_owner)
|
||||
candidates: list[Path] = []
|
||||
if scope_id:
|
||||
scoped_stem = f"{output_stem}.{sanitize_token(scope_id)}"
|
||||
candidates.extend([
|
||||
final_dir / f"{scoped_stem}.orthologs.txt",
|
||||
final_dir / f"{scoped_stem}.ortholog_pairs.txt",
|
||||
])
|
||||
candidates.extend([
|
||||
final_dir / f"{output_stem}.orthologs.txt",
|
||||
final_dir / f"{output_stem}.ortholog_pairs.txt",
|
||||
])
|
||||
return candidates
|
||||
|
||||
|
||||
def _reference_species_from_manifest_data(data: dict[str, Any]) -> str:
|
||||
for key in ("reference_species", "referenceSpecies"):
|
||||
value = data.get(key)
|
||||
if isinstance(value, list) and value:
|
||||
@@ -1905,11 +1931,10 @@ NCBI_REFERENCE_ANNOTATION_TYPES = {
|
||||
}
|
||||
|
||||
|
||||
def _reference_bundle_file_plan(requested_stages: Optional[list[str]], include_recommended: bool = False) -> list[str]:
|
||||
stages = _normalized_dependency_stages(requested_stages)
|
||||
files = ["pep"]
|
||||
if "kaks" in stages:
|
||||
files.append("cds")
|
||||
def _reference_bundle_file_plan(requested_stages: Optional[list[str]], include_recommended: bool = False) -> list[str]:
|
||||
# PPI owns reference preparation. Download PEP and CDS together so the
|
||||
# later Ka/Ks choice can reuse the exact same bundle without a second job.
|
||||
files = ["pep", "cds"]
|
||||
if include_recommended:
|
||||
for item in ("gff3", "genome"):
|
||||
if item not in files:
|
||||
@@ -2005,7 +2030,7 @@ def _download_url_to_file(
|
||||
raise RuntimeError("downloaded file is empty")
|
||||
temp_path.replace(output_path)
|
||||
return total
|
||||
except (urllib.error.URLError, TimeoutError, OSError, RuntimeError) as exc:
|
||||
except (http.client.IncompleteRead, urllib.error.URLError, TimeoutError, OSError, RuntimeError) as exc:
|
||||
last_error = exc
|
||||
if temp_path.exists():
|
||||
try:
|
||||
@@ -2119,6 +2144,8 @@ def _fasta_header_index(fasta_path: Path | str | None) -> dict[str, dict[str, st
|
||||
if not seq_id or seq_id in records:
|
||||
continue
|
||||
gene_match = re.search(r"\[gene=([^\]]+)\]", header)
|
||||
if not gene_match:
|
||||
gene_match = re.search(r"(?:^|\s)gene:([^\s]+)", header)
|
||||
ncbi_gene_match = re.search(r"\[db_xref=GeneID:(\d+)\]", header)
|
||||
protein_match = re.search(r"\[protein_id=([^\]]+)\]", header)
|
||||
records[seq_id] = {
|
||||
@@ -2220,7 +2247,7 @@ def download_reference_annotation_bundle(
|
||||
include_recommended=include_recommended,
|
||||
)
|
||||
assembly = {"accession": accession}
|
||||
emit({"phase": "select_assembly", "message": "正在解析推荐参考物种装配", "percentage": 5, "indeterminate": True})
|
||||
emit({"phase": "select_assembly", "message": "Resolving the recommended reference assembly", "percentage": 5, "indeterminate": True})
|
||||
if not accession:
|
||||
assembly = _select_ncbi_reference_assembly(taxon_id=taxon_id, species_name=species_name, timeout_sec=min(timeout_sec, 60))
|
||||
accession = assembly.get("accession", "")
|
||||
@@ -2231,24 +2258,24 @@ def download_reference_annotation_bundle(
|
||||
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
work_dir = ws / "reference_bundle_download" / f"{code}_{stamp}"
|
||||
zip_path = work_dir / f"{code}.ncbi_dataset.zip"
|
||||
emit({"phase": "download", "message": "正在下载 NCBI 参考注释包", "percentage": 10, "indeterminate": True})
|
||||
emit({"phase": "download", "message": "Downloading reference PEP and CDS from NCBI", "percentage": 10, "indeterminate": True})
|
||||
download_kwargs = {"timeout_sec": timeout_sec}
|
||||
if progress_callback:
|
||||
download_kwargs["progress_callback"] = lambda event: emit({"message": "正在下载 NCBI 参考注释包", **event})
|
||||
download_kwargs["progress_callback"] = lambda event: emit({"message": "Downloading reference PEP and CDS from NCBI", **event})
|
||||
bytes_downloaded = _download_url_to_file(download_url, zip_path, **download_kwargs)
|
||||
if not zipfile.is_zipfile(zip_path):
|
||||
raise RuntimeError(
|
||||
f"NCBI reference bundle download returned an invalid zip archive: {zip_path}"
|
||||
)
|
||||
species_dir = in_dir / code
|
||||
emit({"phase": "extract", "message": "正在解压参考 PEP/CDS", "percentage": 80, "indeterminate": True})
|
||||
emit({"phase": "extract", "message": "Extracting reference PEP and CDS", "percentage": 80, "indeterminate": True})
|
||||
extracted = _extract_ncbi_reference_bundle_zip(
|
||||
zip_path=zip_path,
|
||||
destination_dir=species_dir,
|
||||
species_code=code,
|
||||
requested_file_types=file_types,
|
||||
)
|
||||
emit({"phase": "validate_ids", "message": "正在校验 PEP/CDS ID 对齐", "percentage": 90, "indeterminate": True})
|
||||
emit({"phase": "validate_ids", "message": "Validating PEP/CDS identifier correspondence", "percentage": 90, "indeterminate": True})
|
||||
id_validation = _build_reference_bundle_id_map(
|
||||
species_code=code,
|
||||
pep_path=extracted.get("pep"),
|
||||
@@ -2288,7 +2315,7 @@ def download_reference_annotation_bundle(
|
||||
manifest["manifest_path"] = str(manifest_path)
|
||||
emit({
|
||||
"phase": "completed",
|
||||
"message": "参考物种注释包下载完成",
|
||||
"message": "PPI and Ka/Ks reference species data are ready",
|
||||
"percentage": 100,
|
||||
"indeterminate": False,
|
||||
"bytes_downloaded": bytes_downloaded,
|
||||
@@ -2311,6 +2338,7 @@ def _ppi_only_preflight_response(
|
||||
selected_string_species: str = "",
|
||||
reference_storage_key: str = "",
|
||||
reference_pep: str = "",
|
||||
reference_cds: str = "",
|
||||
mapping_file: Optional[Path] = None,
|
||||
mapping_reference_species: str = "",
|
||||
required_next_stage: str = "",
|
||||
@@ -2319,22 +2347,28 @@ def _ppi_only_preflight_response(
|
||||
) -> dict[str, Any]:
|
||||
candidates = list(species_resolution.get("candidates") or [])
|
||||
selection_mode = str(species_resolution.get("selection_mode") or "unresolved")
|
||||
required_files = ["pep", "cds"] if reference_storage_key else []
|
||||
bundle_files = {}
|
||||
if reference_pep:
|
||||
bundle_files["pep"] = reference_pep
|
||||
if reference_cds:
|
||||
bundle_files["cds"] = reference_cds
|
||||
reference_bundle_status = {
|
||||
"required_files": ["pep"],
|
||||
"required_files": required_files,
|
||||
"recommended_files": [],
|
||||
"files": {"pep": reference_pep} if reference_pep else {},
|
||||
"files": bundle_files,
|
||||
"missing_required": list(missing_required or []),
|
||||
"missing_recommended": [],
|
||||
"mode": (
|
||||
"complete"
|
||||
if reference_pep
|
||||
if not required_files or (reference_pep and reference_cds)
|
||||
else "not_selected"
|
||||
if status in {"confirmation_required", "unresolved", "conflict"}
|
||||
else "unavailable"
|
||||
if status == "resource_error"
|
||||
else "incomplete"
|
||||
),
|
||||
"kaks_ready": True,
|
||||
"kaks_ready": (not required_files) or bool(reference_pep and reference_cds),
|
||||
}
|
||||
diagnostics = {
|
||||
"issue_code": issue_code,
|
||||
@@ -2343,7 +2377,7 @@ def _ppi_only_preflight_response(
|
||||
"required_next_stage": required_next_stage,
|
||||
"recommendation": message,
|
||||
"dependency_owner_stages": ["ppi"],
|
||||
"required_bundle_files": ["pep"],
|
||||
"required_bundle_files": required_files,
|
||||
"reference_bundle_status": reference_bundle_status,
|
||||
"ortholog_dependency_role": "producer" if required_next_stage == "ortholog" else "not_requested",
|
||||
"reference_recommendations": candidates,
|
||||
@@ -2399,24 +2433,12 @@ def _effective_ppi_owner_scope_id(
|
||||
scope_data: dict[str, Any],
|
||||
reference_storage_key: str,
|
||||
) -> str:
|
||||
scope_mode = str(scope_data.get("scope_mode") or "target_plus_reference")
|
||||
comparative_species = [
|
||||
str(x) for x in scope_data.get("comparative_species", []) if str(x)
|
||||
]
|
||||
if scope_mode == "target_only":
|
||||
return str(scope_data.get("scope_id") or "") or build_scope_id(
|
||||
family,
|
||||
selected_targets,
|
||||
[str(x) for x in scope_data.get("reference_species", []) if str(x)],
|
||||
comparative_species,
|
||||
scope_mode,
|
||||
)
|
||||
return build_scope_id(
|
||||
family,
|
||||
selected_targets,
|
||||
[reference_storage_key],
|
||||
comparative_species,
|
||||
scope_mode,
|
||||
[],
|
||||
[],
|
||||
"target_only",
|
||||
)
|
||||
|
||||
|
||||
@@ -2616,6 +2638,16 @@ def _resolve_ppi_only_snapshot_preflight(
|
||||
),
|
||||
None,
|
||||
)
|
||||
explicit_model_fallback = bool(
|
||||
str(ppi_reference_selection_mode or "").strip().lower() == "model_fallback"
|
||||
and chosen_taxon_id == ARABIDOPSIS_TAXON_ID
|
||||
and chosen_taxon_id in snapshot.registry
|
||||
)
|
||||
if explicit_model_fallback:
|
||||
chosen = {
|
||||
"selected_string_taxon_id": chosen_taxon_id,
|
||||
"selected_string_name": snapshot.nodes[chosen_taxon_id].taxon_name,
|
||||
}
|
||||
if not (ppi_reference_user_confirmed and chosen):
|
||||
return _ppi_only_preflight_response(
|
||||
status="confirmation_required",
|
||||
@@ -2628,6 +2660,13 @@ def _resolve_ppi_only_snapshot_preflight(
|
||||
requested_ppi_species=requested_ppi_species,
|
||||
dependency_owner=dependency_owner,
|
||||
)
|
||||
if explicit_model_fallback:
|
||||
species_resolution = {
|
||||
**species_resolution,
|
||||
"selection_mode": "model_fallback",
|
||||
"candidate_source": "explicit_model_fallback",
|
||||
"issue_code": "",
|
||||
}
|
||||
|
||||
storage_key = str(ppi_reference_storage_key or "").strip() or f"string_taxon_{chosen_taxon_id}"
|
||||
selected_name = str(chosen.get("selected_string_name") or "")
|
||||
@@ -2638,16 +2677,16 @@ def _resolve_ppi_only_snapshot_preflight(
|
||||
ws, final_dir, in_dir, scope_data, storage_key, family, "pep", family_only=False
|
||||
)
|
||||
reference_pep = str(reference_pep_path) if reference_pep_path else ""
|
||||
mapping_candidates: list[Path] = []
|
||||
if scope_id:
|
||||
mapping_candidates.extend([
|
||||
final_dir / f"{prefix}.{sanitize_token(scope_id)}.orthologs.txt",
|
||||
final_dir / f"{prefix}.{sanitize_token(scope_id)}.ortholog_pairs.txt",
|
||||
])
|
||||
mapping_candidates.extend([
|
||||
final_dir / f"{prefix}.orthologs.txt",
|
||||
final_dir / f"{prefix}.ortholog_pairs.txt",
|
||||
])
|
||||
reference_cds_path = resolve_species_sequence_file(
|
||||
ws, final_dir, in_dir, scope_data, storage_key, family, "cds", family_only=False
|
||||
)
|
||||
reference_cds = str(reference_cds_path) if reference_cds_path else ""
|
||||
mapping_candidates = ortholog_mapping_candidates(
|
||||
final_dir,
|
||||
prefix,
|
||||
dependency_owner="ppi",
|
||||
scope_id=scope_id,
|
||||
)
|
||||
mapping_file = None if ignore_existing_ortholog else next(
|
||||
(path for path in mapping_candidates if path.exists() and path.stat().st_size > 0),
|
||||
None,
|
||||
@@ -2696,18 +2735,24 @@ def _resolve_ppi_only_snapshot_preflight(
|
||||
"selected_string_species": species_to_string_code(selected_name, ""),
|
||||
"reference_storage_key": storage_key,
|
||||
"reference_pep": reference_pep,
|
||||
"reference_cds": reference_cds,
|
||||
"mapping_file": mapping_file,
|
||||
"mapping_reference_species": mapping_reference_species,
|
||||
"dependency_owner": dependency_owner,
|
||||
}
|
||||
if not reference_pep:
|
||||
missing_reference_files = [
|
||||
file_type
|
||||
for file_type, file_path in (("pep", reference_pep), ("cds", reference_cds))
|
||||
if not file_path
|
||||
]
|
||||
if missing_reference_files:
|
||||
return _ppi_only_preflight_response(
|
||||
status="needs_reference_data",
|
||||
issue_code="ppi_reference_data_missing",
|
||||
message="The confirmed STRING reference requires a protein FASTA (PEP) before Ortholog.",
|
||||
message="The confirmed STRING reference requires matching PEP and CDS before Ortholog.",
|
||||
decision="needs_reference_bundle",
|
||||
required_next_stage="",
|
||||
missing_required=["pep"],
|
||||
missing_required=missing_reference_files,
|
||||
**common,
|
||||
)
|
||||
if mapping_file and not mapping_matches:
|
||||
@@ -2775,9 +2820,7 @@ def resolve_ppi_species_resolution(
|
||||
dependency_stages = _normalized_dependency_stages(requested_stages)
|
||||
needs_ppi = "ppi" in dependency_stages
|
||||
needs_kaks = "kaks" in dependency_stages
|
||||
required_bundle_files = ["pep"]
|
||||
if needs_kaks and "cds" not in required_bundle_files:
|
||||
required_bundle_files.append("cds")
|
||||
required_bundle_files = ["pep", "cds"]
|
||||
recommended_bundle_files = ["gff3", "genome"] if needs_kaks else []
|
||||
selected_targets = canonicalize_species_codes_for_job(in_dir, [str(x).strip() for x in (target_species or []) if str(x).strip()])
|
||||
references = canonicalize_species_codes_for_job(in_dir, [str(x).strip() for x in (reference_species or []) if str(x).strip()])
|
||||
@@ -2818,6 +2861,22 @@ def resolve_ppi_species_resolution(
|
||||
)
|
||||
|
||||
if needs_ppi and not needs_kaks:
|
||||
scope_id = _effective_ppi_owner_scope_id(
|
||||
family=family,
|
||||
selected_targets=selected_targets,
|
||||
scope_data=scope_data,
|
||||
reference_storage_key=str(ppi_reference_storage_key or ""),
|
||||
)
|
||||
scope_data = {
|
||||
**scope_data,
|
||||
"family": family,
|
||||
"scope_id": scope_id,
|
||||
"scope_mode": "target_only",
|
||||
"target_species": list(selected_targets),
|
||||
"reference_species": [],
|
||||
"comparative_species": [],
|
||||
"selected_species": list(selected_targets),
|
||||
}
|
||||
result = _resolve_ppi_only_snapshot_preflight(
|
||||
job=job,
|
||||
prefix=prefix,
|
||||
@@ -2886,17 +2945,15 @@ def resolve_ppi_species_resolution(
|
||||
and target_string_code
|
||||
and selected_string_species == target_string_code
|
||||
)
|
||||
mapping_candidates: list[Path] = []
|
||||
if not use_target_string_directly or needs_kaks:
|
||||
if scope_id:
|
||||
mapping_candidates.extend([
|
||||
final_dir / f"{prefix}.{sanitize_token(scope_id)}.orthologs.txt",
|
||||
final_dir / f"{prefix}.{sanitize_token(scope_id)}.ortholog_pairs.txt",
|
||||
])
|
||||
mapping_candidates.extend([
|
||||
final_dir / f"{prefix}.orthologs.txt",
|
||||
final_dir / f"{prefix}.ortholog_pairs.txt",
|
||||
])
|
||||
mapping_candidates: list[Path] = []
|
||||
if not use_target_string_directly or needs_kaks:
|
||||
mapping_owner = str(dependency_owner or "").strip().lower()
|
||||
mapping_candidates = ortholog_mapping_candidates(
|
||||
final_dir,
|
||||
prefix,
|
||||
dependency_owner=mapping_owner if mapping_owner in {"ppi", "kaks"} else "",
|
||||
scope_id=scope_id,
|
||||
)
|
||||
mapping_file = None if ignore_existing_ortholog else next((p for p in mapping_candidates if p.exists() and p.stat().st_size > 0), None)
|
||||
|
||||
reference_data_source = ""
|
||||
@@ -2917,7 +2974,7 @@ def resolve_ppi_species_resolution(
|
||||
reference_data_source = str(ref_pep) if ref_pep else ""
|
||||
if ref_pep:
|
||||
reference_bundle_files["pep"] = str(ref_pep)
|
||||
if needs_kaks:
|
||||
if "cds" in required_bundle_files:
|
||||
ref_cds = resolve_species_sequence_file(
|
||||
ws,
|
||||
final_dir,
|
||||
@@ -3355,6 +3412,7 @@ RNA_SEQ_STAGES = {
|
||||
"rnaseq_wgcna",
|
||||
"rnaseq_candidate_family",
|
||||
"rnaseq_family_score",
|
||||
"rnaseq_report",
|
||||
}
|
||||
|
||||
|
||||
@@ -6025,6 +6083,7 @@ def run_stage(
|
||||
ppi_reference_selection_mode: str = "",
|
||||
ppi_reference_user_confirmed: bool = False,
|
||||
dependency_owner: str = "",
|
||||
ortholog_mapping_owner: str = "",
|
||||
ppi_score: int = 200,
|
||||
# Genome Prepare 参数
|
||||
genome_prepare_type: str = "auto",
|
||||
@@ -8383,7 +8442,32 @@ def run_stage(
|
||||
expected_stages = [str(value).strip() for value in expected_stages if str(value).strip()]
|
||||
report_action = str(kwargs.get("report_action") or kwargs.get("reportAction") or "generate").strip()
|
||||
if report_action == "generate" and not expected_stages:
|
||||
return "❌ 基因家族结果报告生成需要非空 expected_stages", manifest
|
||||
report_stage_order = (
|
||||
"identify", "phylogeny", "reference_classification", "motif",
|
||||
"domain", "structure", "combo", "promoter", "tfbs",
|
||||
"synteny", "karyotype", "interspecies_synteny", "pep",
|
||||
)
|
||||
recovered_stages = set()
|
||||
for candidate in final_dir.glob("*.manifest.json"):
|
||||
try:
|
||||
payload = json.loads(candidate.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
continue
|
||||
candidate_stage = str(payload.get("stage") or "").strip()
|
||||
candidate_prefix = str(
|
||||
payload.get("prefix")
|
||||
or (payload.get("parameters") or {}).get("prefix")
|
||||
or ""
|
||||
).strip()
|
||||
if (
|
||||
candidate_stage in report_stage_order
|
||||
and str(payload.get("status") or "") == "success"
|
||||
and candidate_prefix == prefix
|
||||
):
|
||||
recovered_stages.add(candidate_stage)
|
||||
expected_stages = [stage_id for stage_id in report_stage_order if stage_id in recovered_stages]
|
||||
if report_action == "generate" and not expected_stages:
|
||||
return "❌ 基因家族结果报告生成需要非空 expected_stages,且当前前缀没有可恢复的成功 manifest", manifest
|
||||
stage_statuses = kwargs.get("stage_statuses") or kwargs.get("stageStatuses") or {}
|
||||
if isinstance(stage_statuses, str):
|
||||
try:
|
||||
@@ -8392,6 +8476,8 @@ def run_stage(
|
||||
return "❌ 基因家族结果报告的 stage_statuses 不是合法 JSON", manifest
|
||||
if not isinstance(stage_statuses, dict):
|
||||
return "❌ 基因家族结果报告的 stage_statuses 必须是对象", manifest
|
||||
if not stage_statuses:
|
||||
stage_statuses = {stage_id: "success" for stage_id in expected_stages}
|
||||
report_use_model_raw = kwargs.get("report_use_model", kwargs.get("reportUseModel", True))
|
||||
rnaseq_scope_id = str(kwargs.get("rnaseq_scope_id") or kwargs.get("rnaseqScopeId") or "").strip()
|
||||
cmd = [
|
||||
@@ -11407,6 +11493,42 @@ def run_stage(
|
||||
return "✅ RNA-seq WGCNA completed.", wgcna_manifest
|
||||
return f"❌ RNA-seq WGCNA failed (exit code: {code}). Log: {log_file}", wgcna_manifest if wgcna_manifest.exists() else manifest
|
||||
|
||||
if stage == "rnaseq_report":
|
||||
report_action = str(kwargs.get("rnaseq_report_action") or kwargs.get("report_action") or "generate").strip().lower()
|
||||
if report_action not in {"generate", "open"}:
|
||||
report_action = "generate"
|
||||
expected_raw = kwargs.get("rnaseq_expected_stages") or kwargs.get("expected_stages") or ""
|
||||
if isinstance(expected_raw, (list, tuple, set)):
|
||||
expected_arg = ",".join(str(item).strip() for item in expected_raw if str(item).strip())
|
||||
else:
|
||||
expected_arg = str(expected_raw or "").strip()
|
||||
report_scope_id = str(kwargs.get("rnaseq_scope_id") or kwargs.get("rnaseqScopeId") or "").strip()
|
||||
report_use_model = kwargs.get("rnaseq_report_use_model", kwargs.get("report_use_model", True))
|
||||
report_cmd = [
|
||||
sys.executable,
|
||||
"apps/cli/run_rnaseq_report.py",
|
||||
"--workspace", str(ws),
|
||||
"--prefix", prefix,
|
||||
"--job-id", job.job_id,
|
||||
"--action", report_action,
|
||||
"--log-file", str(log_file),
|
||||
]
|
||||
if report_scope_id:
|
||||
report_cmd.extend(["--rnaseq-scope-id", report_scope_id])
|
||||
if expected_arg:
|
||||
report_cmd.extend(["--expected-stages", expected_arg])
|
||||
if not _truthy(report_use_model):
|
||||
report_cmd.append("--no-model")
|
||||
print(f"[CMD] {' '.join(report_cmd)}")
|
||||
code = run_cmd_logged(report_cmd, log_file, cwd=Path("."))
|
||||
report_manifest = final_dir / f"{prefix}.rnaseq_report.manifest.json"
|
||||
if code == 0 and report_manifest.exists():
|
||||
action_label = "打开" if report_action == "open" else "生成"
|
||||
return f"✅ RNA-seq 结果报告已{action_label}。日志:{log_file}", report_manifest
|
||||
if code == 2:
|
||||
return "⚠️ 暂无可生成的 RNA-seq 结果:当前 prefix/scope 下没有成功阶段。", None
|
||||
return f"❌ RNA-seq 结果报告生成失败(exit code: {code})。日志:{log_file}", report_manifest if report_manifest.exists() else None
|
||||
|
||||
if stage == "rnaseq_candidate_family":
|
||||
_SPECIES_RESOURCE_ALIASES = {
|
||||
"os": ["Os", "Oryza", "Oryza_sativa", "rice"],
|
||||
@@ -12450,20 +12572,42 @@ def run_stage(
|
||||
if not references:
|
||||
return "❌ Ortholog scope 中没有 reference_species,无法确定 subject/reference 物种。", manifest
|
||||
reference_species = references[0]
|
||||
subject_fasta = first_existing_scope_file(ortholog_scope_data, reference_species, ["family_pep", "pep"])
|
||||
if not subject_fasta:
|
||||
subject_fasta = resolve_species_family_pep(ws, final_dir, in_dir, reference_species, family)
|
||||
if not subject_fasta:
|
||||
subject_fasta = resolve_species_sequence_file(
|
||||
ws,
|
||||
final_dir,
|
||||
if ppi_owned_producer:
|
||||
subject_fasta = resolve_species_sequence_file(
|
||||
ws,
|
||||
final_dir,
|
||||
in_dir,
|
||||
ortholog_scope_data,
|
||||
reference_species,
|
||||
family,
|
||||
"pep",
|
||||
family_only=False,
|
||||
)
|
||||
"pep",
|
||||
family_only=False,
|
||||
)
|
||||
else:
|
||||
subject_fasta = first_existing_scope_file(
|
||||
ortholog_scope_data,
|
||||
reference_species,
|
||||
["family_pep", "pep"],
|
||||
)
|
||||
if not subject_fasta:
|
||||
subject_fasta = resolve_species_family_pep(
|
||||
ws,
|
||||
final_dir,
|
||||
in_dir,
|
||||
reference_species,
|
||||
family,
|
||||
)
|
||||
if not subject_fasta:
|
||||
subject_fasta = resolve_species_sequence_file(
|
||||
ws,
|
||||
final_dir,
|
||||
in_dir,
|
||||
ortholog_scope_data,
|
||||
reference_species,
|
||||
family,
|
||||
"pep",
|
||||
family_only=False,
|
||||
)
|
||||
if not subject_fasta:
|
||||
return (
|
||||
f"❌ Ortholog stage requires reference protein FASTA for {reference_species}. "
|
||||
@@ -12510,11 +12654,14 @@ def run_stage(
|
||||
ortholog_pairs_has_data,
|
||||
write_ortholog_run_sidecar,
|
||||
)
|
||||
ortholog_owner = str(dependency_owner or "").strip().lower()
|
||||
if ortholog_owner not in {"", "ppi", "kaks"}:
|
||||
return f"❌ Unsupported Ortholog dependency owner: {dependency_owner}", manifest
|
||||
|
||||
scope_fields: dict[str, Any] = {}
|
||||
if ortholog_scope_data:
|
||||
source_scope_mode = str(ortholog_scope_data.get("scope_mode") or "")
|
||||
preserve_target_only_scope = ppi_owned_producer and source_scope_mode == "target_only"
|
||||
preserve_target_only_scope = ppi_owned_producer
|
||||
effective_scope_id = str(ortholog_scope_data.get("scope_id") or "")
|
||||
if ppi_owned_producer:
|
||||
effective_scope_id = _effective_ppi_owner_scope_id(
|
||||
@@ -12524,19 +12671,19 @@ def run_stage(
|
||||
reference_storage_key=reference_species,
|
||||
)
|
||||
scope_fields = {
|
||||
"scope_file": str(ortholog_scope_path),
|
||||
"scope_file": str(ortholog_scope_path) if source_scope_mode == "target_only" else "",
|
||||
"scope_id": effective_scope_id,
|
||||
"scope_mode": ortholog_scope_data.get("scope_mode", ""),
|
||||
"scope_mode": "target_only" if ppi_owned_producer else ortholog_scope_data.get("scope_mode", ""),
|
||||
"target_species": [record[0] for record in run_specs],
|
||||
"reference_species": (
|
||||
ortholog_scope_data.get("reference_species", [])
|
||||
if preserve_target_only_scope
|
||||
[]
|
||||
if ppi_owned_producer
|
||||
else [reference_species] if reference_species else []
|
||||
),
|
||||
"comparative_species": ortholog_scope_data.get("comparative_species", []),
|
||||
"comparative_species": [] if ppi_owned_producer else ortholog_scope_data.get("comparative_species", []),
|
||||
"selected_species": (
|
||||
ortholog_scope_data.get("selected_species", [record[0] for record in run_specs])
|
||||
if preserve_target_only_scope
|
||||
[record[0] for record in run_specs]
|
||||
if ppi_owned_producer
|
||||
else [record[0] for record in run_specs] + ([reference_species] if reference_species else [])
|
||||
),
|
||||
"family": family,
|
||||
@@ -12615,9 +12762,10 @@ def run_stage(
|
||||
run_identities.append(completed_identity)
|
||||
|
||||
try:
|
||||
pair_count, ortholog_manifest = materialize_ortholog_final_outputs(
|
||||
final_dir=final_dir,
|
||||
prefix=prefix,
|
||||
pair_count, ortholog_manifest = materialize_ortholog_final_outputs(
|
||||
final_dir=final_dir,
|
||||
prefix=prefix,
|
||||
dependency_owner=ortholog_owner,
|
||||
run_records=run_records,
|
||||
job_id=job.job_id,
|
||||
workspace=ws,
|
||||
@@ -12791,18 +12939,15 @@ def run_stage(
|
||||
and target_string_taxon_id
|
||||
and selected_execution_taxon_id == target_string_taxon_id
|
||||
)
|
||||
mapping_candidates = []
|
||||
if not use_target_string_directly:
|
||||
if scope_id:
|
||||
mapping_candidates.extend([
|
||||
final_dir / f"{prefix}.{sanitize_token(scope_id)}.orthologs.txt",
|
||||
final_dir / f"{prefix}.{sanitize_token(scope_id)}.ortholog_pairs.txt",
|
||||
])
|
||||
mapping_candidates.extend([
|
||||
final_dir / f"{prefix}.orthologs.txt",
|
||||
final_dir / f"{prefix}.ortholog_pairs.txt",
|
||||
])
|
||||
mapping_file = next((p for p in mapping_candidates if p.exists() and p.stat().st_size > 0), None)
|
||||
mapping_candidates = []
|
||||
if not use_target_string_directly:
|
||||
mapping_candidates = ortholog_mapping_candidates(
|
||||
final_dir,
|
||||
prefix,
|
||||
dependency_owner="ppi" if ppi_owner_intent else "",
|
||||
scope_id=scope_id,
|
||||
)
|
||||
mapping_file = next((p for p in mapping_candidates if p.exists() and p.stat().st_size > 0), None)
|
||||
if mapping_file:
|
||||
gene_id_file = ws / "ppi" / "analysis" / f"{prefix}.{sanitize_token(scope_id) if scope_id else 'scope'}.ppi_source_ids.txt"
|
||||
mapped_source_count = build_ppi_gene_ids_from_ortholog_mapping(mapping_file, gene_id_file, ppi_selected_targets)
|
||||
@@ -12987,7 +13132,7 @@ def run_stage(
|
||||
if mapping_file:
|
||||
effective_normalization_label = (
|
||||
ppi_reference_storage_key
|
||||
if ppi_owner_auto_fallback and ppi_reference_storage_key
|
||||
if ppi_reference_storage_key
|
||||
else ppi_reference_species or ppi_species
|
||||
)
|
||||
elif len(ppi_selected_targets) == 1:
|
||||
@@ -13054,6 +13199,41 @@ def run_stage(
|
||||
if not ppi_storage_key:
|
||||
ppi_storage_key = str(ppi_species)
|
||||
id_map_file = Path(job.input_dir) / ppi_storage_key / f"{ppi_storage_key}.pep_cds_idmap.tsv"
|
||||
if mapping_file and not id_map_file.is_file():
|
||||
reference_family = str(
|
||||
(ppi_scope_data or {}).get("family")
|
||||
or (prefix.split(".", 1)[1] if "." in prefix else prefix)
|
||||
)
|
||||
reference_pep = resolve_species_sequence_file(
|
||||
ws,
|
||||
final_dir,
|
||||
in_dir,
|
||||
ppi_scope_data or {},
|
||||
ppi_storage_key,
|
||||
reference_family,
|
||||
"pep",
|
||||
family_only=False,
|
||||
)
|
||||
reference_cds = resolve_species_sequence_file(
|
||||
ws,
|
||||
final_dir,
|
||||
in_dir,
|
||||
ppi_scope_data or {},
|
||||
ppi_storage_key,
|
||||
reference_family,
|
||||
"cds",
|
||||
family_only=False,
|
||||
)
|
||||
if reference_pep and reference_cds:
|
||||
try:
|
||||
_build_reference_bundle_id_map(
|
||||
species_code=ppi_storage_key,
|
||||
pep_path=reference_pep,
|
||||
cds_path=reference_cds,
|
||||
output_path=id_map_file,
|
||||
)
|
||||
except (OSError, UnicodeError, csv.Error) as exc:
|
||||
print(f"[PPI] Could not build species-matched ID map: {exc}")
|
||||
if mapping_file and not id_map_file.is_file():
|
||||
return fail_ppi_execution(
|
||||
f"❌ PPI projected analysis requires species-matched ID map: {id_map_file}"
|
||||
@@ -13335,22 +13515,23 @@ def run_stage(
|
||||
f"Target.{target_species_name}.{family}",
|
||||
])
|
||||
candidate_prefixes = list(dict.fromkeys(str(x) for x in candidate_prefixes if str(x)))
|
||||
mapping_candidates = []
|
||||
if scope_id:
|
||||
for candidate_prefix in candidate_prefixes:
|
||||
mapping_candidates.extend([
|
||||
final_dir / f"{candidate_prefix}.{sanitize_token(scope_id)}.orthologs.txt",
|
||||
final_dir / f"{candidate_prefix}.{sanitize_token(scope_id)}.ortholog_pairs.txt",
|
||||
])
|
||||
for candidate_prefix in candidate_prefixes:
|
||||
mapping_candidates.extend([
|
||||
final_dir / f"{candidate_prefix}.orthologs.txt",
|
||||
final_dir / f"{candidate_prefix}.ortholog_pairs.txt",
|
||||
])
|
||||
mapping_candidates.append(in_dir / "ortholog_pairs.txt")
|
||||
scoped_pairs_source = next((p for p in mapping_candidates if p.exists() and p.stat().st_size > 0), None)
|
||||
if not scoped_pairs_source:
|
||||
return "❌ KaKs scope 模式需要 Ortholog 映射文件。请先运行 Ortholog 阶段。", manifest
|
||||
selected_mapping_owner = str(ortholog_mapping_owner or "").strip().lower()
|
||||
if selected_mapping_owner not in {"", "ppi", "kaks"}:
|
||||
return f"❌ KaKs Ortholog 映射所有者无效: {ortholog_mapping_owner}", manifest
|
||||
mapping_candidates = []
|
||||
for candidate_prefix in candidate_prefixes:
|
||||
mapping_candidates.extend(ortholog_mapping_candidates(
|
||||
final_dir,
|
||||
candidate_prefix,
|
||||
dependency_owner=selected_mapping_owner,
|
||||
scope_id=scope_id,
|
||||
))
|
||||
if not selected_mapping_owner:
|
||||
mapping_candidates.append(in_dir / "ortholog_pairs.txt")
|
||||
scoped_pairs_source = next((p for p in mapping_candidates if p.exists() and p.stat().st_size > 0), None)
|
||||
if not scoped_pairs_source:
|
||||
owner_hint = f"({selected_mapping_owner} 专属)" if selected_mapping_owner else ""
|
||||
return f"❌ KaKs scope 模式需要{owner_hint} Ortholog 映射文件。", manifest
|
||||
|
||||
ref_cds = resolve_species_sequence_file(ws, final_dir, in_dir, kaks_scope_data, ref_species, family, "cds")
|
||||
ref_pep = resolve_species_sequence_file(ws, final_dir, in_dir, kaks_scope_data, ref_species, family, "pep", family_only=False)
|
||||
|
||||
+929
-325
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+1181
-377
File diff suppressed because it is too large
Load Diff
@@ -124,6 +124,17 @@
|
||||
};
|
||||
}
|
||||
|
||||
async function extractEnvDeclaration(files) {
|
||||
for (const file of files) {
|
||||
const path = normalizedRelativePath(file).toLowerCase();
|
||||
if (!path.endsWith('.sh')) continue;
|
||||
const content = await file.text();
|
||||
const m = content.match(/^\s*#\s*lightweight-env\s*:\s*([A-Za-z0-9_.-]+)/m);
|
||||
if (m) return m[1];
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function encodeBase64(file) {
|
||||
const buffer = await file.arrayBuffer();
|
||||
const bytes = new Uint8Array(buffer);
|
||||
@@ -146,11 +157,12 @@
|
||||
payload.files.push(await encodeBase64(file));
|
||||
payload.relative_paths.push(normalizedRelativePath(file));
|
||||
}
|
||||
const response = await fetch('/api/job/import-folder', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(payload),
|
||||
});
|
||||
const response = await hooks.uploadExtensionWithProgress(
|
||||
'/api/job/import-folder',
|
||||
JSON.stringify(payload),
|
||||
'正在镜像导入 Job',
|
||||
{ 'Content-Type': 'application/json' },
|
||||
);
|
||||
const data = await response.json();
|
||||
if (!response.ok || !data.success) {
|
||||
throw new Error(data.detail || data.error || data.message || 'Job 导入失败');
|
||||
@@ -160,6 +172,8 @@
|
||||
|
||||
async function resolveRuntimeEnv(files) {
|
||||
const dependencies = await extractDependencies(files);
|
||||
const envDeclaration = await extractEnvDeclaration(files);
|
||||
if (envDeclaration) dependencies.env_declaration = envDeclaration;
|
||||
console.log('[lightWeightExtensionTest] 提取的依赖:', JSON.stringify(dependencies), '| 扫描文件数:', files.length);
|
||||
try {
|
||||
const response = await fetch('/api/extension-stages/envs/resolve', {
|
||||
@@ -268,12 +282,23 @@
|
||||
function mountApi() {
|
||||
window.LightWeightExtensionTest = {
|
||||
async handleDrop(hooks, event) {
|
||||
const progress = hooks.lightweightUploadProgress;
|
||||
try {
|
||||
if (progress) {
|
||||
progress.active = true;
|
||||
progress.percent = 0;
|
||||
progress.loaded = 0;
|
||||
progress.total = 0;
|
||||
progress.speed = 0;
|
||||
progress.label = '本地文件读取中';
|
||||
}
|
||||
hooks.lightweightExtensionTestStatus.value = '本地文件读取中...';
|
||||
const files = await readDroppedFiles(event?.dataTransfer);
|
||||
await processFolderFiles(hooks, files);
|
||||
} catch (error) {
|
||||
const message = `失败:${error.message || '目录处理失败'}`;
|
||||
hooks.lightweightExtensionTestStatus.value = message;
|
||||
if (progress) progress.active = false;
|
||||
notifyError(message);
|
||||
}
|
||||
},
|
||||
|
||||
+472
-52
@@ -45,6 +45,7 @@ PROJECT_ROOT = Path(__file__).parent.parent.parent.parent
|
||||
DEFAULT_RUNTIME_ENV = os.environ.get("BBAV5_DEFAULT_RUNTIME_ENV", "test4py311")
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
from gfa_core.project_paths import get_conda_env_bin, get_conda_env_rscript, get_configured_path, get_wgs_bcftools, get_wgs_java, get_wgs_samtools
|
||||
from gfa_core.mas import confirmation_snapshot as _confirmation_snapshot
|
||||
|
||||
ASAW_BIN = get_conda_env_bin("ASAW")
|
||||
ASAW_RSCRIPT = get_conda_env_rscript("ASAW")
|
||||
@@ -705,9 +706,10 @@ def _confirmed_config_species_scope(job_id: str) -> Dict[str, list[str]]:
|
||||
if not job_id:
|
||||
return {}
|
||||
try:
|
||||
from gfa_core.mas.confirmation_snapshot import build_confirmation_snapshot
|
||||
|
||||
confirmation = build_confirmation_snapshot(PROJECT_ROOT, job_id)
|
||||
confirmation = _confirmation_snapshot.build_confirmation_snapshot(
|
||||
PROJECT_ROOT,
|
||||
job_id,
|
||||
)
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
if not confirmation.get("config_confirmed"):
|
||||
@@ -880,7 +882,8 @@ def _species_role_conflict_message(conflicts: list[Dict[str, str]]) -> str:
|
||||
return (
|
||||
"检测到本轮请求与项目物种角色不一致:"
|
||||
+ detail_text
|
||||
+ "。系统已阻止执行,请按 config.xlsx 中已确认的角色重新描述分析范围。"
|
||||
+ "。系统已阻止执行,请按 config.xlsx 中已确认的角色重新描述分析范围;"
|
||||
"如需修改项目物种角色,请发送“打开 config.xlsx”。"
|
||||
)
|
||||
|
||||
|
||||
@@ -1090,6 +1093,8 @@ def _hydrate_goal_terms_from_current_job(
|
||||
return hydrated
|
||||
|
||||
config_scope = _confirmed_config_species_scope(job_id)
|
||||
if not config_scope and (INPUT_ROOT / job_id / "config.xlsx").is_file():
|
||||
config_scope = _configured_species_scope(job_id)
|
||||
for config_key, term_key in (
|
||||
("targetSpecies", "target_species"),
|
||||
("referenceSpecies", "reference_species"),
|
||||
@@ -1483,18 +1488,62 @@ def _canonicalize_species_codes_for_job_strict(job_id: str, species_codes: list[
|
||||
|
||||
|
||||
def _reference_bundle_species_available(job_id: str, species_code: str, require_cds: bool = False) -> tuple[bool, str]:
|
||||
"""Return whether a downloaded job-local reference bundle can satisfy PPI/KaKs."""
|
||||
"""Return whether job-local or config-resolved reference data can satisfy PPI/KaKs."""
|
||||
code = str(species_code or "").strip()
|
||||
if not job_id or not code:
|
||||
return False, "missing job_id/species_code"
|
||||
species_dir = INPUT_ROOT / job_id / code
|
||||
input_dir = INPUT_ROOT / job_id
|
||||
species_dir = input_dir / code
|
||||
pep_path = species_dir / f"{code}.pep.fasta"
|
||||
cds_path = species_dir / f"{code}.cds.fasta"
|
||||
if not pep_path.exists() or pep_path.stat().st_size <= 0:
|
||||
local_pep_ready = pep_path.exists() and pep_path.stat().st_size > 0
|
||||
local_cds_ready = cds_path.exists() and cds_path.stat().st_size > 0
|
||||
if local_pep_ready and (not require_cds or local_cds_ready):
|
||||
return True, str(species_dir)
|
||||
|
||||
try:
|
||||
scope = build_gene_family_scope(
|
||||
input_dir=input_dir,
|
||||
workspace=WORKSPACE_ROOT / job_id,
|
||||
prefix="",
|
||||
target_species=[],
|
||||
reference_species=[code],
|
||||
comparative_species=[],
|
||||
scope_mode="target_plus_reference",
|
||||
)
|
||||
species_files = scope.files.get(code, {})
|
||||
|
||||
def has_sequence(file_type: str) -> bool:
|
||||
return any(
|
||||
record.get("path")
|
||||
and Path(str(record["path"])).exists()
|
||||
and Path(str(record["path"])).stat().st_size > 0
|
||||
for record in species_files.get(file_type, [])
|
||||
)
|
||||
|
||||
if has_sequence("pep") and (not require_cds or has_sequence("cds")):
|
||||
return True, str(input_dir)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if not local_pep_ready:
|
||||
return False, f"missing PEP: {pep_path}"
|
||||
if require_cds and (not cds_path.exists() or cds_path.stat().st_size <= 0):
|
||||
return False, f"missing CDS: {cds_path}"
|
||||
return True, str(species_dir)
|
||||
return False, f"missing CDS: {cds_path}"
|
||||
|
||||
|
||||
def _technical_reference_bundle_lookup_code(req: Any, species_code: str) -> str:
|
||||
"""Resolve the local bundle identity without changing the biological species code."""
|
||||
code = str(species_code or "").strip()
|
||||
storage_key = str(getattr(req, "ppiReferenceStorageKey", "") or "").strip()
|
||||
if not storage_key or not bool(getattr(req, "ppiReferenceUserConfirmed", False)):
|
||||
return code
|
||||
stage = str(getattr(req, "stage", "") or "").strip().lower()
|
||||
if stage == "ppi":
|
||||
return storage_key
|
||||
mapping_owner = str(getattr(req, "orthologMappingOwner", "") or "").strip().lower()
|
||||
if stage == "kaks" and mapping_owner == "ppi":
|
||||
return storage_key
|
||||
return code
|
||||
|
||||
|
||||
def _normalize_species_alias_for_job(value: str) -> str:
|
||||
@@ -1686,6 +1735,23 @@ def _looks_like_gene_family_workflow_query(message: str) -> bool:
|
||||
return any(token in text for token in ["基因家族分析", "家族分析", "完整分析", "全流程", "全部阶段", "完整流程"])
|
||||
|
||||
|
||||
def _full_gene_family_workflow_query_text(message: str, history: list[Any]) -> str:
|
||||
"""Recover the user's workflow request across the MainAgent confirmation turn."""
|
||||
current = str(message or "").strip()
|
||||
if _looks_like_gene_family_workflow_query(current):
|
||||
return current
|
||||
for item in reversed(history or []):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
role = str(item.get("role") or item.get("sender") or "").strip().lower()
|
||||
if role not in {"user", "human"}:
|
||||
continue
|
||||
text = str(item.get("content") or item.get("message") or item.get("text") or "").strip()
|
||||
if _looks_like_gene_family_workflow_query(text):
|
||||
return text
|
||||
return ""
|
||||
|
||||
|
||||
def _looks_like_intraspecies_synteny_query(message: str) -> bool:
|
||||
text = str(message or "").lower()
|
||||
return (
|
||||
@@ -1989,7 +2055,7 @@ def _downstream_gene_family_stages_for_context(message: str) -> list[str]:
|
||||
stages = [
|
||||
"phylogeny", "motif", "domain", "structure", "combo",
|
||||
"promoter", "tfbs", "synteny", "interspecies_synteny", "karyotype", "pep",
|
||||
"ortholog", "kaks", "ppi",
|
||||
"ppi", "kaks",
|
||||
]
|
||||
if any(token in text for token in ["表达", "expression", "rna"]):
|
||||
stages.append("expression")
|
||||
@@ -1999,7 +2065,6 @@ def _downstream_gene_family_stages_for_context(message: str) -> list[str]:
|
||||
if any(token in text for token in ["种间", "比较物种", "interspecies"]):
|
||||
stages.append("interspecies_synteny")
|
||||
if any(token in text for token in ["kaks", "ka/ks", "选择压力"]):
|
||||
stages.append("ortholog")
|
||||
stages.append("kaks")
|
||||
if any(token in text for token in ["ppi", "互作"]):
|
||||
stages.append("ppi")
|
||||
@@ -2070,6 +2135,9 @@ def _resolve_contextual_gene_family_request(
|
||||
confirmed = _confirmed_subfamily_species(job_id, family)
|
||||
confirmed_species = [item["species"] for item in confirmed if item.get("source") == "confirmed"]
|
||||
mentioned = _extract_species_codes_from_text_for_job(job_id, text)
|
||||
has_explicit_scope = bool(
|
||||
mentioned or structured_targets or structured_refs or structured_comps
|
||||
)
|
||||
prefix_species = ""
|
||||
if request_prefix and "." in request_prefix:
|
||||
prefix_species = request_prefix.split(".", 1)[0].strip()
|
||||
@@ -2462,7 +2530,7 @@ def _stage_detail_defaults(stage_id: str) -> Dict[str, Any]:
|
||||
"interspecies_synteny": ["identify"],
|
||||
"karyotype": ["identify", "synteny"],
|
||||
"ortholog": ["identify"],
|
||||
"kaks": ["identify", "ortholog"],
|
||||
"kaks": ["identify"],
|
||||
"ppi": ["identify"],
|
||||
"expression": ["identify"],
|
||||
}
|
||||
@@ -2489,18 +2557,16 @@ FULL_GENE_FAMILY_WORKFLOW_STAGES = [
|
||||
"interspecies_synteny",
|
||||
"karyotype",
|
||||
"pep",
|
||||
"ortholog",
|
||||
"kaks",
|
||||
"ppi",
|
||||
"kaks",
|
||||
]
|
||||
|
||||
FORMAL_GENE_FAMILY_CORE_STAGES = {"phylogeny", "motif", "domain", "structure", "combo"}
|
||||
FULL_GENE_FAMILY_TARGET_ONLY_STAGES = {"promoter", "tfbs", "synteny", "karyotype", "pep"}
|
||||
FULL_GENE_FAMILY_TARGET_ONLY_STAGES = {"promoter", "tfbs", "synteny", "karyotype", "pep", "ppi"}
|
||||
FULL_GENE_FAMILY_REFERENCE_STAGES = FORMAL_GENE_FAMILY_CORE_STAGES | {
|
||||
"interspecies_synteny",
|
||||
"ortholog",
|
||||
"kaks",
|
||||
"ppi",
|
||||
}
|
||||
SCOPE_IDENTITY_PARAM_KEYS = (
|
||||
"scopeFile",
|
||||
@@ -2628,6 +2694,8 @@ def _normalize_full_gene_family_workflow_plan(
|
||||
if sid not in ordered:
|
||||
ordered.append(sid)
|
||||
for sid in stage_ids:
|
||||
if sid == "ortholog":
|
||||
continue
|
||||
if sid and sid not in ordered:
|
||||
ordered.append(sid)
|
||||
|
||||
@@ -2656,6 +2724,8 @@ def _normalize_full_gene_family_workflow_plan(
|
||||
detail["dependencies"] = ["phylogeny", "motif", "domain", "structure"]
|
||||
detail_params["comboPanels"] = ["motif", "domain", "structure"]
|
||||
detail_params["comboPanelSource"] = "regular_gene_family"
|
||||
if sid in {"ppi", "kaks"}:
|
||||
detail["dependencies"] = ["identify"]
|
||||
if sid in GENE_FAMILY_SCOPE_STAGES:
|
||||
detail["params"] = enrich_stage_params_with_context(sid, plan_prefix, detail_params)
|
||||
else:
|
||||
@@ -4676,7 +4746,7 @@ def _celery_start_script_for_current_profile() -> Path:
|
||||
port = os.environ.get("PLANTAI_WEB_PORT", "").strip()
|
||||
if prefix.endswith("_9000") or redis_url.endswith("/10") or port == "9000":
|
||||
return PROJECT_ROOT / "start_celery_9000.sh"
|
||||
return PROJECT_ROOT / "start_celery_7880.sh"
|
||||
return PROJECT_ROOT / "scripts" / "legacy_startup" / "start_celery_7880.sh"
|
||||
|
||||
|
||||
def _current_celery_profile() -> Dict[str, Any]:
|
||||
@@ -4686,7 +4756,11 @@ def _current_celery_profile() -> Dict[str, Any]:
|
||||
prefix = get_queue_prefix()
|
||||
except Exception:
|
||||
prefix = os.environ.get("BBAV5_QUEUE_PREFIX", "plantai_dev_7880").strip() or "plantai_dev_7880"
|
||||
redis_default = "redis://localhost:6379/10" if prefix.endswith("_9000") or os.environ.get("PLANTAI_WEB_PORT", "").strip() == "9000" else "redis://localhost:6379/9"
|
||||
current_port = os.environ.get("PLANTAI_WEB_PORT", "").strip()
|
||||
if prefix.endswith("_9000") or current_port == "9000":
|
||||
redis_default = "redis://localhost:6379/10"
|
||||
else:
|
||||
redis_default = "redis://localhost:6379/9"
|
||||
redis_url = os.environ.get("BBAV5_REDIS_URL", redis_default).strip() or redis_default
|
||||
script = _celery_start_script_for_current_profile()
|
||||
log_dir = Path(os.environ.get("BBAV5_CELERY_LOG_DIR", "") or f"logs/celery_{prefix}")
|
||||
@@ -4985,12 +5059,6 @@ orchestrator = None
|
||||
langgraph_orchestrator = None
|
||||
if MAS_AVAILABLE:
|
||||
try:
|
||||
# 清除Python模块缓存,确保配置重新加载
|
||||
import sys
|
||||
modules_to_remove = [k for k in sys.modules.keys() if 'gfa_core.mas' in k]
|
||||
for m in modules_to_remove:
|
||||
del sys.modules[m]
|
||||
|
||||
# 加载MAS配置
|
||||
from gfa_core.mas.config import config as mas_config
|
||||
llm_config = mas_config.load_llm_config()
|
||||
@@ -5182,6 +5250,7 @@ def _prepare_execute_technical_reference_scope(
|
||||
stage == "ortholog"
|
||||
and str(getattr(req, "dependencyOwner", "") or "").strip().lower() == "ppi"
|
||||
)
|
||||
ppi_consumer_intent = stage == "ppi"
|
||||
if ppi_owner_intent:
|
||||
missing_fields = []
|
||||
if not bool(getattr(req, "ppiReferenceUserConfirmed", False)):
|
||||
@@ -5202,6 +5271,8 @@ def _prepare_execute_technical_reference_scope(
|
||||
)
|
||||
ppi_owned_ortholog = ppi_owner_intent
|
||||
if ppi_owned_ortholog:
|
||||
req.analysisRoute = "single_species"
|
||||
req.memberSetId = ""
|
||||
technical_reference_species = [
|
||||
str(getattr(req, "ppiReferenceStorageKey", "") or "").strip()
|
||||
]
|
||||
@@ -5260,12 +5331,22 @@ def _prepare_execute_technical_reference_scope(
|
||||
scope_params["scope_id"] = original_scope_id
|
||||
scope_params["refSpeciesList"] = list(identity.reference_species)
|
||||
scope_params["_validatedPpiOwnerScopeId"] = original_scope_id
|
||||
req.memberSetId = f"{original_scope_id}:single_species:all_members"
|
||||
else:
|
||||
scope_params["scopeMode"] = "target_only"
|
||||
scope_params["referenceSpecies"] = []
|
||||
scope_params["refSpeciesList"] = []
|
||||
scope_params["comparativeSpecies"] = []
|
||||
scope_params["scopeFile"] = ""
|
||||
elif ppi_consumer_intent:
|
||||
req.analysisRoute = "single_species"
|
||||
req.memberSetId = ""
|
||||
technical_reference_species = []
|
||||
scope_params["scopeMode"] = "target_only"
|
||||
scope_params["referenceSpecies"] = []
|
||||
scope_params["refSpeciesList"] = []
|
||||
scope_params["comparativeSpecies"] = []
|
||||
scope_params["scopeFile"] = ""
|
||||
else:
|
||||
technical_reference_species = []
|
||||
if stage in {"ortholog", "ppi", "kaks"}:
|
||||
@@ -6907,6 +6988,7 @@ class ExecuteRequest(BaseModel):
|
||||
ppiReferenceSelectionMode: str = ""
|
||||
ppiReferenceUserConfirmed: bool = False
|
||||
dependencyOwner: str = ""
|
||||
orthologMappingOwner: str = ""
|
||||
identifyBlastReferenceSpecies: List[str] = []
|
||||
primaryReferenceSpecies: List[str] = []
|
||||
refSpeciesList: List[str] = []
|
||||
@@ -7819,6 +7901,7 @@ def run_stage_sync(task_id: str, stage: str, job_id: str, params: dict):
|
||||
ppi_reference_selection_mode=params.get("ppiReferenceSelectionMode") or params.get("ppi_reference_selection_mode") or "",
|
||||
ppi_reference_user_confirmed=_bool_param("ppiReferenceUserConfirmed", "ppi_reference_user_confirmed", default=False),
|
||||
dependency_owner=params.get("dependencyOwner") or params.get("dependency_owner") or "",
|
||||
ortholog_mapping_owner=params.get("orthologMappingOwner") or params.get("ortholog_mapping_owner") or "",
|
||||
ppi_score=params.get("ppiScore", 200),
|
||||
# KaKs参数
|
||||
kaks_neutral_rate=params.get("kaksNeutralRate", 7.08e-9),
|
||||
@@ -8568,7 +8651,7 @@ def run_stage_sync(task_id: str, stage: str, job_id: str, params: dict):
|
||||
async def root():
|
||||
"""返回Vue3前端页面"""
|
||||
return FileResponse(
|
||||
Path(__file__).parent / "index.html",
|
||||
Path(__file__).parent / "index_ml.html",
|
||||
headers={
|
||||
"Cache-Control": "no-cache, no-store, must-revalidate",
|
||||
"Pragma": "no-cache",
|
||||
@@ -8920,6 +9003,7 @@ class ExtensionStageResolveIntentRequest(BaseModel):
|
||||
class ExtensionStageEnvResolveRequest(BaseModel):
|
||||
python_imports: List[str] = Field(default_factory=list)
|
||||
r_packages: List[str] = Field(default_factory=list)
|
||||
env_declaration: Optional[str] = None
|
||||
|
||||
|
||||
|
||||
@@ -9016,6 +9100,24 @@ async def _r_packages_for_env(env_name: str) -> set[str]:
|
||||
|
||||
@app.post("/api/extension-stages/envs/resolve")
|
||||
async def resolve_extension_stage_env(req: ExtensionStageEnvResolveRequest):
|
||||
# 工具显式声明的环境优先(run.sh 里 # lightweight-env: <env>)
|
||||
if req.env_declaration:
|
||||
env = req.env_declaration.strip()
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"conda", "run", "-n", env, "python", "--version",
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
await proc.communicate()
|
||||
if proc.returncode == 0:
|
||||
return {
|
||||
"success": True,
|
||||
"runtime_env": env,
|
||||
"reason": "explicit_env_declaration",
|
||||
"dependency": env,
|
||||
"fallback": False,
|
||||
}
|
||||
|
||||
python_imports = _normalized_dependency_names(req.python_imports)
|
||||
r_packages = _normalized_dependency_names(req.r_packages)
|
||||
if not python_imports and not r_packages:
|
||||
@@ -11650,7 +11752,12 @@ async def execute_stage(req: ExecuteRequest):
|
||||
for raw_value in raw_values:
|
||||
raw_code = str(raw_value or "").strip()
|
||||
if raw_code and raw_code.lower() in technical_ref_lookup:
|
||||
available, reason = _reference_bundle_species_available(job_id, raw_code, require_cds=require_cds)
|
||||
bundle_lookup_code = _technical_reference_bundle_lookup_code(req, raw_code)
|
||||
available, reason = _reference_bundle_species_available(
|
||||
job_id,
|
||||
bundle_lookup_code,
|
||||
require_cds=require_cds,
|
||||
)
|
||||
if not available:
|
||||
raise ValueError(
|
||||
f"PPI/KaKs 技术参考物种 {raw_code} 已选择但参考包不完整:{reason};"
|
||||
@@ -12350,6 +12457,7 @@ async def execute_stage(req: ExecuteRequest):
|
||||
"ppiScore": getattr(req, 'ppiScore', 200),
|
||||
# KaKs参数
|
||||
"kaksNeutralRate": getattr(req, 'kaksNeutralRate', 7.08e-9),
|
||||
"orthologMappingOwner": getattr(req, 'orthologMappingOwner', ''),
|
||||
# Genome Prepare参数
|
||||
"genomePrepareType": getattr(req, 'genomePrepareType', 'auto'),
|
||||
# Interspecies species selection
|
||||
@@ -13382,6 +13490,134 @@ def restart_project(request: Request, payload: AdminOpsRequest):
|
||||
}
|
||||
|
||||
|
||||
@app.post("/api/admin/validate-qa-sidecar")
|
||||
def validate_qa_sidecar(request: Request, payload: AdminOpsRequest):
|
||||
"""Launch the fixed full-graph QA sidecar validation workflow."""
|
||||
auth = _validate_admin_ops_request(request, payload, "validate-qa-sidecar")
|
||||
script = PROJECT_ROOT / "scripts" / "validate_qa_v1_1_73_sidecar.sh"
|
||||
if not script.is_file():
|
||||
return {
|
||||
"success": False,
|
||||
"status": "not_configured",
|
||||
"message": f"QA sidecar validation script is missing: {script}",
|
||||
"auth": auth,
|
||||
}
|
||||
|
||||
log_path = PROJECT_ROOT / ".runtime" / "qa_v1_1_73" / "sidecar_admin.log"
|
||||
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
log_handle = open(log_path, "a", encoding="utf-8")
|
||||
subprocess.Popen(
|
||||
["bash", str(script)],
|
||||
cwd=str(PROJECT_ROOT),
|
||||
stdout=log_handle,
|
||||
stderr=subprocess.STDOUT,
|
||||
start_new_session=True,
|
||||
)
|
||||
return {
|
||||
"success": True,
|
||||
"status": "validation_submitted",
|
||||
"message": "QA sidecar validation submitted on the fixed loopback port.",
|
||||
"auth": auth,
|
||||
"log": str(log_path),
|
||||
}
|
||||
except Exception as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"status": "submit_failed",
|
||||
"message": str(exc),
|
||||
"auth": auth,
|
||||
"log": str(log_path),
|
||||
}
|
||||
|
||||
|
||||
def _qa_sidecar_process_status() -> Dict[str, Any]:
|
||||
runtime_dir = PROJECT_ROOT / ".runtime" / "qa_v1_1_73" / "formal_sidecar_7945"
|
||||
pidfile = runtime_dir / "qa_7945.pid"
|
||||
result: Dict[str, Any] = {
|
||||
"pidfile": str(pidfile),
|
||||
"pidfile_exists": pidfile.is_file(),
|
||||
"running": False,
|
||||
"identity_verified": False,
|
||||
}
|
||||
if not pidfile.is_file():
|
||||
return result
|
||||
|
||||
try:
|
||||
pid = int(pidfile.read_text(encoding="utf-8").strip())
|
||||
except (OSError, ValueError) as exc:
|
||||
result["error"] = f"invalid pidfile: {exc}"
|
||||
return result
|
||||
|
||||
result["pid"] = pid
|
||||
proc_dir = Path("/proc") / str(pid)
|
||||
cmdline_path = proc_dir / "cmdline"
|
||||
if not cmdline_path.is_file():
|
||||
result["status"] = "stale_pidfile"
|
||||
return result
|
||||
|
||||
try:
|
||||
cmdline = cmdline_path.read_bytes().replace(b"\0", b" ").decode("utf-8", errors="replace").strip()
|
||||
except OSError as exc:
|
||||
result["error"] = f"cannot inspect process: {exc}"
|
||||
return result
|
||||
|
||||
expected_markers = ("uvicorn", "scripts.qa_v1_1_73_server:app", "7945")
|
||||
result.update(
|
||||
{
|
||||
"running": True,
|
||||
"identity_verified": all(marker in cmdline for marker in expected_markers),
|
||||
"command": cmdline,
|
||||
"status": "running",
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
@app.post("/api/admin/qa-sidecar-status")
|
||||
def qa_sidecar_status(request: Request, payload: AdminOpsRequest):
|
||||
"""Inspect only the fixed QA sidecar process on loopback port 7945."""
|
||||
auth = _validate_admin_ops_request(request, payload, "qa-sidecar-status")
|
||||
return {"success": True, "auth": auth, "sidecar": _qa_sidecar_process_status()}
|
||||
|
||||
|
||||
@app.post("/api/admin/stop-qa-sidecar")
|
||||
def stop_qa_sidecar(request: Request, payload: AdminOpsRequest):
|
||||
"""Stop only the identity-verified QA sidecar on loopback port 7945."""
|
||||
auth = _validate_admin_ops_request(request, payload, "stop-qa-sidecar")
|
||||
before = _qa_sidecar_process_status()
|
||||
if not before.get("running"):
|
||||
return {
|
||||
"success": True,
|
||||
"status": "already_stopped",
|
||||
"auth": auth,
|
||||
"before": before,
|
||||
}
|
||||
if not before.get("identity_verified"):
|
||||
return {
|
||||
"success": False,
|
||||
"status": "identity_mismatch",
|
||||
"message": "PID exists but is not the fixed QA sidecar; refusing to signal it.",
|
||||
"auth": auth,
|
||||
"before": before,
|
||||
}
|
||||
|
||||
os.kill(int(before["pid"]), signal.SIGTERM)
|
||||
deadline = time.monotonic() + 5.0
|
||||
after = _qa_sidecar_process_status()
|
||||
while after.get("running") and time.monotonic() < deadline:
|
||||
time.sleep(0.1)
|
||||
after = _qa_sidecar_process_status()
|
||||
stopped = not after.get("running")
|
||||
return {
|
||||
"success": stopped,
|
||||
"status": "stopped" if stopped else "stop_submitted",
|
||||
"auth": auth,
|
||||
"before": before,
|
||||
"after": after,
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
class ResourceConfigRequest(BaseModel):
|
||||
@@ -15549,7 +15785,7 @@ def _apply_reference_bundle_stale_watchdog(task: Dict[str, Any]) -> None:
|
||||
progress = task.get("progress") if isinstance(task.get("progress"), dict) else {}
|
||||
progress = {
|
||||
"phase": progress.get("phase") or status or "running",
|
||||
"message": progress.get("message") or task.get("message") or "参考物种注释包仍在后台下载",
|
||||
"message": progress.get("message") or task.get("message") or "PPI and Ka/Ks reference species data are still downloading",
|
||||
"percentage": progress.get("percentage"),
|
||||
"indeterminate": progress.get("indeterminate", True),
|
||||
"pending_seconds": task["pending_seconds"],
|
||||
@@ -15567,13 +15803,13 @@ def _apply_reference_bundle_stale_watchdog(task: Dict[str, Any]) -> None:
|
||||
def run_reference_bundle_download_task(task_id: str, payload: Dict[str, Any]) -> None:
|
||||
tasks[task_id].update({
|
||||
"status": "running",
|
||||
"message": "正在后台下载参考物种注释包",
|
||||
"message": "Downloading PPI and Ka/Ks reference species PEP and CDS",
|
||||
"start_time": datetime.now().isoformat(),
|
||||
"timeout_sec": int(payload.get("timeout_sec") or 300),
|
||||
"payload": dict(payload),
|
||||
"progress": {
|
||||
"phase": "queued",
|
||||
"message": "参考物种注释包下载任务已启动",
|
||||
"message": "PPI and Ka/Ks reference species data download started",
|
||||
"percentage": 0,
|
||||
"indeterminate": True,
|
||||
},
|
||||
@@ -15604,7 +15840,7 @@ def run_reference_bundle_download_task(task_id: str, payload: Dict[str, Any]) ->
|
||||
progress.setdefault("indeterminate", percentage is None)
|
||||
tasks[task_id].update({
|
||||
"status": "running",
|
||||
"message": progress.get("message") or "正在后台下载参考物种注释包",
|
||||
"message": progress.get("message") or "Downloading PPI and Ka/Ks reference species PEP and CDS",
|
||||
"progress": progress,
|
||||
})
|
||||
_persist_reference_bundle_task(tasks[task_id])
|
||||
@@ -15623,11 +15859,11 @@ def run_reference_bundle_download_task(task_id: str, payload: Dict[str, Any]) ->
|
||||
success = result.get("status") == "success"
|
||||
tasks[task_id].update({
|
||||
"status": "success" if success else "failed",
|
||||
"message": "参考物种注释包下载完成" if success else "参考物种注释包下载未满足必需文件",
|
||||
"message": "PPI and Ka/Ks reference species data are ready" if success else "PPI and Ka/Ks reference species data are incomplete",
|
||||
"end_time": datetime.now().isoformat(),
|
||||
"progress": {
|
||||
"phase": "completed" if success else "failed",
|
||||
"message": "参考物种注释包下载完成" if success else "参考物种注释包下载未满足必需文件",
|
||||
"message": "PPI and Ka/Ks reference species data are ready" if success else "PPI and Ka/Ks reference species data are incomplete",
|
||||
"percentage": 100 if success else None,
|
||||
"indeterminate": False if success else True,
|
||||
},
|
||||
@@ -15670,7 +15906,7 @@ async def download_reference_bundle_api(req: ReferenceBundleDownloadRequest):
|
||||
"stage": "reference_bundle_download",
|
||||
"job_id": req.job_id,
|
||||
"status": "pending",
|
||||
"message": "参考物种注释包下载任务已提交",
|
||||
"message": "PPI and Ka/Ks reference species data download submitted",
|
||||
"created_at": datetime.now().isoformat(),
|
||||
"species_code": req.speciesCode,
|
||||
"species_name": req.speciesName,
|
||||
@@ -17132,7 +17368,78 @@ def _gene_family_report_plan_context(job_id: str, prefix: str = "") -> Optional[
|
||||
"rnaseqScopeId": "",
|
||||
}
|
||||
reports.append((path.stat().st_mtime, context))
|
||||
return max(reports, key=lambda item: item[0])[1] if reports else None
|
||||
if reports:
|
||||
return max(reports, key=lambda item: item[0])[1]
|
||||
|
||||
# Older jobs may have complete, successful stage manifests but no saved
|
||||
# confirmation-plan snapshot. Reconstruct a deterministic report plan from
|
||||
# one manifest prefix rather than rejecting those verifiable results.
|
||||
report_stage_order = (
|
||||
"identify",
|
||||
"phylogeny",
|
||||
"reference_classification",
|
||||
"motif",
|
||||
"domain",
|
||||
"structure",
|
||||
"combo",
|
||||
"promoter",
|
||||
"tfbs",
|
||||
"synteny",
|
||||
"karyotype",
|
||||
"interspecies_synteny",
|
||||
"pep",
|
||||
)
|
||||
report_stage_set = set(report_stage_order)
|
||||
recovered_by_prefix: Dict[str, Dict[str, tuple[float, Dict[str, Any]]]] = {}
|
||||
for path in final_dir.glob("*.manifest.json") if final_dir.is_dir() else ():
|
||||
try:
|
||||
manifest = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
continue
|
||||
stage = str(manifest.get("stage") or "").strip()
|
||||
manifest_prefix = str(
|
||||
manifest.get("prefix")
|
||||
or (manifest.get("parameters") or {}).get("prefix")
|
||||
or ""
|
||||
).strip()
|
||||
if (
|
||||
stage not in report_stage_set
|
||||
or str(manifest.get("status") or "") != "success"
|
||||
or not manifest_prefix
|
||||
or (requested_prefix and manifest_prefix != requested_prefix)
|
||||
):
|
||||
continue
|
||||
stage_map = recovered_by_prefix.setdefault(manifest_prefix, {})
|
||||
candidate = (path.stat().st_mtime, manifest)
|
||||
if stage not in stage_map or candidate[0] > stage_map[stage][0]:
|
||||
stage_map[stage] = candidate
|
||||
|
||||
if not recovered_by_prefix:
|
||||
return None
|
||||
|
||||
def recovered_rank(item: tuple[str, Dict[str, tuple[float, Dict[str, Any]]]]) -> tuple[int, int, float, str]:
|
||||
manifest_prefix, stage_map = item
|
||||
latest_mtime = max((entry[0] for entry in stage_map.values()), default=0.0)
|
||||
return (
|
||||
1 if "identify" in stage_map else 0,
|
||||
len(stage_map),
|
||||
latest_mtime,
|
||||
manifest_prefix,
|
||||
)
|
||||
|
||||
recovered_prefix, recovered_stages = max(recovered_by_prefix.items(), key=recovered_rank)
|
||||
planned = [stage for stage in report_stage_order if stage in recovered_stages]
|
||||
if not planned:
|
||||
return None
|
||||
return {
|
||||
"prefix": recovered_prefix,
|
||||
"expectedStages": planned,
|
||||
"stageStatuses": {stage: "success" for stage in planned},
|
||||
"scopeId": "",
|
||||
"analysisRoute": "",
|
||||
"memberSetId": "",
|
||||
"rnaseqScopeId": "",
|
||||
}
|
||||
|
||||
|
||||
async def _run_agent_chat(
|
||||
@@ -17146,11 +17453,6 @@ async def _run_agent_chat(
|
||||
executes analysis stages here.
|
||||
"""
|
||||
session_id = request.session_id or f"session_{int(time.time())}"
|
||||
# Inject language instruction for non-Chinese languages
|
||||
if request.language and request.language != DEFAULT_AGENT_LANG:
|
||||
lang_map = {"en": "Please respond in English."}
|
||||
lang_instruction = lang_map.get(request.language, f"Please respond in {request.language}.")
|
||||
request.message = f"{lang_instruction}\n\n{request.message}"
|
||||
completed_stages: list[str] = []
|
||||
if request.job_id:
|
||||
final_dir = PROJECT_ROOT / "workspace" / request.job_id / "final"
|
||||
@@ -17647,6 +17949,11 @@ async def _run_agent_chat(
|
||||
if isinstance(outcome.get("proposal"), dict)
|
||||
else None
|
||||
),
|
||||
recommendation=(
|
||||
outcome.get("recommendation")
|
||||
if isinstance(outcome.get("recommendation"), dict)
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
return {
|
||||
@@ -17668,8 +17975,17 @@ async def _run_agent_chat(
|
||||
"model_used": request.model,
|
||||
"session_id": session_id,
|
||||
"input_paths": list(outcome.get("input_paths") or []),
|
||||
"recommendation": (
|
||||
outcome.get("recommendation")
|
||||
if isinstance(outcome.get("recommendation"), dict)
|
||||
else None
|
||||
),
|
||||
"handoff_request": (
|
||||
str((outcome.get("proposal") or {}).get("request") or "").strip()
|
||||
str(
|
||||
(outcome.get("proposal") or {}).get("original_request")
|
||||
or (outcome.get("proposal") or {}).get("request")
|
||||
or ""
|
||||
).strip()
|
||||
if isinstance(outcome.get("proposal"), dict)
|
||||
else request.message
|
||||
) or request.message,
|
||||
@@ -18549,6 +18865,41 @@ class AgentPlanRequest(BaseModel):
|
||||
language: str = DEFAULT_AGENT_LANG # "zh" or "en"
|
||||
|
||||
|
||||
def _authorized_workflow_action_stages(intent: Dict[str, Any]) -> List[str]:
|
||||
candidate = intent or {}
|
||||
params = candidate.get("params") or {}
|
||||
action_id = str(
|
||||
candidate.get("workflow_action_id") or params.get("workflowActionId") or ""
|
||||
).strip()
|
||||
authorized = list(dict.fromkeys(_param_list(params.get("authorizedStages"))))
|
||||
declared = set(_param_list(candidate.get("stages")))
|
||||
if not action_id or not authorized or not set(authorized).issubset(declared):
|
||||
return []
|
||||
try:
|
||||
from gfa_core.mas.workflow_catalog import load_canonical_workflow_contract
|
||||
|
||||
contract = load_canonical_workflow_contract("gene_family")
|
||||
transition = next(item for item in contract.transitions if item.id == action_id)
|
||||
actions = {item.id: item for item in contract.actions}
|
||||
evidence_stages = {
|
||||
item.id: item.stage for item in contract.evidence_requirements
|
||||
}
|
||||
contract_stages = [
|
||||
evidence_stages[requirement_id]
|
||||
for requirement_id in transition.evidence_requirement_ids
|
||||
]
|
||||
contract_stages.append(actions[transition.action_id].target_stage)
|
||||
except (ImportError, StopIteration, KeyError):
|
||||
return []
|
||||
if set(authorized) != set(contract_stages):
|
||||
return []
|
||||
return [
|
||||
stage
|
||||
for stage in SUBFAMILY_PREDICTION_PLAN_STAGES
|
||||
if stage in set(contract_stages)
|
||||
]
|
||||
|
||||
|
||||
SUBFAMILY_PREDICTION_PLAN_STAGES = [
|
||||
"identify",
|
||||
"phylogeny",
|
||||
@@ -18832,9 +19183,13 @@ def _build_contextual_gene_family_plan_response(request: Any, session_id: str, c
|
||||
detail["params"] = enrich_stage_params_with_context(stage_id, plan_prefix, detail_params)
|
||||
stages_detail.append(detail)
|
||||
|
||||
if _looks_like_gene_family_workflow_query(getattr(request, "message", "") or "") and not subfamily_prerequisite_mode:
|
||||
full_workflow_query = _full_gene_family_workflow_query_text(
|
||||
getattr(request, "message", "") or "",
|
||||
getattr(request, "history", []) or [],
|
||||
)
|
||||
if full_workflow_query and not subfamily_prerequisite_mode:
|
||||
stage_ids, stages_detail = _normalize_full_gene_family_workflow_plan(
|
||||
message=getattr(request, "message", "") or "",
|
||||
message=full_workflow_query,
|
||||
stage_ids=stage_ids,
|
||||
stages_detail=stages_detail,
|
||||
plan_params=plan_params,
|
||||
@@ -20636,6 +20991,7 @@ async def agent_plan(request: AgentPlanRequest):
|
||||
request.intent = dict(request.intent or {})
|
||||
request.intent["params"] = dict(request.intent.get("params") or {})
|
||||
_apply_reconciled_species_to_intent(request.intent, plan_role_resolution)
|
||||
authorized_workflow_stages = _authorized_workflow_action_stages(request.intent)
|
||||
query_lower_for_context = (request.message or "").lower()
|
||||
result_report_query = (
|
||||
"结果报告" in query_lower_for_context
|
||||
@@ -20705,7 +21061,7 @@ async def agent_plan(request: AgentPlanRequest):
|
||||
and not _looks_like_gene_family_workflow_query(query_lower_for_context)
|
||||
and not _requires_confirmed_subfamily_query(query_lower_for_context)
|
||||
)
|
||||
if subfamily_prediction_query_for_context:
|
||||
if subfamily_prediction_query_for_context or authorized_workflow_stages:
|
||||
requested_prefix_for_subfamily = request.prefix or ""
|
||||
prefix_species, prefix_family = _prefix_species_family(requested_prefix_for_subfamily)
|
||||
family_for_subfamily = _resolve_authoritative_gene_family(
|
||||
@@ -20768,7 +21124,27 @@ async def agent_plan(request: AgentPlanRequest):
|
||||
}
|
||||
if family_for_subfamily and targets_for_subfamily:
|
||||
prefix_for_subfamily = f"{targets_for_subfamily[0]}.{family_for_subfamily}" if len(targets_for_subfamily) == 1 else f"Target.{family_for_subfamily}"
|
||||
deterministic_stages = ["identify", "phylogeny", "motif", "domain", "structure", "combo", "reference_classification"]
|
||||
deterministic_stages = authorized_workflow_stages or [
|
||||
"identify",
|
||||
"phylogeny",
|
||||
"motif",
|
||||
"domain",
|
||||
"structure",
|
||||
"combo",
|
||||
"reference_classification",
|
||||
]
|
||||
authorized_action_id = str(
|
||||
(request.intent or {}).get("workflow_action_id")
|
||||
or ((request.intent or {}).get("params") or {}).get(
|
||||
"workflowActionId"
|
||||
)
|
||||
or ""
|
||||
).strip()
|
||||
plan_description = (
|
||||
f"执行已确认的工作流业务动作 {authorized_action_id}"
|
||||
if authorized_workflow_stages
|
||||
else "执行 target/comparative + reference 亚家族分类"
|
||||
)
|
||||
classification_species_for_subfamily = list(targets_for_subfamily)
|
||||
classification_species_for_subfamily, role_targets_for_subfamily, role_comps_for_subfamily = _split_subfamily_selected_species_by_config_roles(
|
||||
request.job_id,
|
||||
@@ -20790,6 +21166,12 @@ async def agent_plan(request: AgentPlanRequest):
|
||||
"confirmPrefixBySpecies": confirm_prefix_by_species,
|
||||
"subfamilyState": subfamily_state_for_plan,
|
||||
}
|
||||
if authorized_workflow_stages:
|
||||
request_action_params = (request.intent or {}).get("params") or {}
|
||||
plan_params["workflowActionId"] = authorized_action_id
|
||||
plan_params["authorizedStages"] = _param_list(
|
||||
request_action_params.get("authorizedStages")
|
||||
)
|
||||
if identify_selection_for_subfamily:
|
||||
plan_params["identifySpeciesSelection"] = _identify_selection_with_default(
|
||||
identify_selection_for_subfamily,
|
||||
@@ -20859,7 +21241,7 @@ async def agent_plan(request: AgentPlanRequest):
|
||||
"comparative_species": role_comps_for_subfamily,
|
||||
"scopeMode": plan_params["scopeMode"],
|
||||
"prefix": prefix_for_subfamily,
|
||||
"description": "执行 target/comparative + reference 亚家族分类",
|
||||
"description": plan_description,
|
||||
"estimated_time": f"{estimated_minutes}分钟",
|
||||
"estimated_total_time": f"{estimated_minutes}分钟",
|
||||
"execution_strategy": "parallel",
|
||||
@@ -20881,10 +21263,15 @@ async def agent_plan(request: AgentPlanRequest):
|
||||
"comparative_species": role_comps_for_subfamily,
|
||||
"scopeMode": plan_params["scopeMode"],
|
||||
"stages": deterministic_stages,
|
||||
"description": "执行目标物种+参考物种亚家族分类",
|
||||
"description": plan_description,
|
||||
"confidence": 0.98,
|
||||
"prefix": prefix_for_subfamily,
|
||||
"params": plan_params,
|
||||
**(
|
||||
{"workflow_action_id": authorized_action_id}
|
||||
if authorized_action_id
|
||||
else {}
|
||||
),
|
||||
},
|
||||
"completed_stages": completed_stages,
|
||||
"session_id": session_id,
|
||||
@@ -22607,9 +22994,13 @@ async def agent_plan(request: AgentPlanRequest):
|
||||
intent["gene_family"] = combo_family
|
||||
|
||||
full_workflow_normalized = False
|
||||
if _looks_like_gene_family_workflow_query(request.message or "") and not subfamily_prediction_query:
|
||||
full_workflow_query = _full_gene_family_workflow_query_text(
|
||||
request.message or "",
|
||||
request.history or [],
|
||||
)
|
||||
if full_workflow_query and not subfamily_prediction_query:
|
||||
stage_ids, stages_detail = _normalize_full_gene_family_workflow_plan(
|
||||
message=request.message or "",
|
||||
message=full_workflow_query,
|
||||
stage_ids=stage_ids,
|
||||
stages_detail=stages_detail,
|
||||
plan_params=plan_params,
|
||||
@@ -22641,7 +23032,7 @@ async def agent_plan(request: AgentPlanRequest):
|
||||
|
||||
if full_workflow_normalized:
|
||||
stage_ids, stages_detail = _normalize_full_gene_family_workflow_plan(
|
||||
message=request.message or "",
|
||||
message=full_workflow_query,
|
||||
stage_ids=stage_ids,
|
||||
stages_detail=stages_detail,
|
||||
plan_params=plan_params,
|
||||
@@ -22876,6 +23267,7 @@ class AgentRunRequest(BaseModel):
|
||||
job_id: str = ""
|
||||
intent: dict = {} # 前端可传入已有的intent
|
||||
stages: list = [] # 要执行的阶段列表
|
||||
language: str = DEFAULT_AGENT_LANG # "zh" or "en"
|
||||
|
||||
|
||||
@app.post("/api/agent/run")
|
||||
@@ -22909,6 +23301,7 @@ async def agent_run(request: AgentRunRequest):
|
||||
)
|
||||
|
||||
# 添加额外的运行时字段
|
||||
initial_state["language"] = request.language
|
||||
initial_state["status"] = "running"
|
||||
initial_state["intent_result"] = request.intent or {} # 传入已有的 intent
|
||||
initial_state["completed_stages"] = []
|
||||
@@ -24916,6 +25309,8 @@ async def rag_query(request: RAGQueryRequest):
|
||||
|
||||
基于文献知识库回答问题
|
||||
"""
|
||||
import requests
|
||||
|
||||
try:
|
||||
from gfa_core.mas.dual_kb_qa import query_dual_kb_qa
|
||||
|
||||
@@ -24941,6 +25336,32 @@ async def rag_query(request: RAGQueryRequest):
|
||||
payload["evidence_rag_plan"] = evidence_rag_plan
|
||||
payload["augmented_query"] = augmented_query
|
||||
return payload
|
||||
except requests.Timeout:
|
||||
augmented_query = request.query or ""
|
||||
try:
|
||||
augmented_query, evidence_rag_plan = _augment_rag_query_with_candidate_context(request)
|
||||
_save_candidate_rag_references(
|
||||
request,
|
||||
evidence_rag_plan,
|
||||
augmented_query,
|
||||
sources=[],
|
||||
status="request_timeout",
|
||||
note="complete QA exceeded the configured compatibility timeout",
|
||||
)
|
||||
except Exception:
|
||||
evidence_rag_plan = {}
|
||||
return JSONResponse(
|
||||
status_code=504,
|
||||
content={
|
||||
"success": False,
|
||||
"error_code": "request_timeout",
|
||||
"message": "完整知识图谱问答在配置时限内没有返回。",
|
||||
"answer": "",
|
||||
"sources": [],
|
||||
"mode": "gene_family_mas_qa_timeout",
|
||||
"evidence_rag_plan": evidence_rag_plan,
|
||||
},
|
||||
)
|
||||
except Exception as e:
|
||||
import traceback
|
||||
augmented_query = request.query or ""
|
||||
@@ -24973,7 +25394,6 @@ async def rag_query_stream(request: RAGQueryRequest):
|
||||
for frame in stream_dual_kb_qa(
|
||||
augmented_query,
|
||||
top_k=request.top_k or 10,
|
||||
timeout=300.0,
|
||||
):
|
||||
if frame.startswith("event: complete"):
|
||||
try:
|
||||
@@ -25246,7 +25666,7 @@ async def root():
|
||||
"""
|
||||
返回前端首页
|
||||
"""
|
||||
index_file = WEB_DIR / "index.html"
|
||||
index_file = WEB_DIR / "index_ml.html"
|
||||
if index_file.exists():
|
||||
return FileResponse(
|
||||
str(index_file),
|
||||
|
||||
@@ -712,6 +712,179 @@ window.__I18N_EN__ = {
|
||||
" 已暂停依赖 confirmed_subfamily.tsv": " has paused stages depending on confirmed_subfamily.tsv",
|
||||
";": ";",
|
||||
" 个分析阶段": " analysis stages",
|
||||
" 个阶段": " stages"
|
||||
};
|
||||
" 个阶段": " stages",
|
||||
|
||||
// ===== 阶段 7:ElMessage/ElMessageBox i18n keys =====
|
||||
// 注意:key 为中文原文(zh 模式直接返回 key),value 为 English
|
||||
// tp() 类 key 需要同时注册变量 placeholder 版本和 static 文案
|
||||
|
||||
// ElMessageBox
|
||||
"删除会话": "Delete Session",
|
||||
"取消 PPI": "Cancel PPI",
|
||||
"自动注释并继续": "Auto-annotate & Continue",
|
||||
"我先去上传背景文件": "Upload Background Files First",
|
||||
"确认设计并写入": "Confirm Design & Write",
|
||||
"返回修改": "Return to Modify",
|
||||
"确认第一候选": "Confirm First Candidate",
|
||||
"使用 bcftools 重试": "Retry with bcftools",
|
||||
"GATK 执行失败": "GATK Execution Failed",
|
||||
"确认 DE 分析设计": "Confirm DE Analysis Design",
|
||||
"确认 PPI 的 STRING 近源参考物种": "Confirm STRING Close Reference Species for PPI",
|
||||
|
||||
// autoAnnotation 标题
|
||||
"GO/KEGG 富集背景缺失": "GO/KEGG Enrichment Background Missing",
|
||||
"KEGG 富集背景缺失": "KEGG Enrichment Background Missing",
|
||||
"GO 富集背景缺失": "GO Enrichment Background Missing",
|
||||
"RNA-seq 富集背景缺失": "RNA-seq Enrichment Background Missing",
|
||||
|
||||
// 会话管理
|
||||
"示例下载功能当前不可用": "Example download is currently unavailable",
|
||||
"已Clear当前会话": "Current session cleared",
|
||||
|
||||
// autoFill
|
||||
"请至少填写一个要预览修改的字段": "Please fill at least one field to preview modifications",
|
||||
"没有匹配的草稿行": "No matching draft rows",
|
||||
"请先预览要应用的修改": "Please preview modifications before applying",
|
||||
"请先生成或加载草稿": "Please generate or load a draft first",
|
||||
"生成草稿失败": "Failed to generate draft",
|
||||
"确认失败": "Confirmation failed",
|
||||
"已写入正式 config.xlsx": "Formal config.xlsx written",
|
||||
"已写入正式 samples.yaml": "Formal samples.yaml written",
|
||||
"请输入文件路径": "Please enter file path",
|
||||
"加载失败": "Load failed",
|
||||
|
||||
// RAG 解读
|
||||
"请求解读服务失败": "Interpretation service request failed",
|
||||
"RAG解读已刷新": "RAG interpretation refreshed",
|
||||
"RAG解读刷新失败": "RAG interpretation refresh failed",
|
||||
"下载报告失败": "Failed to download report",
|
||||
"JSON数据已加载到控制台": "JSON data loaded to console",
|
||||
"解读内容已复制": "Interpretation content copied",
|
||||
"复制失败": "Copy failed",
|
||||
|
||||
// 知识图谱
|
||||
"知识图谱加载完成": "Knowledge graph loaded",
|
||||
"未找到匹配的节点": "No matching node found",
|
||||
"视图已重置": "View reset",
|
||||
"请先通过问答加载子图,或点击「重新加载」": "Please load subgraph via Q&A first, or click Reload",
|
||||
|
||||
// PPI/STRING
|
||||
"当前没有可用 job_id": "No job_id currently available",
|
||||
"该 ID 已存在": "This ID already exists",
|
||||
"至少保留一个 gene ID": "Keep at least one gene ID",
|
||||
|
||||
// Working Directory
|
||||
"切换失败": "Switch failed",
|
||||
|
||||
// 图片/可视化
|
||||
"请先选择一张图片": "Please select an image first",
|
||||
"无法确定启动子输出前缀。": "Cannot determine promoter output prefix.",
|
||||
"请至少选择一个启动子元件或类别。": "Please select at least one promoter element or category.",
|
||||
"请至少选择一种图类型。": "Please select at least one plot type.",
|
||||
"无法确定进化树输出标签,请先选择树图或填写输出标签。": "Cannot determine phylogeny tree output label. Please select a tree image or fill in the output label first.",
|
||||
"进化树图重绘完成,已切换到新图": "Phylogeny tree replot complete; switched to new image",
|
||||
"无法确定种内共线性输出标签,请先选择 Circos 图。": "Cannot determine intraspecies synteny output label. Please select a Circos image first.",
|
||||
"无法确定 combo 输出标签,请先选择 combo 图。": "Cannot determine combo output label. Please select a combo image first.",
|
||||
"Combo 修订至少选择两个类型。": "Combo revision requires at least two panel types.",
|
||||
"Combo 图修订完成,已切换到新图": "Combo plot revision complete; switched to new image",
|
||||
"无法确定MSA输出标签,请先选择对应图片或填写输出标签。": "Cannot determine MSA output label. Please select corresponding image or fill in output label.",
|
||||
"请填写MSA alignment 起始列和终止列,或选择其他模式。": "Please fill in MSA alignment start/end columns or choose another mode.",
|
||||
"请填写参考基因及其起止残基编号。": "Please fill in reference gene and its start/end residue numbers.",
|
||||
|
||||
// GWAS/诊断
|
||||
"请填写 GWAS 物种简称(用于结果前缀)": "Please fill in GWAS species abbreviation (for result prefix)",
|
||||
"请先选择有效 VCF": "Please select a valid VCF first",
|
||||
"请先SelectAnalysis Stage": "Please select an analysis stage first",
|
||||
"诊断未发现明确问题,请查看阶段错误日志。": "Diagnosis found no clear issues. Please check stage error logs.",
|
||||
"诊断仅提示信息,无需确认修复。": "Diagnosis is informational only; no fix confirmation needed.",
|
||||
"✅ 已应用修复方案,继续执行...": "✅ Fix applied; continuing execution...",
|
||||
"❌ 应用修复失败": "❌ Fix application failed",
|
||||
"❌ 网络错误,请重试": "❌ Network error; please retry",
|
||||
"⏸ 已跳过修复;该问题会阻断下游,后续阶段已暂停。": "⏸ Fix skipped; issue blocks downstream; subsequent stages paused.",
|
||||
"已关闭诊断提示。": "Diagnosis prompt closed.",
|
||||
|
||||
// 任务/消息
|
||||
"当前消息未成功写入会话历史,刷新后可能丢失": "Message not persisted to session history; may be lost on refresh",
|
||||
"路径已复制": "Path copied",
|
||||
|
||||
// 拓展分析
|
||||
"当前筛选没有成员,未重建 final_members。": "No members matching current filter; final_members not rebuilt.",
|
||||
"当前选择为按物种分别分析,但后端没有返回完整的分物种亚家族结果;将尝试打开可用的联合结果。": "Per-species analysis selected but backend returned incomplete results; attempting to open available joint results.",
|
||||
"亚家族预测已完成,但未能自动打开确认表;请发送\"调出亚家族确认表\"手动打开。": "Subfamily prediction completed but confirmation table could not open automatically; please send \"show subfamily review table\" manually.",
|
||||
"没有可确认的亚家族标签。": "No subfamily labels to confirm.",
|
||||
"已请求停止当前拓展分析;状态仍会继续轮询直到进程退出。": "Stop requested for current extension; polling will continue until process exits.",
|
||||
"No running task to stop": "No running task to stop",
|
||||
"Task(s) stopped (backend + Celery when available)": "Task(s) stopped (backend + Celery when available)",
|
||||
"File uploaded successfully": "File uploaded successfully",
|
||||
"未检测到可用文件": "No usable files detected",
|
||||
"请先生成拓展分析阶段。": "Please generate the extension stage first.",
|
||||
"发布前验收已通过": "Pre-publish validation passed",
|
||||
"请填写名称,并上传原始代码。": "Please provide name and upload source code.",
|
||||
"拓展分析阶段已生成并发布": "Extension stage generated and published",
|
||||
"确认已保存": "Confirmation saved",
|
||||
"验证成功": "Validation successful",
|
||||
"验证未通过": "Validation not passed",
|
||||
"拓展分析阶段已发布": "Extension stage published",
|
||||
"拓展分析阶段执行完成": "Extension stage execution complete",
|
||||
"拓展分析阶段执行未成功,请查看结果日志": "Extension stage execution incomplete; please check result logs",
|
||||
"该阶段已在正式列表中": "This stage is already in the formal list",
|
||||
"已加入正式阶段列表,并默认纳入本次执行": "Added to formal stage list and included in this execution",
|
||||
|
||||
// 方法学
|
||||
"请先选择 PDF 或 MinerU content_list JSON 文献文件": "Please select a PDF or MinerU content_list JSON literature file first",
|
||||
"方法学提取启动失败": "Methodology extraction failed to start",
|
||||
"方法学 sandbox 示例读取失败": "Failed to read methodology sandbox examples",
|
||||
"请先选择或输入 methodology_dir": "Please select or enter methodology_dir first",
|
||||
"方法学 sandbox 已构建": "Methodology sandbox built",
|
||||
"请先构建临时库": "Please build the sandbox first",
|
||||
"请输入方法学问题": "Please enter methodology question",
|
||||
"方法学 sandbox 查询失败": "Methodology sandbox query failed",
|
||||
"方法学方案已保存,可先准备数据,稍后再按该方案执行": "Methodology plan saved; you may prepare data first and execute later",
|
||||
"方法学方案保存失败": "Failed to save methodology plan",
|
||||
"当前已有批量任务在执行,请等待完成后再提交方法学流程": "A batch task is currently running; please wait for completion before submitting methodology workflow",
|
||||
"请至少选择一个可执行的 bbav5 阶段": "Please select at least one executable bbav5 stage",
|
||||
"亚家族参考物种未写入当前 job 配置,已生成方法学本地计划,请在批量确认页核对参考物种": "Subfamily reference species not written to current job config; local methodology plan generated. Please verify reference species on batch confirmation page.",
|
||||
"方法学执行规划失败": "Methodology execution planning failed",
|
||||
|
||||
// Other
|
||||
"请Input内容": "Please enter content",
|
||||
"请先选择或创建 Working Directory": "Please select or create a Working Directory first",
|
||||
"规划请求失败": "Plan request failed",
|
||||
"重新绘图功能开发中...": "Replot feature is under development...",
|
||||
|
||||
// ===== tp() keys — 使用完整中文模板作为 key =====
|
||||
"确定删除会话「{title}」吗?该操作会从列表中移除,但保留后台记录。": "Delete session \"{title}\"? This removes it from the list but keeps backend records.",
|
||||
"切换到: {title}": "Switched to: {title}",
|
||||
"预览 {rows} 行:{fields}。确认无误后点击 Apply。": "Preview {rows} rows: {fields}. Click Apply once confirmed.",
|
||||
"已应用 {rows} 行修改。": "Applied {rows} row modifications.",
|
||||
"已取消预览,表格已恢复到预览前。": "Preview cancelled, table restored.",
|
||||
"链特异性为必填:还有 {count} 个样本未填写({examples})": "Library strandness required: {count} sample(s) unfilled ({examples})",
|
||||
"设计提示:{error};将保留原流程的兼容处理。": "Design tip: {error}; keeping original compatible handling.",
|
||||
"已加载 {count} files": "Loaded {count} files",
|
||||
"加载图谱失败: {message}": "Graph load failed: {message}",
|
||||
"找到节点: {name}": "Node found: {name}",
|
||||
"读取表格失败: {reason}": "Table read failed: {reason}",
|
||||
"PPI Reset 已完成: {reset_id}": "PPI Reset complete: {reset_id}",
|
||||
"STRING 链接文件内容:{preview}": "STRING link file content: {preview}",
|
||||
"读取文件失败: {message}": "File read failed: {message}",
|
||||
"已创建新Working Directory: {job_id}": "New Working Directory created: {job_id}",
|
||||
"创建失败: {message}": "Creation failed: {message}",
|
||||
"获取Working Directory列表失败: {message}": "Failed to get Working Directory list: {message}",
|
||||
"已切换到Working Directory: {job_id}": "Switched to Working Directory: {job_id}",
|
||||
"切换失败: {message}": "Switch failed: {message}",
|
||||
"进化树图重绘失败:{message}": "Phylogeny tree replot failed: {message}",
|
||||
"Combo 图修订失败:{message}": "Combo plot revision failed: {message}",
|
||||
"任务: {message}\n状态: {status}\n进度: {progress}%": "Task: {message}\nStatus: {status}\nProgress: {progress}%",
|
||||
"停止拓展分析失败:{message}": "Failed to stop extension: {message}",
|
||||
"已选择 {count} 个原始代码文件": "{count} code file(s) selected",
|
||||
"已选择 {count} 个测试数据文件": "{count} test data file(s) selected",
|
||||
"已拖入 {count} 个文件,已自动填充拓展包": "{count} file(s) dropped; extension package auto-filled",
|
||||
"❌ {message}": "❌ {message}",
|
||||
"Upload complete. Latest Excel is now used as config.xlsx.{warningText}": "Upload complete. Latest Excel is now used as config.xlsx.{warningText}",
|
||||
"{title},需要自动注释": "{title}, Auto-annotation Required",
|
||||
|
||||
// String close species (tp with multi-line)
|
||||
"目标物种不在 STRING 固定快照中。请确认是否采用排序第一的近源物种:": "Target species not in STRING fixed snapshot. Confirm using the top-ranked close species:",
|
||||
"确认后才会下载参考 PEP、生成 Ortholog,并执行 PPI。": "Reference PEP will be downloaded, Ortholog generated, and PPI executed only after confirmation.",
|
||||
"是否切换为\"自动注释 + 继续富集\"?": "Switch to \"Auto-annotation + Continue Enrichment\"?"
|
||||
};
|
||||
|
||||
@@ -63,7 +63,7 @@ env BBAV5_QUEUE_PREFIX=plantai_dev_7908 \
|
||||
BBAV5_REDIS_URL=redis://localhost:6379/9 \
|
||||
BBAV5_CELERY_LOG_DIR=logs/celery_plantai_dev_7908 \
|
||||
BBAV5_WORKER_NAME_PREFIX=plantai_dev_7908 \
|
||||
./start_celery_7880.sh
|
||||
./start_celery_9000.sh
|
||||
|
||||
env BBAV5_QUEUE_PREFIX=plantai_dev_7908 \
|
||||
BBAV5_REDIS_URL=redis://localhost:6379/9 \
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
# PPI First, KaKs Reference Choice Implementation Plan
|
||||
|
||||
> **Working repository:** `~/remote-projects/5090/teamWorkspace/sj-2/PlantAI-dev`
|
||||
>
|
||||
> **Goal:** In the normal full gene-family workflow, show PPI and KaKs but not their internal Ortholog producers. Run PPI first. After PPI reaches a real terminal outcome, open the normal KaKs confirmation dialog and let the user choose either the PPI technical reference or the configured project reference. Keep the two mappings and manifests isolated.
|
||||
|
||||
## Task determination
|
||||
|
||||
This is a temporary execution-closure task, not the long-term KaKs reference-quality project.
|
||||
|
||||
The user-visible workflow is:
|
||||
|
||||
```text
|
||||
normal gene-family plan
|
||||
-> PPI (direct STRING or hidden PPI-owned Ortholog)
|
||||
-> KaKs reference confirmation
|
||||
-> PPI technical reference
|
||||
-> configured project reference (for the current Ft case: At)
|
||||
-> hidden KaKs-owned Ortholog only when an exact reusable mapping is absent
|
||||
-> KaKs
|
||||
```
|
||||
|
||||
The current failure has three connected causes:
|
||||
|
||||
1. The deterministic full-workflow normalizer adds `ortholog` to the visible plan, while KaKs also has a hard `ortholog` dependency.
|
||||
2. The frontend inserts one visible Ortholog stage and lets PPI and KaKs compete for that stage's owner/reference state.
|
||||
3. All Ortholog producers materialize the same `final/<prefix>.orthologs.txt` and `<prefix>.ortholog.manifest.json`; PPI-near-reference and KaKs-At runs can overwrite each other.
|
||||
|
||||
Fixing only the plan list, only the popup order, or only the mapping filename is insufficient. These three seams must change together.
|
||||
|
||||
## First-principles invariants
|
||||
|
||||
1. PPI chooses a STRING route for PPI; KaKs chooses a comparative reference for KaKs. One choice must not silently mutate the other.
|
||||
2. Ortholog remains a supported explicit analysis stage, but is an internal producer when required by PPI or KaKs.
|
||||
3. KaKs waits for PPI sequencing without declaring a hard PPI dependency. PPI failure, explicit skip, or unavailable PPI reference must still release the KaKs dialog.
|
||||
4. A PPI-owned mapping and a KaKs-owned mapping may coexist for one prefix and different references.
|
||||
5. A mapping is reusable only when target/member FASTA identity, reference identity, reference PEP checksum, scope identity, and mapping checksum match.
|
||||
6. PPI remains PEP-only. Missing KaKs CDS must never prevent PPI submission or completion.
|
||||
7. The configured reference remains unchanged for unrelated stages. A KaKs selection is written only to KaKs-owned parameters/state.
|
||||
|
||||
## Scope and assumptions
|
||||
|
||||
### In scope
|
||||
|
||||
- The normal/default full gene-family plan no longer contains implicit `ortholog`.
|
||||
- The normal plan orders PPI before KaKs.
|
||||
- KaKs has no visible/static hard dependency on Ortholog or PPI; runtime orchestration supplies its internal mapping.
|
||||
- The existing KaKs stage confirmation dialog gains two reference-source choices.
|
||||
- PPI-owned and KaKs-owned Ortholog outputs use separate namespaces.
|
||||
- Same-reference exact mapping reuse is supported.
|
||||
- PPI completion/failure/explicit skip releases KaKs; a reference download still running does not.
|
||||
- PPI/KaKs failures remain isolated from unrelated stages.
|
||||
|
||||
### Preserved behavior
|
||||
|
||||
- An explicit user request for “直系同源基因鉴定/Ortholog” still produces and displays the standalone Ortholog stage and canonical standalone outputs.
|
||||
- Direct-target STRING PPI remains direct and does not create Ortholog.
|
||||
- Existing PPI STRING species selection, ID normalization, PPI CLI, KaKs CLI, and scientific algorithms remain unchanged.
|
||||
- Methodology execution, subfamily workflow, main-Agent handoff, intent Agent, PlanningAgentV2 implementation, and analysis YAML are not redesigned.
|
||||
|
||||
### Deferred
|
||||
|
||||
- The generic high-quality KaKs reference selector using taxonomy, assembly quality, BUSCO, annotation completeness, and PEP/CDS quality.
|
||||
- Automatic ranking of multiple non-PPI KaKs candidate species.
|
||||
- Generic scheduler refactoring or frontend component extraction.
|
||||
|
||||
## Reference-choice rules
|
||||
|
||||
The KaKs dialog appears only after the PPI stage has settled for this batch.
|
||||
|
||||
| PPI outcome | PPI-reference option | Config-reference option | KaKs behavior |
|
||||
|---|---|---|---|
|
||||
| PPI direct-target STRING | Hidden; there is no separate comparative PPI reference | Show configured reference | Continue with config choice |
|
||||
| PPI reference selected and PEP/CDS/mapping exact | Enabled | Show configured reference | Reuse PPI mapping if selected |
|
||||
| PPI reference selected, CDS missing | Enabled with “需补齐 CDS” | Show configured reference | If selected, download/complete only the KaKs bundle, then continue |
|
||||
| PPI reference selected, mapping missing/stale | Enabled if PEP/CDS can be prepared | Show configured reference | Run hidden KaKs-owned Ortholog |
|
||||
| PPI failed or was skipped | Show only if a separate PPI reference and usable files exist | Show configured reference | PPI outcome does not block confirmation |
|
||||
| Neither choice has PEP+CDS | Disabled with exact missing-file reasons | Disabled with exact missing-file reasons | Pause only KaKs |
|
||||
|
||||
The reference choice is explicit. No silent fallback from one option to the other.
|
||||
|
||||
## Task 1 — Freeze the visible planning contract
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `apps/web/vue3/main.py`
|
||||
- Modify: `gfa_core/mas/stage_catalog.py`
|
||||
- Modify: `apps/web/vue3/index.html`
|
||||
- Modify: `tests/unit/test_task_status_attachment.py`
|
||||
- Modify: `tests/unit/test_subfamily_prediction_plan.py` or add one focused full-workflow plan test
|
||||
- Modify: `tests/unit/test_ppi_frontend_display_contract.py`
|
||||
|
||||
### Steps
|
||||
|
||||
1. Add failing tests for the normal “执行基因家族分析” response:
|
||||
- `ortholog` is absent from `plan.stages` and `stages_detail`;
|
||||
- `ppi` and `kaks` are present;
|
||||
- `ppi` appears before `kaks`;
|
||||
- standalone explicit Ortholog remains supported.
|
||||
2. Remove implicit Ortholog from `_downstream_gene_family_stages_for_context()` and `FULL_GENE_FAMILY_WORKFLOW_STAGES`.
|
||||
3. Change the normal static KaKs dependency from `identify + ortholog` to `identify` in the normal planning catalog, plan-detail builder, and frontend dependency table.
|
||||
4. Keep the standalone Ortholog catalog entry and explicit Ortholog intent unchanged.
|
||||
5. Do not filter Ortholog out in the rendered frontend. The backend plan itself must be correct.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- A normal full plan displays PPI and KaKs, never implicit Ortholog.
|
||||
- An explicit Ortholog request still displays Ortholog.
|
||||
- No frontend-only post-processing is used to disguise a wrong backend plan.
|
||||
|
||||
## Task 2 — Give internal mappings owner-specific identities
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `gfa_core/ortholog_materialize.py`
|
||||
- Modify: `apps/web/pipeline_backend.py`
|
||||
- Modify: `apps/web/vue3/main.py` only for the narrow execution-request fields needed to carry mapping ownership
|
||||
- Modify: `tests/unit/test_ppi_ortholog_owner_identity_task5b.py`
|
||||
- Modify: `tests/unit/test_pipeline_backend_execution_env.py`
|
||||
|
||||
### Output contract
|
||||
|
||||
Use one path helper rather than repeating filename construction:
|
||||
|
||||
```text
|
||||
standalone: final/<prefix>.orthologs.txt
|
||||
PPI owner: final/<prefix>.ppi.orthologs.txt
|
||||
KaKs owner: final/<prefix>.kaks.orthologs.txt
|
||||
```
|
||||
|
||||
Apply the same owner namespace to pairs, raw pairs, BLAST output, sidecar, and Ortholog manifest. Keep `manifest.stage = "ortholog"`; record `dependency_owner` and the effective mapping identity in parameters/provenance.
|
||||
|
||||
### Steps
|
||||
|
||||
1. Add failing coexistence tests that materialize PPI-owned Spinacia and KaKs-owned At mappings for the same prefix and assert both remain unchanged.
|
||||
2. Extend `materialize_ortholog_final_outputs()` with a narrow owner/output namespace. Empty owner preserves every current standalone filename.
|
||||
3. In `run_stage("ortholog")`, accept only `""`, `"ppi"`, or `"kaks"` as mapping owner and pass it to materialization.
|
||||
4. Centralize owner-aware candidate paths for preflight and execution. Do not let PPI or KaKs fall through to a different owner's mapping merely because that file exists.
|
||||
5. Add a narrow KaKs mapping-source owner field to `ExecuteRequest` and backend forwarding. Prefer an owner token over a client-supplied arbitrary filesystem path.
|
||||
6. Validate reuse against the existing Ortholog identity sidecar:
|
||||
- selected reference storage/species identity;
|
||||
- target/query FASTA identity;
|
||||
- member identity;
|
||||
- reference PEP checksum;
|
||||
- scope identity;
|
||||
- mapping checksum.
|
||||
7. Preserve canonical filenames and resolution for explicit standalone Ortholog.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- PPI Spinacia and KaKs At mappings coexist.
|
||||
- Rerunning one owner does not change the other owner's file or Manifest checksum.
|
||||
- A stale/wrong-reference mapping is rejected rather than silently reused.
|
||||
- Existing standalone Ortholog tests still pass.
|
||||
|
||||
## Task 3 — Sequence KaKs after PPI without a hard dependency
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `apps/web/vue3/index.html`
|
||||
- Modify: `tests/unit/test_ppi_owner_scheduling_task5.py`
|
||||
- Modify: `tests/unit/test_ppi_frontend_display_contract.py`
|
||||
|
||||
### Steps
|
||||
|
||||
1. Add scheduler tests for a batch containing both PPI and KaKs:
|
||||
- KaKs is not submitted while PPI is pending, running, or paused for an active reference download;
|
||||
- PPI completed releases KaKs;
|
||||
- PPI failed releases KaKs;
|
||||
- PPI explicitly skipped releases KaKs;
|
||||
- unrelated stages continue while PPI waits.
|
||||
2. Add one narrow soft sequencing state to the existing batch state. Do not encode PPI as a KaKs dependency in `stageDependencies`.
|
||||
3. Release the gate only on a real PPI terminal outcome. Generic `paused` is not sufficient because an active background reference download is resumable.
|
||||
4. Remove the path that inserts/replaces a visible `ortholog` item in `batchExecuteStages` for PPI/KaKs ownership.
|
||||
5. Reuse the existing direct internal producer submission helper, but track its result under the owner stage rather than `stageExecutionStatus.ortholog`.
|
||||
6. Ensure PPI internal-producer failure pauses/fails only PPI and still releases KaKs confirmation when terminal.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- There is no Ortholog row or Ortholog confirmation popup in the normal batch.
|
||||
- KaKs confirmation cannot race ahead of PPI.
|
||||
- PPI failure cannot cause KaKs to be dependency-failed.
|
||||
- Other unrelated stages are not serialized behind PPI.
|
||||
|
||||
## Task 4 — Add the KaKs reference choice to the existing dialog
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `apps/web/vue3/index.html`
|
||||
- Modify: `apps/web/vue3/main.py` only if a read-only candidate-readiness payload is needed
|
||||
- Modify: `apps/web/pipeline_backend.py` only to return owner-aware readiness/identity data
|
||||
- Modify: `tests/unit/test_ppi_frontend_display_contract.py`
|
||||
- Modify: `tests/unit/test_ppi_snapshot_preflight_task3.py`
|
||||
|
||||
### Steps
|
||||
|
||||
1. Reuse the existing stage-confirmation dialog and its PPI reference card styles. Do not create a second visual system or a synthetic visible stage.
|
||||
2. When opening the KaKs confirmation dialog, build two candidates:
|
||||
- `ppi_reference`: the separate PPI technical reference, only when one exists;
|
||||
- `project_reference`: the configured reference from the confirmed project scope.
|
||||
3. For each candidate, show species label/code, source, PEP status, CDS status, mapping status, and the reason when unavailable.
|
||||
4. Store the confirmed choice only in KaKs stage params, for example `kaksReferenceSource`, `referenceSpecies/refSpeciesList`, and `orthologMappingOwner`. Do not update PPI owner fields or unrelated plan-level reference fields.
|
||||
5. If the PPI candidate lacks CDS, use the existing reference-bundle endpoint with `requestedStages=["kaks"]` to complete that selected species. PPI remains completed and is never reopened or downgraded.
|
||||
6. If the selected reference lacks an exact mapping:
|
||||
- submit a hidden `dependencyOwner="kaks"` Ortholog producer;
|
||||
- on success, submit KaKs with `orthologMappingOwner="kaks"`;
|
||||
- if the exact PPI-owned mapping is reusable, submit KaKs with `orthologMappingOwner="ppi"` and do not recompute.
|
||||
7. If neither candidate can be made ready, pause only KaKs and retain a retryable, explicit reason.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- For the Ft case, the dialog can show the PPI near-reference and configured At as separate choices.
|
||||
- Selecting At cannot change PPI's reference TaxID, mapping, Manifest, or artifacts.
|
||||
- Selecting the PPI reference cannot change config/project reference for other stages.
|
||||
- A missing CDS or mapping produces a KaKs-only action/pause.
|
||||
|
||||
## Task 5 — Synchronize the minimum contracts
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `gfa_core/schemas/stages/ppi_contract_v1.md`
|
||||
- Modify: `gfa_core/schemas/stages/kaks_contract_v1.md`
|
||||
- Modify: `gfa_core/schemas/stages/ortholog_contract_v1.md`
|
||||
|
||||
### Steps
|
||||
|
||||
1. Document PPI-owned and KaKs-owned internal Ortholog identities and standalone Ortholog preservation.
|
||||
2. Document that PPI and KaKs readiness/reference state are independent.
|
||||
3. Document the KaKs reference-source choice and exact mapping-reuse rule.
|
||||
4. Document soft sequencing: PPI terminal outcome releases KaKs, but PPI success is not a KaKs prerequisite.
|
||||
|
||||
Do not change CLI inputs or the scientific method contracts beyond these execution/provenance facts.
|
||||
|
||||
## Task 6 — Verification without polluting the user's real case
|
||||
|
||||
### Automated tests
|
||||
|
||||
Run the focused tests first:
|
||||
|
||||
```bash
|
||||
/home/xia-lab/miniconda3/envs/test4py311/bin/python -m pytest -q \
|
||||
tests/unit/test_task_status_attachment.py \
|
||||
tests/unit/test_ppi_frontend_display_contract.py \
|
||||
tests/unit/test_ppi_owner_scheduling_task5.py \
|
||||
tests/unit/test_ppi_ortholog_owner_identity_task5b.py \
|
||||
tests/unit/test_ppi_snapshot_preflight_task3.py \
|
||||
tests/unit/test_pipeline_backend_execution_env.py -k 'ppi or kaks or ortholog or full_workflow'
|
||||
```
|
||||
|
||||
Then run the related gene-family planning/API suite discovered from failures. Do not broaden to unrelated WGS/RNA-seq work unless a shared contract test fails.
|
||||
|
||||
### Isolated real-data matrix
|
||||
|
||||
Create a dedicated test job ID/folder. Do not write PPI, KaKs, or Ortholog outputs into `job_1786184894919_b7d741/final` or any user acceptance job.
|
||||
|
||||
Validate these routes:
|
||||
|
||||
1. **Ft + PPI near reference + KaKs same reference**
|
||||
- visible plan has PPI/KaKs and no Ortholog;
|
||||
- PPI-owned mapping succeeds;
|
||||
- KaKs dialog appears after PPI;
|
||||
- choose PPI reference;
|
||||
- exact mapping is reused;
|
||||
- PPI and KaKs succeed.
|
||||
2. **Ft + PPI near reference + KaKs At**
|
||||
- choose configured At;
|
||||
- KaKs-owned At mapping is generated;
|
||||
- PPI near-reference mapping and Manifest checksums remain unchanged;
|
||||
- KaKs succeeds with At PEP/CDS.
|
||||
3. **PPI failure/skip isolation**
|
||||
- force PPI to fail or explicitly skip it;
|
||||
- KaKs dialog still appears;
|
||||
- choose At and complete KaKs;
|
||||
- unrelated runnable stages keep their own statuses.
|
||||
4. **PPI active download gate**
|
||||
- while PPI reference download is running, KaKs dialog does not open;
|
||||
- after PPI settles, it opens once.
|
||||
5. **Standalone Ortholog regression**
|
||||
- explicitly request Ortholog;
|
||||
- visible standalone stage and canonical outputs remain available.
|
||||
|
||||
For each route capture request payloads, task statuses, owner-specific Manifests, mapping checksums, and final stage statuses in the dedicated test folder.
|
||||
|
||||
### Live service verification
|
||||
|
||||
After unit/API verification, restart only the sj-2 service on port 9016. Verify the served HTML contains the new contract, then exercise one isolated frontend job. Do not restart unrelated workers/services unless backend code loading proves it is necessary.
|
||||
|
||||
## Stop rules
|
||||
|
||||
Stop and re-diagnose if implementation requires any of the following:
|
||||
|
||||
- changing PPI/STRING ID mapping or network analysis logic;
|
||||
- changing KaKs calculation logic or neutral-rate semantics;
|
||||
- modifying main-Agent/intent-Agent prompts or analysis YAML;
|
||||
- making PPI wait for CDS;
|
||||
- writing PPI reference into shared project reference fields;
|
||||
- overwriting another owner's mapping/Manifest;
|
||||
- hiding Ortholog only by frontend display filtering while it remains in the normal backend plan.
|
||||
|
||||
## Final acceptance criteria
|
||||
|
||||
- Normal full gene-family planning shows PPI and KaKs, not implicit Ortholog.
|
||||
- PPI is processed before KaKs reference confirmation.
|
||||
- KaKs offers the PPI technical reference and configured reference when valid.
|
||||
- PPI success is not required for KaKs to continue.
|
||||
- PPI and KaKs mappings/manifests are owner-isolated and reproducible.
|
||||
- At selection for KaKs does not mutate PPI; PPI-near-reference selection does not mutate unrelated stages.
|
||||
- No original user job is polluted by verification outputs.
|
||||
@@ -0,0 +1,252 @@
|
||||
# PPI 专属 Ortholog 参考 PEP 身份一致性修订计划
|
||||
|
||||
> **执行约束:** 实施时必须继续遵循 `karpathy-guidelines`:先写可失败的回归测试,只做最小生产改动,每一步都用可观察证据验收;不得顺手重构前端、规划层、下载层或身份校验层。
|
||||
|
||||
**目标:** 当 PPI 使用已下载近源物种或用户显式确认的 At 作为技术参考物种时,PPI 专属 Ortholog 的生产端与 PPI 二次预检必须使用同一份全蛋白组 PEP,使 Ortholog → PPI → Ka/Ks 能继续调度,同时保持普通独立 Ortholog 的原有 family PEP 语义不变。
|
||||
|
||||
**架构边界:** PPI 的技术参考物种只属于 `dependency_owner=ppi` 的隐藏依赖,不进入公共规划阶段,也不改变 `target_only` 生物学 scope。修复点仅位于后端 Ortholog 输入解析分支;严格的路径、SHA256、taxon、storage key、mapping identity 校验继续保留。
|
||||
|
||||
**技术栈:** Python 3.11、FastAPI、Celery、pytest;真实验证使用 sj-2 的 9016 Web/API 和对应 Celery workers。
|
||||
|
||||
---
|
||||
|
||||
## 1. 已确认的问题
|
||||
|
||||
### 1.1 当前真实失败链
|
||||
|
||||
真实任务 `job_1786243816439_04955b` 已给出完整证据链:
|
||||
|
||||
1. 菠菜参考包下载尝试 3 次后因 `IncompleteRead(488265 bytes read)` 失败;重试机制工作正常。
|
||||
2. 用户显式选择 At 兜底后,PPI 专属 Ortholog 成功生成 24 对映射。
|
||||
3. 随后的 PPI 二次预检返回 `stale_ortholog_identity_conflict`,PPI 未提交。
|
||||
4. Ka/Ks 正确识别同一参考准备链,未再次启动下载,但因 PPI 映射未通过身份校验而暂停。
|
||||
|
||||
### 1.2 根因不是“身份校验太严格”,而是生产者与消费者解析了不同文件
|
||||
|
||||
| 环节 | 实际解析文件 | 结果 |
|
||||
|---|---|---|
|
||||
| PPI 专属 Ortholog 生产端 | `input/.../At.SPL.pep.fasta` | family PEP,9,929 bytes,SHA256 `42dca87b...` |
|
||||
| PPI 二次预检消费端 | `input/.../At/At.pep.fasta` | whole-proteome PEP,34,646,980 bytes,SHA256 `193f9d45...` |
|
||||
|
||||
两者虽然都叫 At,但不是同一个输入身份。Ortholog Manifest 和 sidecar 如实记录了 family PEP;PPI 预检按全蛋白组 PEP 构建期望身份,因此拒绝复用是正确行为。
|
||||
|
||||
### 1.3 精确代码位置
|
||||
|
||||
生产端位于 `apps/web/pipeline_backend.py` 的 `stage == "ortholog"` 分支:
|
||||
|
||||
```python
|
||||
subject_fasta = first_existing_scope_file(
|
||||
ortholog_scope_data,
|
||||
reference_species,
|
||||
["family_pep", "pep"],
|
||||
)
|
||||
if not subject_fasta:
|
||||
subject_fasta = resolve_species_family_pep(...)
|
||||
if not subject_fasta:
|
||||
subject_fasta = resolve_species_sequence_file(..., family_only=False)
|
||||
```
|
||||
|
||||
即使 `ppi_owned_producer=True`,它仍优先选 family PEP。
|
||||
|
||||
消费端 `_resolve_ppi_only_snapshot_preflight()` 经 `_ppi_mapping_identity_matches_current_inputs()` 使用 `resolve_species_sequence_file(..., family_only=False)`,解析的是技术参考物种的 whole-proteome PEP。
|
||||
|
||||
### 1.4 最近三轮修订与本次失败的关系
|
||||
|
||||
| 修订轮次 | 已产生的变化 | 当前结论 |
|
||||
|---|---|---|
|
||||
| PPI owner/scope 修订 | 隐藏 Ortholog 归 PPI 所有;公共 scope 保持 `target_only`;增加严格身份校验和二次预检 | 设计正确,必须保留 |
|
||||
| 下载进度展示修订 | Tip 旁显示下载任务、字节数和终态 | 只增强可见性,没有制造数据链问题 |
|
||||
| PEP+CDS 一次下载修订 | PPI 阶段一次准备 PEP+CDS;`IncompleteRead` 最多重试 3 次;Ka/Ks 不再重复下载 | 当前日志证明已生效;它暴露了此前项目记录中已暂缓的 PEP 身份不一致 |
|
||||
|
||||
因此,本次不能回退上述三轮修订,也不能通过删除严格身份校验“让它通过”。
|
||||
|
||||
## 2. 冻结的最小修订方案
|
||||
|
||||
### 2.1 唯一生产代码改动
|
||||
|
||||
**文件:** `apps/web/pipeline_backend.py`
|
||||
|
||||
在 `stage == "ortholog"` 的 subject 解析处,仅对 `ppi_owned_producer` 分流:
|
||||
|
||||
```python
|
||||
if ppi_owned_producer:
|
||||
subject_fasta = resolve_species_sequence_file(
|
||||
ws,
|
||||
final_dir,
|
||||
in_dir,
|
||||
ortholog_scope_data,
|
||||
reference_species,
|
||||
family,
|
||||
"pep",
|
||||
family_only=False,
|
||||
)
|
||||
else:
|
||||
subject_fasta = first_existing_scope_file(
|
||||
ortholog_scope_data,
|
||||
reference_species,
|
||||
["family_pep", "pep"],
|
||||
)
|
||||
if not subject_fasta:
|
||||
subject_fasta = resolve_species_family_pep(
|
||||
ws, final_dir, in_dir, reference_species, family
|
||||
)
|
||||
if not subject_fasta:
|
||||
subject_fasta = resolve_species_sequence_file(
|
||||
ws,
|
||||
final_dir,
|
||||
in_dir,
|
||||
ortholog_scope_data,
|
||||
reference_species,
|
||||
family,
|
||||
"pep",
|
||||
family_only=False,
|
||||
)
|
||||
```
|
||||
|
||||
该分流保证:
|
||||
|
||||
- PPI 专属 Ortholog 与 PPI 预检调用同一个 whole-proteome resolver;
|
||||
- 菠菜缓存参考和 At 兜底都走同一规则,而不是写死 Ft、菠菜或 At;
|
||||
- 普通独立 Ortholog 仍优先使用 family PEP,不改变既有科研语义;
|
||||
- Ka/Ks 继续复用 PPI 已确认的 bundle/mapping owner,不新增第二次 Ortholog。
|
||||
|
||||
### 2.2 明确禁止的改动
|
||||
|
||||
本次不得修改:
|
||||
|
||||
- `apps/web/vue3/index.html` 的规划、调度、弹窗和进度状态机;
|
||||
- `apps/web/vue3/main.py` 的接口契约和下载任务实现;
|
||||
- PlanningAgentV2、主 Agent、意图 Agent 或任何 YAML 阶段定义;
|
||||
- `gfa_core/ortholog_materialize.py` 的严格 identity 计算与比较;
|
||||
- `stage_context.reference_species`、公共 `scope_id` 或 PPI 的 `target_only` 语义;
|
||||
- PEP+CDS 一次下载、3 次重试、Ka/Ks 禁止二次下载的现有逻辑;
|
||||
- 全局调整 `first_existing_scope_file(["family_pep", "pep"])` 的优先级。
|
||||
|
||||
## 3. 实施任务
|
||||
|
||||
### Task 1:先建立能复现真实冲突的 RED 测试
|
||||
|
||||
**文件:** `tests/unit/test_ppi_ortholog_owner_identity_task5b.py`
|
||||
|
||||
- [ ] 增加 At 同时存在两套 PEP 的 fixture:
|
||||
- `input/At.SPL.pep.fasta`:family PEP;
|
||||
- `input/At/At.pep.fasta`:whole-proteome PEP;
|
||||
- `input/At/At.cds.fasta`:与 whole PEP 使用相同 transcript ID,如 `AT5G16970.1`。
|
||||
- [ ] 以 `dependency_owner="ppi"`、`ppi_reference_storage_key="At"`、`ppi_reference_taxon_id="3702"` 执行隐藏 Ortholog。
|
||||
- [ ] 断言当前代码的 `--subject` 错误指向 `At.SPL.pep.fasta`,并使“必须指向 `At/At.pep.fasta`”的新断言先失败。
|
||||
- [ ] 断言生成映射后再次调用 `resolve_ppi_species_resolution()`;当前代码应因 reference PEP SHA 不同而不是 `ready`。
|
||||
|
||||
运行:
|
||||
|
||||
```bash
|
||||
/home/xia-lab/miniconda3/envs/test4py311/bin/python -m pytest -q \
|
||||
tests/unit/test_ppi_ortholog_owner_identity_task5b.py \
|
||||
-k "whole_reference_pep"
|
||||
```
|
||||
|
||||
预期 RED:失败信息明确显示 expected `At/At.pep.fasta`,actual `At.SPL.pep.fasta`;不得用其他异常作为“红灯”。
|
||||
|
||||
### Task 2:实施 PPI owner 专属的 resolver 分流
|
||||
|
||||
**文件:** `apps/web/pipeline_backend.py`
|
||||
|
||||
- [ ] 按 2.1 的代码结构修改 subject 解析。
|
||||
- [ ] 不改变 `references=[ppi_owner_storage_key]`、`_effective_ppi_owner_scope_id()`、`member_set_id=""`、owner-specific 输出命名。
|
||||
- [ ] 保留找不到 whole PEP 时的明确失败,不允许静默回退到 family PEP;否则生产端和预检会再次形成双重真相。
|
||||
- [ ] 运行语法检查:
|
||||
|
||||
```bash
|
||||
/home/xia-lab/miniconda3/envs/test4py311/bin/python -m py_compile \
|
||||
apps/web/pipeline_backend.py
|
||||
```
|
||||
|
||||
### Task 3:补齐身份与非回归测试
|
||||
|
||||
**文件:** `tests/unit/test_ppi_ortholog_owner_identity_task5b.py`
|
||||
|
||||
- [ ] GREEN 断言 PPI-owned Ortholog 命令的 `--subject` 精确等于 whole-proteome PEP。
|
||||
- [ ] 断言 PPI owner sidecar 与 Manifest 的 `reference_pep_sha256`、`reference_storage_key`、`reference_taxon_id`、`mapping_path` 全部对应同一 whole PEP。
|
||||
- [ ] 断言二次 PPI preflight 返回 `status == "ready"`,不再出现 `stale_ortholog_identity_conflict`。
|
||||
- [ ] 断言 PPI Manifest 仍为 `scope_mode == "target_only"`,`stage_context.reference_species == []`,`selected_species == ["Ft"]`。
|
||||
- [ ] 新增普通独立 Ortholog 回归用例:当 family PEP 与 whole PEP 同时存在时,非 PPI owner 仍选择 family PEP。
|
||||
- [ ] 为直接相关的旧 fixture 补充 CDS,使其进入身份校验分支;不改与本问题无关的前端文本断言或 effective-member fixture。
|
||||
|
||||
运行:
|
||||
|
||||
```bash
|
||||
/home/xia-lab/miniconda3/envs/test4py311/bin/python -m pytest -q \
|
||||
tests/unit/test_ppi_ortholog_owner_identity_task5b.py
|
||||
|
||||
/home/xia-lab/miniconda3/envs/test4py311/bin/python -m pytest -q \
|
||||
tests/unit/test_ppi_ortholog_owner_identity_task5b.py \
|
||||
tests/unit/test_ppi_owner_scheduling_task5.py \
|
||||
tests/unit/test_ppi_frontend_display_contract.py
|
||||
```
|
||||
|
||||
验收时必须把修订前基线与修订后结果分开报告。已知旧前端字符串断言、旧 fixture 缺 CDS、effective-member 相关失败不得被伪装为本次新增回归,也不得借机修改生产代码。
|
||||
|
||||
### Task 4:在独立验证 job 做真实 API 闭环
|
||||
|
||||
**位置:** sj-2 独立创建 `job_codex_ppi_kaks_pep_identity_<timestamp>`,不得把验证产物写入用户当前 job 的 `final/`。
|
||||
|
||||
- [ ] 从真实 Ft/At 案例只读复用以下输入:Ft 家族成员 PEP、At family PEP、At whole PEP、At whole CDS、scope/config。
|
||||
- [ ] 初次调用 `POST /api/ppi/species-resolution`,使用:
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": "job_codex_ppi_kaks_pep_identity_<timestamp>",
|
||||
"prefix": "Ft.SPL",
|
||||
"targetSpecies": ["Ft"],
|
||||
"referenceSpecies": [],
|
||||
"geneFamily": "SPL",
|
||||
"ppiReferenceTaxonId": "3702",
|
||||
"ppiReferenceStorageKey": "At",
|
||||
"ppiReferenceSelectionMode": "explicit_fallback",
|
||||
"ppiReferenceUserConfirmed": true,
|
||||
"dependencyOwner": "ppi",
|
||||
"requestedStages": ["ppi"]
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] 初次预检必须为 `needs_ortholog`,且不再要求下载完整 At bundle。
|
||||
- [ ] 调用 `POST /api/execute` 执行隐藏 `ortholog`,请求保留 `dependencyOwner="ppi"`、`targetSpecies=["Ft"]`、`referenceSpecies=[]` 和 PPI owner 字段。
|
||||
- [ ] 轮询任务直至成功,并检查后端命令日志:`--query` 为 Ft family PEP,`--subject` 为 `At/At.pep.fasta`。
|
||||
- [ ] 检查 PPI-owned Ortholog 产生有效映射、sidecar、Manifest,映射数量大于 0;所有 reference identity 字段一致。
|
||||
- [ ] 再次调用 `/api/ppi/species-resolution`,必须返回 `ready`,不得返回 stale identity conflict。
|
||||
- [ ] 调用 `/api/execute` 执行 PPI,轮询到 terminal success;检查 PPI Manifest 和网络产物,不以“任务已提交”代替成功。
|
||||
- [ ] 进入 Ka/Ks 确认路径,确认它复用 PPI 选择的参考 bundle 和 `orthologMappingOwner=ppi`;不得出现第二次参考下载或第二个 Ortholog producer。
|
||||
- [ ] 执行 Ka/Ks 至 terminal success,并核对 Ka/Ks Manifest 的 reference/mapping provenance。
|
||||
- [ ] 最后确认活动任务数为 0、前端计时漏斗停止、PPI/KaKs 的 terminal 状态能正常显示。
|
||||
|
||||
### Task 5:覆盖近源物种路径并区分外部下载故障
|
||||
|
||||
- [ ] 若 `string_taxon_3562` 已有完整且非损坏的 PEP+CDS bundle,在独立验证 job 重复 Task 4,要求 PPI-owned Ortholog 的 `--subject` 为 `string_taxon_3562` whole PEP,二次预检为 `ready`。
|
||||
- [ ] 若菠菜下载仍因远端 `IncompleteRead` 失败,只验收重试次数、字节进度、terminal failure 和 At 手动兜底入口;明确记录为外部下载失败,不得把它误判为本次 identity 修订失败。
|
||||
- [ ] 不为通过测试而在代码中预设 Ft、菠菜或 At;两条路径必须由同一 storage-key + whole-PEP 解析规则驱动。
|
||||
|
||||
### Task 6:部署 sj-2 并做用户前端验收
|
||||
|
||||
- [ ] 确认修改发生在 `/home/xzq/5090/teamWorkspace/sj-2/PlantAI-dev`,不是 sj 或主项目。
|
||||
- [ ] 重启 sj-2 的 9016 Web 服务及该项目全部 Celery workers;启动时显式使用远端 Conda 根 `/home/xzq/miniconda3`。
|
||||
- [ ] 检查 9016 健康接口、Web PID、Celery PID 和日志所指项目根均为 sj-2。
|
||||
- [ ] 用户前端验证前,只清理独立验证 job;不得删除用户任务或公共 data。
|
||||
- [ ] 用户用新 job 验收以下两条路径:
|
||||
1. 近源参考包可用:一次准备 PEP+CDS → 一个隐藏 PPI-owned Ortholog → PPI → Ka/Ks 确认与复用;
|
||||
2. 近源下载失败:显示进度和失败 → 用户确认 At → 一个隐藏 PPI-owned Ortholog → PPI → Ka/Ks 确认与复用。
|
||||
|
||||
## 4. 完成判据
|
||||
|
||||
只有同时满足以下条件,才能宣称修复完成:
|
||||
|
||||
- PPI-owned Ortholog 的实际 `--subject` 与 PPI 预检解析出的 whole PEP 是同一路径、同一 SHA256;
|
||||
- Ortholog Manifest、identity sidecar、PPI preflight 的 taxon/storage/path/hash/mapping identity 完全一致;
|
||||
- PPI 二次预检为 `ready`,PPI 真实执行成功并生成有效产物;
|
||||
- Ka/Ks 在 PPI 后出现确认,复用已准备的 PEP+CDS 和 PPI mapping owner,不二次下载、不二次生成 Ortholog;
|
||||
- 公共规划中不出现 Ortholog,PPI 公共 scope 仍为 Ft `target_only`;
|
||||
- 独立 Ortholog 保持 family PEP 优先,既有分析语义不变;
|
||||
- 前端无无限旋转,所有 task 到达 terminal 状态;
|
||||
- 真实验证产物只存在独立测试目录,不污染用户 job。
|
||||
|
||||
## 5. 回滚边界
|
||||
|
||||
若新回归测试证明 `resolve_species_sequence_file(..., family_only=False)` 在某种 storage key 下不能稳定解析 whole PEP,只回滚本次单一分流并保留 RED 证据;不得回滚 owner/scope、严格 identity、PEP+CDS 一次下载或前端进度显示。此时应先补清 resolver 输入契约,再重新评估,不得用 family PEP 或禁用 SHA 校验作为临时通行证。
|
||||
@@ -0,0 +1,43 @@
|
||||
# PPI/KaKs Integration Commit and Push Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Commit the validated PPI → owner-specific Ortholog → Ka/Ks workflow and push it to the main PlantAI repository without including runtime data.
|
||||
|
||||
**Architecture:** Preserve the tested cross-layer change as one functional integration: frontend orchestration, API/backend identity resolution, owner-namespaced Ortholog materialization, contracts, and their tests. Keep input, workspace, downloaded bundles, logs, caches, and shared data outside Git.
|
||||
|
||||
**Tech Stack:** Vue 3 single-page frontend, FastAPI, Celery, Python, pytest, Git.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Push must be a normal fast-forward update; never force-push.
|
||||
- Do not stage `input/`, `workspace/`, `final/`, downloaded reference bundles, logs, PID files, caches, or shared `data` content.
|
||||
- Preserve standalone, PPI-owner, and KaKs-owner Ortholog namespaces.
|
||||
- Verify the relevant regression suite before committing and re-check the committed tree before pushing.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Clean the commit boundary
|
||||
|
||||
**Files:** All currently modified tracked source, contract, and test files; the three PPI/KaKs plan/spec documents.
|
||||
|
||||
- [ ] Remove whitespace errors without changing runtime behavior.
|
||||
- [ ] Update stale source-contract assertions to the current unified dialog and producer variable names.
|
||||
- [ ] Confirm no runtime outputs are staged.
|
||||
|
||||
### Task 2: Verify the integrated workflow
|
||||
|
||||
**Files:** `tests/unit/test_ppi*.py`, reference-bundle tests, handoff/snapshot tests, and the affected production files.
|
||||
|
||||
- [ ] Run JavaScript syntax validation for `apps/web/vue3/index.html`.
|
||||
- [ ] Run the focused PPI/KaKs, reference download, owner identity, and context regression tests.
|
||||
- [ ] Run `git diff --check` and inspect the final staged file list.
|
||||
|
||||
### Task 3: Commit and push
|
||||
|
||||
**Files:** Only the verified source, contracts, tests, and documentation from Tasks 1–2.
|
||||
|
||||
- [ ] Fetch `origin/master` and verify the push remains fast-forward.
|
||||
- [ ] Commit with a PPI/KaKs integration message.
|
||||
- [ ] Push the committed branch to `origin/master` without force.
|
||||
- [ ] Verify the remote master commit ID equals the local committed ID.
|
||||
@@ -0,0 +1,383 @@
|
||||
# PPI/KaKs Reference Route Handoff Implementation Plan
|
||||
|
||||
**Goal:** 修复 PPI 完成后 Ka/Ks 的两条参考路线,使用户确认的参考物种与 Ortholog owner 原样传递到预检和执行,不发生二次推导、重复下载或错误暂停。
|
||||
|
||||
**Architecture:** 保留现有规划、后端 API、CLI 和 owner 专属 Manifest 契约。PlanningAgent 只规划 PPI 与 Ka/Ks;前端收集并传递用户选择;后端 resolver 只验证所选路线;隐藏 Ortholog producer 只在所选 owner 缺少映射时执行。
|
||||
|
||||
**Tech Stack:** Vue 3 内嵌脚本、FastAPI、pytest、Node 动态行为测试、Celery、现有 `/api/ppi/species-resolution` 与 `/api/execute`。
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- 修订项目固定为 `/home/xzq/5090/teamWorkspace/sj-2/PlantAI-dev`,服务端口固定为 `9016`。
|
||||
- 不修改 MainAgent、PlanningAgent、阶段目录、CLI、分析算法、Manifest Schema 或后端 API 契约。
|
||||
- 规划继续只显示 PPI 和 Ka/Ks,不显示 Ortholog。
|
||||
- PPI 技术参考选择只属于 PPI;Ka/Ks 必须在 PPI settled 后单独确认 `reuse PPI` 或 `project reference`。
|
||||
- 用户确认是路线权威来源;前端不得在确认后从 batch/plan 全局状态重新推导另一条路线。
|
||||
- 真实验证使用隔离 job 与独立输出目录,不写入用户 job 的 `final/`。
|
||||
- 不实现跨 owner 哈希复用、页面刷新恢复、统一状态机、断点续传或其它扩展。
|
||||
- 完成修订和验证后再重启 sj-2 Web 与全部 Celery worker;不自动 commit/push。
|
||||
|
||||
---
|
||||
|
||||
### Task 1: 用精确行为测试锁定两条路线
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/unit/test_ppi_owner_scheduling_task5.py`
|
||||
- Inspect only: `tests/unit/test_ppi_frontend_display_contract.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `kaksReferenceChoiceParams`、`buildPpiKaksDependencyRequest`、`ppiNeedsNearReferenceDecision`、`prepareKaksSelectedReference`。
|
||||
- Produces: 精确复现 PPI 映射复用、At 独立映射、确认参数直传和 PPI-only 近源规则的 RED 测试。
|
||||
|
||||
- [ ] **Step 1: 添加 PPI 近源复用路线测试**
|
||||
|
||||
构造以下确认结果和 PPI owner 状态:
|
||||
|
||||
```javascript
|
||||
const confirmed = kaksReferenceChoiceParams('ppi', 'string_taxon_3562', 'At');
|
||||
const ppiOwner = {
|
||||
ppiReferenceTaxonId: '3562',
|
||||
ppiReferenceStorageKey: 'string_taxon_3562',
|
||||
ppiReferenceSelectionMode: 'tree_nearest',
|
||||
ppiReferenceUserConfirmed: true
|
||||
};
|
||||
```
|
||||
|
||||
断言 `buildPpiKaksDependencyRequest(['kaks'], confirmed)` 的关键结果为:
|
||||
|
||||
```javascript
|
||||
{
|
||||
requestedStages: ['kaks'],
|
||||
dependencyOwner: 'ppi',
|
||||
referenceSpecies: ['string_taxon_3562'],
|
||||
ppiReferenceTaxonId: '3562',
|
||||
ppiReferenceStorageKey: 'string_taxon_3562',
|
||||
ppiReferenceSelectionMode: 'tree_nearest',
|
||||
ppiReferenceUserConfirmed: true
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 添加项目 At 路线测试**
|
||||
|
||||
```javascript
|
||||
const confirmed = kaksReferenceChoiceParams('project', 'string_taxon_3562', 'At');
|
||||
```
|
||||
|
||||
断言 `dependencyOwner='kaks'`、`referenceSpecies=['At']`,且不携带菠菜 PPI owner 身份。
|
||||
|
||||
- [ ] **Step 3: 添加 PPI-only 近源规则测试**
|
||||
|
||||
```javascript
|
||||
const resolution = {
|
||||
status: 'needs_ortholog',
|
||||
selected_string_species: 'ath',
|
||||
diagnostics: {
|
||||
reference_recommendations: [{species_code: 'string_taxon_3562'}]
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
断言:
|
||||
|
||||
```javascript
|
||||
ppiNeedsNearReferenceDecision(resolution, ['ppi']) === true;
|
||||
ppiNeedsNearReferenceDecision(resolution, ['kaks']) === false;
|
||||
```
|
||||
|
||||
- [ ] **Step 4: 添加确认参数直传测试**
|
||||
|
||||
Stub `ensurePpiKaksDependencyOwner`,执行 `prepareKaksSelectedReference(confirmed)`,断言精确调用:
|
||||
|
||||
```javascript
|
||||
ensurePpiKaksDependencyOwner('kaks', confirmed);
|
||||
```
|
||||
|
||||
- [ ] **Step 5: 执行 RED 验证**
|
||||
|
||||
```bash
|
||||
/home/xzq/miniconda3/envs/test4py311/bin/python -m pytest -q \
|
||||
tests/unit/test_ppi_owner_scheduling_task5.py \
|
||||
-k 'kaks and (reuse or project or confirmed or near_reference)'
|
||||
```
|
||||
|
||||
Expected: 新增测试按预期失败;不得因测试夹具或语法错误失败。
|
||||
|
||||
---
|
||||
|
||||
### Task 2: 让 Ka/Ks 预检消费用户确认后的唯一参数
|
||||
|
||||
**Files:**
|
||||
- Modify: `apps/web/vue3/index.html:14534-14908`
|
||||
- Test: `tests/unit/test_ppi_owner_scheduling_task5.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `confirmedStageParamsForExecution('kaks', stageParams)` 的返回值。
|
||||
- Produces: `buildPpiKaksDependencyRequest(stageIds, stageParamsOverride)` 与 `ensurePpiKaksDependencyOwner(ownerStageId, stageParamsOverride)`。
|
||||
|
||||
- [ ] **Step 1: 让请求构造器接受确认参数,不新增抽象层**
|
||||
|
||||
```javascript
|
||||
const buildPpiKaksDependencyRequest = (stageIds = [], stageParamsOverride = null) => {
|
||||
const dependencyStage = (batchExecuteStages.value || [])
|
||||
.find(stage => getStageId(stage) === stageIds[0]) || {};
|
||||
const storedStageParams = typeof dependencyStage === 'object'
|
||||
? (dependencyStage.params || {})
|
||||
: {};
|
||||
const stageParams = stageParamsOverride && typeof stageParamsOverride === 'object'
|
||||
? stageParamsOverride
|
||||
: storedStageParams;
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 按 mapping owner 决定是否保留 PPI 身份**
|
||||
|
||||
```javascript
|
||||
const mappingOwner = stageParams.orthologMappingOwner || ownerStageId;
|
||||
const usesPpiOwner = mappingOwner === 'ppi';
|
||||
```
|
||||
|
||||
仅 `usesPpiOwner` 时,从 `stageParams → batchParams → planParams` 读取 `ppiReferenceTaxonId`、`ppiReferenceStorageKey`、`ppiReferenceSelectionMode`、`ppiReferenceUserConfirmed`。项目 At 路线保持这些字段为空。
|
||||
|
||||
- [ ] **Step 3: 让 owner 预检接受确认参数**
|
||||
|
||||
```javascript
|
||||
const ensurePpiKaksDependencyOwner = async (ownerStageId, stageParamsOverride = null) => {
|
||||
const requestedStages = [ownerStageId];
|
||||
const requestBody = buildPpiKaksDependencyRequest(requestedStages, stageParamsOverride);
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Ka/Ks 准备函数传入确认参数**
|
||||
|
||||
```javascript
|
||||
const prepareKaksSelectedReference = async (stageParams = {}) => {
|
||||
const ready = await ensurePpiKaksDependencyOwner('kaks', stageParams);
|
||||
```
|
||||
|
||||
后续保持:owner 为 `ppi` 时复用已有映射;owner 为 `kaks` 时只生成一次 Ka/Ks 专属 Ortholog。
|
||||
|
||||
- [ ] **Step 5: 执行 GREEN 验证**
|
||||
|
||||
```bash
|
||||
/home/xzq/miniconda3/envs/test4py311/bin/python -m pytest -q tests/unit/test_ppi_owner_scheduling_task5.py
|
||||
```
|
||||
|
||||
Expected: 全文件通过;原有 PPI owner single-flight、running 状态、失败不 resume 行为仍通过。
|
||||
|
||||
---
|
||||
|
||||
### Task 3: 将近源优先规则限制为 PPI,并隔离 At 兜底
|
||||
|
||||
**Files:**
|
||||
- Modify: `apps/web/vue3/index.html:13804-13927`
|
||||
- Modify: `apps/web/vue3/index.html:14336-14400`
|
||||
- Test: `tests/unit/test_ppi_owner_scheduling_task5.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `requestedStages` 与 PPI fallback 用户确认结果。
|
||||
- Produces: PPI-only 近源决策和 fail-closed 的 At fallback。
|
||||
|
||||
- [ ] **Step 1: 限制近源规则归属**
|
||||
|
||||
```javascript
|
||||
const ppiNeedsNearReferenceDecision = (resolution = {}, requestedStages = []) => {
|
||||
if (!requestedStages.includes('ppi')) return false;
|
||||
const selected = String(resolution.selected_string_species || '').toLowerCase();
|
||||
return resolution.status === 'needs_ortholog'
|
||||
&& selected === 'ath'
|
||||
&& ppiHasNearReferenceBeforeArabidopsis(resolution);
|
||||
};
|
||||
```
|
||||
|
||||
在 `runPpiSpeciesResolutionPreflight` 和 `ensurePpiKaksDependencyOwner` 中均传入当前 `requestedStages`。
|
||||
|
||||
- [ ] **Step 2: PPI下载失败后的At兜底不写Ka/Ks参数**
|
||||
|
||||
删除 `applyArabidopsisFallbackForPpiKaks` 中针对 `requestedStages.includes('kaks')` 的参数写入;affected stage 只包含 PPI owner 及其隐藏 Ortholog producer。Ka/Ks 保持原状态,等待 PPI settled 后弹出自己的确认框。
|
||||
|
||||
- [ ] **Step 3: fallback orchestration 失败时保持暂停**
|
||||
|
||||
```javascript
|
||||
const orchestrated = await orchestratePpiOwnerAfterReferenceReady({
|
||||
requestBody,
|
||||
requestedStages: ['ppi'],
|
||||
runPreflight: runPpiSpeciesResolutionPreflight,
|
||||
dependencyStatus: dependencyId => stageExecutionStatus.value[dependencyId],
|
||||
waitForDependency: waitForPpiProducerDependency,
|
||||
ensureOrtholog: async (nextResolution, nextRequest) => {
|
||||
const ready = ensureOrthologStageForPpiKaks(
|
||||
batchExecuteStages.value || [],
|
||||
['ppi'],
|
||||
nextRequest,
|
||||
nextResolution
|
||||
);
|
||||
return {
|
||||
ready: Boolean(ready),
|
||||
producerStage: getPpiKaksInternalOrthologProducer('ppi')
|
||||
};
|
||||
},
|
||||
executeProducer: executePpiOwnedOrthologProducer,
|
||||
resume: async () => resumePpiKaksPausedStages(
|
||||
['ppi'],
|
||||
'拟南芥兜底参考已确认'
|
||||
)
|
||||
});
|
||||
if (orchestrated !== true) {
|
||||
const pauseMessage = '拟南芥兜底参考尚未形成可用的 PPI Ortholog 映射;PPI 保持暂停。';
|
||||
storePpiKaksReferencePause(['ppi'], {}, pauseMessage);
|
||||
addAgentMessage('assistant', `⏸️ ${pauseMessage}`);
|
||||
return false;
|
||||
}
|
||||
```
|
||||
|
||||
不得在该 handler 中调用 `resumePpiKaksPausedStages(['kaks'])`。
|
||||
|
||||
- [ ] **Step 4: 执行调度回归**
|
||||
|
||||
```bash
|
||||
/home/xzq/miniconda3/envs/test4py311/bin/python -m pytest -q \
|
||||
tests/unit/test_ppi_owner_scheduling_task5.py \
|
||||
tests/unit/test_ppi_frontend_display_contract.py
|
||||
```
|
||||
|
||||
Expected: 全部通过;规划中无 Ortholog,PPI 先于 Ka/Ks,Ka/Ks 仍受 `ppiSettledForKaks` 门控。
|
||||
|
||||
---
|
||||
|
||||
### Task 4: 后端 API 路线矩阵验证,不预先修改后端
|
||||
|
||||
**Files:**
|
||||
- Inspect only: `apps/web/vue3/main.py:15185-15225`
|
||||
- Inspect only: `apps/web/pipeline_backend.py:2795-3177`
|
||||
- Inspect only: `apps/web/pipeline_backend.py:13398-13421`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: 前端最终生成的两个 Ka/Ks preflight payload。
|
||||
- Produces: 后端现有契约是否足够的证据。
|
||||
|
||||
- [ ] **Step 1: 菠菜复用路线 API 验证**
|
||||
|
||||
关键 payload:
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": "job_codex_ppi_kaks_route_20260810",
|
||||
"prefix": "Ft.SPL",
|
||||
"targetSpecies": ["Ft"],
|
||||
"referenceSpecies": ["string_taxon_3562"],
|
||||
"ppiReferenceTaxonId": "3562",
|
||||
"ppiReferenceStorageKey": "string_taxon_3562",
|
||||
"ppiReferenceSelectionMode": "tree_nearest",
|
||||
"ppiReferenceUserConfirmed": true,
|
||||
"dependencyOwner": "ppi",
|
||||
"requestedStages": ["kaks"]
|
||||
}
|
||||
```
|
||||
|
||||
Expected: `status=ready`,`ortholog_mapping_source` 指向 PPI owner 映射;不产生下载或新 Ortholog。
|
||||
|
||||
- [ ] **Step 2: 项目 At 路线 API 验证**
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": "job_codex_ppi_kaks_route_20260810",
|
||||
"prefix": "Ft.SPL",
|
||||
"targetSpecies": ["Ft"],
|
||||
"referenceSpecies": ["At"],
|
||||
"dependencyOwner": "kaks",
|
||||
"requestedStages": ["kaks"]
|
||||
}
|
||||
```
|
||||
|
||||
Expected: At PEP/CDS存在且无Ka/Ks owner映射时返回 `status=needs_ortholog`,不是 `needs_reference_data`,也不受菠菜优先规则阻断。
|
||||
|
||||
- [ ] **Step 3: 后端范围门禁**
|
||||
|
||||
如果两个 API 结果符合预期,后端保持零改动。如果不符合,只记录确切请求与响应并停止,另行制定后端最小修订,不能顺手扩展。
|
||||
|
||||
---
|
||||
|
||||
### Task 5: 隔离真实数据闭环验证
|
||||
|
||||
**Files:**
|
||||
- Create at runtime: `input/job_codex_ppi_kaks_route_20260810/`
|
||||
- Create at runtime: `workspace/job_codex_ppi_kaks_route_20260810/`
|
||||
- Verify only: `final/*.manifest.json`、logs、`/api/stage-status`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Ft/SPL现有成员输入、At与菠菜参考PEP/CDS。
|
||||
- Produces: 两条Ka/Ks路线的真实结果,不污染用户job。
|
||||
|
||||
- [ ] **Step 1: 创建隔离 job**
|
||||
|
||||
逐文件只读复用 Ft/SPL 成员输入和参考 FASTA;不得把整个用户 input 目录软链接为可写目录。运行时缓存和结果全部写入隔离 job。
|
||||
|
||||
- [ ] **Step 2: 路线 A:复用 PPI 菠菜映射**
|
||||
|
||||
顺序:PPI owner preflight ready → Ka/Ks确认复用 → Ka/Ks preflight ready → `/api/execute` Ka/Ks。
|
||||
|
||||
Expected: Ka/Ks确认后不下载、不生成第二个Ortholog,请求携带 `orthologMappingOwner=ppi`,Ka/Ks成功生成结果和manifest。
|
||||
|
||||
- [ ] **Step 3: 路线 B:项目 At**
|
||||
|
||||
顺序:Ka/Ks确认At → preflight needs_ortholog → 一次 Ka/Ks owner Ortholog → 二次 preflight ready → `/api/execute` Ka/Ks。
|
||||
|
||||
Expected: At PEP/CDS已存在时不下载,只生成一次Ka/Ks owner Ortholog,请求携带 `orthologMappingOwner=kaks`,Ka/Ks成功生成结果和manifest。
|
||||
|
||||
- [ ] **Step 4: 核对闭环身份**
|
||||
|
||||
核对确认参数、preflight请求、Ortholog命令日志、Ortholog manifest、Ka/Ks执行请求、Ka/Ks日志、Ka/Ks manifest、`/api/stage-status` 的参考物种与 owner 全部一致。
|
||||
|
||||
- [ ] **Step 5: 确认用户目录未被写入**
|
||||
|
||||
比较验证前后用户 job `input/` 与 `final/` 文件清单和 mtime。Expected: 无新增 PPI、Ka/Ks、Ortholog 或缓存文件。
|
||||
|
||||
---
|
||||
|
||||
### Task 6: 范围审计、重启与前端验收
|
||||
|
||||
**Files:**
|
||||
- Verify: git diff
|
||||
- Runtime: sj-2 Web 9016 and all sj-2 Celery workers
|
||||
|
||||
- [ ] **Step 1: 审计改动范围**
|
||||
|
||||
```bash
|
||||
git diff --name-only
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected code/test changes only:
|
||||
|
||||
```text
|
||||
apps/web/vue3/index.html
|
||||
tests/unit/test_ppi_owner_scheduling_task5.py
|
||||
docs/superpowers/plans/2026-08-10-ppi-kaks-reference-route-handoff.md
|
||||
```
|
||||
|
||||
如出现 `planning_v2.py`、`main.py`、`pipeline_backend.py`、CLI、Manifest Schema 或 stage catalog,停止并撤回本轮非必要修改。
|
||||
|
||||
- [ ] **Step 2: 确认无运行任务后重启全部sj-2服务**
|
||||
|
||||
先检查 `/api/tasks` 和 Celery active;确认空闲后重启 sj-2 Web 与 default/heavy/medium/light/rnaseq workers,不影响其它项目端口或队列。
|
||||
|
||||
- [ ] **Step 3: 健康检查**
|
||||
|
||||
Expected: 9016首页HTTP 200;`/api/celery/health` 返回sj-2五类worker在线;9016返回的 `index.html` hash 与磁盘文件一致。
|
||||
|
||||
- [ ] **Step 4: 前端验收标准**
|
||||
|
||||
路线 A:PPI菠菜成功 → Ka/Ks选择复用 → 无下载、无第二次Ortholog → Ka/Ks成功。
|
||||
|
||||
路线 B:PPI菠菜成功 → Ka/Ks选择At → 无下载 → 一次Ka/Ks专属Ortholog → Ka/Ks成功。
|
||||
|
||||
PPI下载失败路线:确认At → PPI At Ortholog和PPI完成 → Ka/Ks随后独立弹窗;失败时保持PPI暂停,不伪报恢复。
|
||||
|
||||
## Completion Gate
|
||||
|
||||
只有同时满足以下条件才可声明完成:
|
||||
|
||||
1. 新增测试先RED后GREEN。
|
||||
2. 两条后端预检矩阵与预期一致。
|
||||
3. 两条真实Ka/Ks路线均执行成功。
|
||||
4. 用户job没有新增测试产物。
|
||||
5. 改动范围仅限前端调度、对应测试和本计划文档。
|
||||
6. 9016 Web与全部sj-2 Celery worker重启并健康。
|
||||
@@ -0,0 +1,7 @@
|
||||
# PPI/KaKs Unified Reference Dialogs Implementation Plan
|
||||
|
||||
1. Add focused frontend contract tests that require both messages to use `showStageConfirmDialog`, ordered candidate lists, grouped reference fields, and their original accept/cancel semantics.
|
||||
2. Add two display-only stage IDs and initialize their view model from the existing resolution objects.
|
||||
3. Add the two panels and dedicated footer actions to the existing stage-confirmation dialog.
|
||||
4. Replace only the two raw `ElMessageBox` calls with queued stage-confirmation calls.
|
||||
5. Run the focused tests, related PPI frontend regression tests, syntax/diff checks, then verify the served 9016 page after a safe restart.
|
||||
@@ -0,0 +1,232 @@
|
||||
# 5090 RAG Stream Timeout Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Make PlantAI RAG requests on 5090 complete through the full 7920 Qdrant + Neo4j + Agentic pipeline without fixed 120/150-second client aborts.
|
||||
|
||||
**Architecture:** The browser uses the existing SSE endpoint and treats timeout as inactivity rather than total elapsed time. PlantAI proxies 7920 keepalives unchanged, while the Python adapter uses separate connect, stream-idle, and non-stream compatibility timeouts. Retrieval and answer-generation behavior remain unchanged.
|
||||
|
||||
**Tech Stack:** FastAPI, `requests`, browser Fetch/ReadableStream, SSE, pytest.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Preserve complete Qdrant + Neo4j retrieval; no Qdrant-only fallback.
|
||||
- Do not change Agent prompts, rounds, evidence selection, or answer composition.
|
||||
- Stream connect timeout: 10 seconds.
|
||||
- Stream idle timeout: configurable, default 45 seconds.
|
||||
- Non-stream compatibility timeout: configurable, default 900 seconds.
|
||||
- Do not commit or push; the user will handle Git later.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Python timeout policy
|
||||
|
||||
**Files:**
|
||||
- Modify: `gfa_core/mas/dual_kb_qa.py`
|
||||
- Modify: `tests/test_dual_kb_qa_timeout.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `QA_V1_1_73_TIMEOUT`, new `QA_V1_1_73_STREAM_IDLE_TIMEOUT` environment variables.
|
||||
- Produces: `_non_stream_timeout(timeout: float) -> tuple[float, float]` and `_stream_timeout() -> tuple[float, float]` policies used by both adapter entry points.
|
||||
|
||||
- [ ] **Step 1: Replace the hard-cap assertions with failing policy tests**
|
||||
|
||||
```python
|
||||
def test_non_stream_defaults_to_900_second_read_timeout(monkeypatch):
|
||||
_force_v1173(monkeypatch)
|
||||
# capture requests.post timeout
|
||||
assert captured == [(10.0, 900.0)]
|
||||
|
||||
def test_non_stream_honors_timeout_above_150(monkeypatch):
|
||||
monkeypatch.setenv("QA_V1_1_73_TIMEOUT", "600")
|
||||
assert captured == [(10.0, 600.0)]
|
||||
|
||||
def test_stream_uses_45_second_idle_read_timeout(monkeypatch):
|
||||
assert captured == [(10.0, 45.0)]
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused tests and verify RED**
|
||||
|
||||
Run: `pytest -q tests/test_dual_kb_qa_timeout.py`
|
||||
|
||||
Expected: failures showing the current 150-second cap.
|
||||
|
||||
- [ ] **Step 3: Implement separate timeout policies**
|
||||
|
||||
```python
|
||||
QA_CONNECT_TIMEOUT_SECONDS = 10.0
|
||||
QA_V1_1_73_DEFAULT_TIMEOUT_SECONDS = 900.0
|
||||
QA_V1_1_73_DEFAULT_STREAM_IDLE_TIMEOUT_SECONDS = 45.0
|
||||
|
||||
def _positive_timeout(value: object, default: float) -> float:
|
||||
try:
|
||||
parsed = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
return parsed if parsed > 0 else default
|
||||
```
|
||||
|
||||
Use `QA_V1_1_73_TIMEOUT` for non-stream reads and `QA_V1_1_73_STREAM_IDLE_TIMEOUT` for stream idle reads. Remove `min(..., 150)`.
|
||||
|
||||
- [ ] **Step 4: Run focused tests and verify GREEN**
|
||||
|
||||
Run: `pytest -q tests/test_dual_kb_qa_timeout.py`
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
### Task 2: Browser SSE inactivity timer and RAG route
|
||||
|
||||
**Files:**
|
||||
- Modify: `apps/web/vue3/index.html`
|
||||
- Modify: `apps/web/vue3/index_ml.html`
|
||||
- Create: `tests/unit/test_rag_stream_frontend_contract.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `/api/agent/rag/stream` SSE frames and comment keepalives.
|
||||
- Produces: `readRagEventStream(url, options, idleTimeoutMs)` that resets its timer on every received chunk; RAG mode calls `requestRagWithProgress`.
|
||||
|
||||
- [ ] **Step 1: Add failing source-contract tests for both page variants**
|
||||
|
||||
```python
|
||||
@pytest.mark.parametrize("name", ["index.html", "index_ml.html"])
|
||||
def test_rag_mode_uses_streaming_request(name):
|
||||
source = (FRONTEND_DIR / name).read_text(encoding="utf-8")
|
||||
rag_block = source[source.index("// V2: RAG模式"):source.index("} catch (err)", source.index("// V2: RAG模式"))]
|
||||
assert "requestRagWithProgress(" in rag_block
|
||||
assert "fetchJsonWithTimeout('/api/agent/rag'" not in rag_block
|
||||
|
||||
@pytest.mark.parametrize("name", ["index.html", "index_ml.html"])
|
||||
def test_stream_timer_is_reset_after_each_chunk(name):
|
||||
source = (FRONTEND_DIR / name).read_text(encoding="utf-8")
|
||||
assert "resetIdleTimer();" in source
|
||||
assert "const { value, done } = await reader.read();" in source
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the frontend contract test and verify RED**
|
||||
|
||||
Run: `pytest -q tests/unit/test_rag_stream_frontend_contract.py`
|
||||
|
||||
Expected: both page variants fail because RAG mode uses the non-stream endpoint and the timer is total-duration based.
|
||||
|
||||
- [ ] **Step 3: Implement the inactivity timer in both page variants**
|
||||
|
||||
Define `resetIdleTimer()` to clear and re-arm the abort timer. Call it before fetch and immediately after every successful `reader.read()` that returns a non-empty chunk. Keep `AbortController` for genuine inactivity.
|
||||
|
||||
- [ ] **Step 4: Route both RAG-mode handlers through SSE**
|
||||
|
||||
Replace the direct `fetchJsonWithTimeout('/api/agent/rag', ..., 120000)` call with:
|
||||
|
||||
```javascript
|
||||
const data = await requestRagWithProgress({
|
||||
query: userInput,
|
||||
mode: 'rag',
|
||||
top_k: 10
|
||||
}, 45000);
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run frontend contract tests and verify GREEN**
|
||||
|
||||
Run: `pytest -q tests/unit/test_rag_stream_frontend_contract.py`
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
### Task 3: PlantAI API proxy contract
|
||||
|
||||
**Files:**
|
||||
- Modify: `apps/web/vue3/main.py`
|
||||
- Modify: `tests/test_dual_kb_qa_timeout.py` or create `tests/unit/test_rag_stream_proxy_contract.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `stream_dual_kb_qa(..., idle_timeout=45)`.
|
||||
- Produces: SSE response with unchanged keepalive/progress/complete/error frames.
|
||||
|
||||
- [ ] **Step 1: Add a failing contract test**
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
MAIN = ROOT / "apps/web/vue3/main.py"
|
||||
|
||||
|
||||
def test_rag_proxy_has_no_inline_150_second_timeout():
|
||||
source = MAIN.read_text(encoding="utf-8")
|
||||
start = source.index('@app.post("/api/agent/rag")')
|
||||
end = source.index("# ==================== DiagnosisAgent API", start)
|
||||
rag_routes = source[start:end]
|
||||
assert "timeout=150.0" not in rag_routes
|
||||
assert "complete QA exceeded 150 seconds" not in rag_routes
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the proxy contract test and verify RED**
|
||||
|
||||
Run: `pytest -q tests/unit/test_rag_stream_proxy_contract.py`
|
||||
|
||||
Expected: failure on the current `timeout=150.0` literals.
|
||||
|
||||
- [ ] **Step 3: Remove inline fixed timeout literals**
|
||||
|
||||
Let `stream_dual_kb_qa` select the 45-second idle timeout and `query_dual_kb_qa` select the 900-second compatibility timeout. Preserve structured timeout and connection error responses.
|
||||
|
||||
- [ ] **Step 4: Run proxy and adapter tests and verify GREEN**
|
||||
|
||||
Run: `pytest -q tests/unit/test_rag_stream_proxy_contract.py tests/test_dual_kb_qa_timeout.py`
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
### Task 4: Regression and 5090 deployment validation
|
||||
|
||||
**Files:**
|
||||
- No production source additions.
|
||||
- Runtime candidate service uses the mounted `sj/PlantAI-dev` tree.
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: 5090 port 7920 and candidate PlantAI port 9010.
|
||||
- Produces: health evidence, SSE keepalive evidence, final answer with graph chains/citations.
|
||||
|
||||
- [ ] **Step 1: Run focused and adjacent regression tests**
|
||||
|
||||
Run: `pytest -q tests/test_dual_kb_qa_timeout.py tests/unit/test_rag_stream_frontend_contract.py tests/unit/test_rag_stream_proxy_contract.py`
|
||||
|
||||
Expected: all pass.
|
||||
|
||||
- [ ] **Step 2: Validate syntax**
|
||||
|
||||
Run: `python -m py_compile gfa_core/mas/dual_kb_qa.py apps/web/vue3/main.py`
|
||||
|
||||
Expected: exit code 0.
|
||||
|
||||
- [ ] **Step 3: Start/restart only the candidate 9010 service on 5090**
|
||||
|
||||
Synchronize the mounted source into the explicit candidate directory `/home/userlab/plantAI/sj/PlantAI-dev-rag-stream-20260820` on 5090, excluding `.git`, caches, results, and virtual environments. Start `apps/web/vue3/main.py` on port 9010 with:
|
||||
|
||||
```bash
|
||||
GENE_FAMILY_MAS_API=http://127.0.0.1:7920 \
|
||||
DUAL_KB_QA_ROUTE=qa-v1-1-73 \
|
||||
QA_V1_1_73_TIMEOUT=900 \
|
||||
QA_V1_1_73_STREAM_IDLE_TIMEOUT=45 \
|
||||
python apps/web/vue3/main.py --host 0.0.0.0 --port 9010
|
||||
```
|
||||
|
||||
Do not restart or replace 7920.
|
||||
|
||||
- [ ] **Step 4: Verify health and SSE keepalive path**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
curl -N --max-time 60 -H 'Content-Type: application/json' \
|
||||
-d '{"query":"What biological processes are regulated by the NAC transcription factor family in plants?","mode":"rag","top_k":10}' \
|
||||
http://127.0.0.1:9010/api/agent/rag/stream
|
||||
```
|
||||
|
||||
Expected: an SSE progress frame or `: keepalive` is received before curl reaches 60 seconds; no 504 frame is returned.
|
||||
|
||||
- [ ] **Step 5: Verify a real long question end to end**
|
||||
|
||||
Run the same curl without a 60-second cap and save the SSE response under `/tmp/plantai-9010-rag-stream-validation.sse`. Success requires an `event: complete` frame whose JSON has a non-empty answer, citations, and `native_graph_chains` or `graph_knowledge_chains`. Record curl's `time_total`; a value greater than 150 seconds is acceptable when earlier progress/keepalive frames are present.
|
||||
|
||||
- [ ] **Step 6: Inspect process and Neo4j state**
|
||||
|
||||
On 5090, check `ps -ef` for only the intended 9010 candidate process and query Neo4j `SHOW TRANSACTIONS` through the existing Neo4j container/CLI. Success requires no transaction from the validation request remaining more than 10 seconds after the complete event.
|
||||
@@ -0,0 +1,21 @@
|
||||
# QA 7920 Formal Integration Implementation Plan
|
||||
|
||||
**Goal:** Make the validated full-graph QA service and PlantAI timeout behavior versioned in this repository.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Work only in this repository for persistent source changes.
|
||||
- Preserve Qdrant + Neo4j and `QA_V1_BOUNDED_NEO4J_TIMEOUT=8`.
|
||||
- Apply a 10-second connect timeout and 150-second complete-answer timeout.
|
||||
- Validate on a sidecar before replacing 7920.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. Add `tests/test_qa7920_formal_deployment.py` and verify it fails because the formal package and startup entrypoint do not exist.
|
||||
2. Vendor the validated QA package under `services/qa_v1_1_73/src/rag/qa_v1_1_73/`, its three direct support modules, and two YAML configs.
|
||||
3. Add `scripts/qa_v1_1_73_server.py` and `scripts/start_qa_v1_1_73_7920.sh`; make the deployment test pass and compile all Python sources.
|
||||
4. Add `tests/test_dual_kb_qa_timeout.py`; verify it fails against the current 300-second/scalar timeout behavior.
|
||||
5. Change `gfa_core/mas/dual_kb_qa.py` to `timeout=(10.0, 150.0)` and change `apps/web/vue3/main.py` to emit HTTP 504 for non-stream timeout and use 150 seconds for streaming.
|
||||
6. Run focused tests and compilation, then start a sidecar with the existing full backends.
|
||||
7. Require a correct `Solanum lycopersicum` answer, non-empty native graph chains, and wall time below 150 seconds before switching 7920.
|
||||
8. Restart PlantAI through its controlled endpoint and repeat the same assertion through 9010. Roll back runtime processes on any failed assertion.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Reference Download Progress Indicator Design
|
||||
|
||||
## Objective
|
||||
|
||||
Make background reference-bundle downloads visible throughout execution so users can tell which species is being downloaded, how much data has arrived, whether the download completed, and why it failed.
|
||||
|
||||
The indicator will appear beside “💡 使用小Tip” in the analysis input toolbar. This change is presentation-only: it must not alter PPI, KaKs, Ortholog, planning, download selection, fallback decisions, or backend execution.
|
||||
|
||||
## Current Problem
|
||||
|
||||
The frontend already polls `/api/task/{task_id}` and stores download state in `referenceBundleDownloadState`. A progress block also exists, but it is rendered inside the batch-execution confirmation dialog. `startBatchExecute()` closes that dialog before execution begins, so the user cannot see the progress block during the actual background download.
|
||||
|
||||
The task API already supplies useful runtime information, including:
|
||||
|
||||
- species label;
|
||||
- task status and message;
|
||||
- percentage when the upstream source provides a total size;
|
||||
- `bytes_downloaded` and `total_bytes`;
|
||||
- terminal success or failure and the failure reason.
|
||||
|
||||
The missing piece is a visible consumer of that existing state.
|
||||
|
||||
## Chosen Design
|
||||
|
||||
### Placement and interaction
|
||||
|
||||
Add a compact “⬇️ 下载进度” status item immediately beside “💡 使用小Tip”. It is hidden when no reference download has been observed in the current page lifecycle.
|
||||
|
||||
While visible, its label reflects the current state:
|
||||
|
||||
- running: `⬇️ 下载中 8.0 MB`;
|
||||
- success: `✅ 下载完成`;
|
||||
- failed: `❌ 下载失败`.
|
||||
|
||||
Hovering over the item opens a compact detail panel using the existing Tip/Upload hover-panel visual pattern. The panel shows:
|
||||
|
||||
- reference species;
|
||||
- running/success/failed status;
|
||||
- determinate progress when total size is known;
|
||||
- indeterminate progress plus downloaded MB when total size is unknown;
|
||||
- backend progress or terminal error message;
|
||||
- a close action for terminal states.
|
||||
|
||||
The running state cannot be dismissed accidentally. Success and failure remain visible until the user closes them. Starting a new download replaces the previous terminal state and shows the new task immediately.
|
||||
|
||||
### State handling
|
||||
|
||||
Reuse `referenceBundleDownloadState`; do not introduce a second download-state authority.
|
||||
|
||||
Extend the state with only the display fields required by the indicator:
|
||||
|
||||
- `bytesDownloaded`;
|
||||
- `totalBytes`.
|
||||
|
||||
`updateReferenceBundleDownloadProgress()` will read these values from the existing task progress payload. If `totalBytes` is unavailable or zero, the progress bar remains indeterminate and the UI reports the downloaded byte count instead of inventing a percentage.
|
||||
|
||||
On success, update the existing state to a visible `success` terminal state at 100% rather than immediately clearing it. On failure, retain the visible `failed` state and its exact error message. `clearReferenceBundleDownloadProgress()` remains the single close/reset operation.
|
||||
|
||||
### Scope boundary
|
||||
|
||||
Only `apps/web/vue3/index.html` is expected to change:
|
||||
|
||||
- toolbar markup beside “使用小Tip”;
|
||||
- narrowly scoped styles matching existing hover components;
|
||||
- the existing `referenceBundleDownloadState` shape and progress updater;
|
||||
- success-state retention in the existing monitor path;
|
||||
- a small byte-formatting display helper if no equivalent helper already exists.
|
||||
|
||||
Explicitly out of scope:
|
||||
|
||||
- backend API changes;
|
||||
- download retry/resume implementation;
|
||||
- fallback-to-At behavior;
|
||||
- PPI, KaKs, or Ortholog execution and dependencies;
|
||||
- planning or stage identity;
|
||||
- cross-refresh persistence;
|
||||
- unrelated frontend refactoring.
|
||||
|
||||
## Error and Terminal-State Behavior
|
||||
|
||||
- Network/download failure: show a red terminal state, downloaded size, and the backend error text.
|
||||
- Unknown content length: show an indeterminate bar and downloaded MB; do not display a fabricated percentage.
|
||||
- Successful completion: show a green 100% state and completion message until closed.
|
||||
- Subsequent download: replace the old terminal state with the new running task.
|
||||
|
||||
The indicator reports state only. Existing confirmation dialogs remain responsible for retry, skip, reference choice, and At fallback decisions.
|
||||
|
||||
## Verification
|
||||
|
||||
Add focused frontend-source/Node tests that exercise the real state-update logic rather than mocking it away.
|
||||
|
||||
Acceptance checks:
|
||||
|
||||
1. A running task with unknown total bytes renders a visible indeterminate state and downloaded MB.
|
||||
2. A running task with known total bytes renders the backend percentage.
|
||||
3. Success remains visible and shows the terminal completion state until close.
|
||||
4. Failure remains visible and shows the exact failure reason until close.
|
||||
5. Close clears a terminal state; a subsequent progress event makes the indicator visible again.
|
||||
6. Existing PPI/KaKs scheduling and download-failure tests continue to pass.
|
||||
7. The rendered indicator sits beside “使用小Tip”, outside the batch confirmation dialog.
|
||||
|
||||
Validation will use the current real job/task payload as a fixture shape, including the observed `IncompleteRead` failure and unknown `total_bytes=0`. No PPI, KaKs, or Ortholog analysis rerun is required to validate this presentation-only change.
|
||||
@@ -0,0 +1,25 @@
|
||||
# PPI/KaKs Unified Reference Dialogs Design
|
||||
|
||||
## Goal
|
||||
|
||||
Render the two PPI/KaKs reference-species messages inside the existing stage-confirmation dialog shell so their information is readable and visually consistent with the rest of the application.
|
||||
|
||||
## Scope
|
||||
|
||||
- Replace the raw STRING reference confirmation message box with a stage-confirmation dialog.
|
||||
- Replace the raw reference-data/Ortholog warning alert with a read-only stage-confirmation dialog.
|
||||
- Show reference candidates as an ordered list.
|
||||
- Group target species, requested stages, required/missing files, current reference species, and Ortholog mapping into the existing PPI reference panel layout.
|
||||
- Normalize both topology-candidate and downloaded-reference recommendation field names for display only.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No changes to PPI/KaKs download, Ortholog generation, identity, scheduling, API payloads, backend validation, or analysis execution.
|
||||
- No changes to which candidate is selected: the first ranked candidate remains the confirmed candidate.
|
||||
- No new automatic confirmation behavior.
|
||||
|
||||
## Interaction contract
|
||||
|
||||
- STRING candidate dialog: `Cancel PPI` resolves false; `Confirm top candidate` resolves true.
|
||||
- Reference status dialog: `OK` only acknowledges the message; the preflight remains blocked exactly as before.
|
||||
- Both dialogs use the existing queue, modal shell, title, progress header, and Element Plus components.
|
||||
@@ -0,0 +1,44 @@
|
||||
# 5090 RAG 流式超时修订设计
|
||||
|
||||
## 目标
|
||||
|
||||
让 5090 上的 PlantAI 问答完整使用 7920 的 Qdrant + Neo4j + Agentic QA。复杂问题即使总耗时超过 150 秒,只要 7920 仍持续发送进度或 keepalive,就不能被 PlantAI 客户端误判为超时。此次不减少检索步骤、不跳过辅助问题、不降级为 Qdrant-only。
|
||||
|
||||
## 已确认根因
|
||||
|
||||
- 7920 的 `/api/v1/qa-v1-1-73/chat/stream` 每 15 秒发送 SSE keepalive。
|
||||
- 页面已经定义流式 RAG 客户端,但实际 RAG 模式仍调用旧的 `/api/agent/rag`,并在 120 秒主动中止。
|
||||
- PlantAI 的非流式代理又显式传入 150 秒;`gfa_core/mas/dual_kb_qa.py` 还会将 V1.1.73 的超时限制为最多 150 秒,并忽略更高的环境配置。
|
||||
- 普通 `/chat` 调用在 150 秒时断开后,7920 的同步后台计算仍会继续,造成用户收到超时但服务端任务继续占用资源。
|
||||
- 5090 当前运行 7920 和旧 Web 9017;指定的 `sj/PlantAI-dev` 尚未作为 9010 运行。
|
||||
|
||||
## 设计
|
||||
|
||||
1. 两套页面入口的 RAG 模式都改用已有的 `requestRagWithProgress`,由 PlantAI `/api/agent/rag/stream` 转发 7920 `/chat/stream`。
|
||||
2. 浏览器流读取从“固定总时长计时器”改为“读取空闲计时器”;每次收到 progress、keepalive 或其他数据块就重置计时器。
|
||||
3. 移除 120/150 秒总时长上限。客户端只保留:
|
||||
- 10 秒连接建立超时;
|
||||
- 可配置的读取空闲超时,默认 45 秒。
|
||||
4. 7920 每 15 秒的 keepalive 会重置读取空闲计时,因此正常长请求可以一直运行;如果 7920、网络或代理真正失联,客户端仍会在空闲超时后返回明确错误。
|
||||
5. 非流式兼容接口不再硬编码 150 秒,改用独立的可配置总超时,默认 900 秒。交互式前端不得走该接口。
|
||||
6. 保持完整图谱检索和现有答案生成逻辑,不调整 Agent 提示词、轮次和证据裁剪策略。
|
||||
|
||||
## 错误处理
|
||||
|
||||
- 连接失败返回 `service_unavailable`。
|
||||
- 流在超过空闲超时后无任何字节返回 `request_timeout`。
|
||||
- 收到 keepalive 但总执行超过 150 秒不报错。
|
||||
- 上游返回结构化错误时原样保留错误码和可读信息。
|
||||
|
||||
## 验证
|
||||
|
||||
- 单元测试证明环境变量高于 150 秒时不再被截断。
|
||||
- 前端契约测试证明 RAG 模式使用流式入口,且每个数据块会重置空闲计时器。
|
||||
- 代理测试模拟持续 keepalive 后最终成功。
|
||||
- 单元测试模拟真正的读取空闲超时并验证 504 错误。
|
||||
- 在 5090 启动指定 `PlantAI-dev` 的候选服务,直连 7920 执行健康检查和真实长问题。
|
||||
- 验证答案仍包含 Neo4j 图链和引用;不以 Qdrant-only 返回作为成功。
|
||||
|
||||
## 回退
|
||||
|
||||
不改动 7920 当前服务。PlantAI 代码修改可通过恢复相关文件或停止候选 9010 服务回退;不 push,由用户后续统一处理 Git。
|
||||
@@ -0,0 +1,17 @@
|
||||
# QA 7920 正式接入设计
|
||||
|
||||
## 目标
|
||||
|
||||
将完成 200 题评估的 QA V1.1.73 源码和启动入口纳入本 Git 仓库,由本机 `127.0.0.1:7920` 为 PlantAI 提供完整 Qdrant + Neo4j 问答,并将整次请求的异常边界设为 150 秒。
|
||||
|
||||
## 设计
|
||||
|
||||
- 将已验证 QA 包放入隔离的 `services/qa_v1_1_73/src/rag/qa_v1_1_73/`,并纳入它直接依赖的 cascaded graph retrieval、relation taxonomy 和 entity alias 模块,避免遮蔽 PlantAI 已使用的外部 `src.rag`。
|
||||
- 配置文件放入 `configs/rag/`;大模型、Qdrant、Neo4j 和论文元数据仍由环境变量指向现有运行资源,不提交数据或密钥。
|
||||
- 正式入口为 `scripts/qa_v1_1_73_server.py`,正式启动脚本为 `scripts/start_qa_v1_1_73_7920.sh`;运行命令和环境中不得出现 `/sandboxes/`。
|
||||
- PlantAI 保持调用同机 `127.0.0.1:7920`。连接超时为 10 秒,完整回答读取超时为 150 秒;超时返回 HTTP 504 或 SSE `request_timeout`。
|
||||
- `QA_V1_BOUNDED_NEO4J_TIMEOUT` 保持 8 秒,不切换 Qdrant-only,不修改图谱或索引。
|
||||
|
||||
## 验证与回退
|
||||
|
||||
先用契约测试证明仓库缺少正式 QA 服务,再加入源码使测试通过。在旁路端口验证健康信息、完整后端身份和历史测试题;只有在 150 秒内正确返回且图谱链非空时才替换 7920。失败时恢复旧 PID,不切换 9010。
|
||||
@@ -278,12 +278,29 @@ def _operation_run(record: dict[str, Any], operation: dict[str, Any]) -> tuple[d
|
||||
return None, "runtime" if action == "run" else "validation"
|
||||
|
||||
|
||||
def _missing_output_dir_hint(record: dict[str, Any], token: str) -> str | None:
|
||||
"""当 stderr 提示输出目录不存在时,给出更明确的提示。"""
|
||||
for log_path in _logs_for(record, token):
|
||||
try:
|
||||
text = Path(log_path).read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
continue
|
||||
if "No such file or directory" in text and "outputs/" in text:
|
||||
return (
|
||||
"输出目录不存在:工具把输出写入 outputs/ 子目录但未创建该目录"
|
||||
"(hmmsearch/diamond 等不会自动建目录),请在 run.sh 开头添加 mkdir -p 对应目录"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _run_error(record: dict[str, Any], token: str, run: dict[str, Any], phase: str) -> dict[str, Any]:
|
||||
details = run.get("error") if isinstance(run.get("error"), dict) else {}
|
||||
binding = run.get("binding") if isinstance(run.get("binding"), dict) else {}
|
||||
binding_error = binding.get("error") if isinstance(binding.get("error"), dict) else {}
|
||||
hint = _missing_output_dir_hint(record, token)
|
||||
message = (
|
||||
details.get("message")
|
||||
hint
|
||||
or details.get("message")
|
||||
or binding_error.get("message")
|
||||
or ("runtime binding was not resolved" if binding.get("status") != "resolved" else "execution did not satisfy the output contract")
|
||||
)
|
||||
|
||||
@@ -349,15 +349,14 @@ def _locate_patch_targets(
|
||||
for anchor in anchors
|
||||
}
|
||||
if reason is None:
|
||||
if len(source_matches) > 1 or len(anchor_identities) != len(anchors):
|
||||
reason = "source_literal_ambiguous"
|
||||
elif (
|
||||
len(source_matches) == 1
|
||||
and source_matches[0][1] != len(anchors)
|
||||
):
|
||||
reason = "source_literal_ambiguous"
|
||||
elif not source_matches:
|
||||
if not source_matches:
|
||||
reason = "source_literal_not_found_in_raw_shell"
|
||||
elif len(source_matches) > 1:
|
||||
# 字面量跨多个脚本,无法唯一定位
|
||||
reason = "source_literal_ambiguous"
|
||||
elif len(anchors) != 1 or len(anchor_identities) != 1:
|
||||
# 需要唯一 anchor 才能安全 patch
|
||||
reason = "source_literal_ambiguous"
|
||||
|
||||
clean_candidate = {
|
||||
key: value
|
||||
@@ -782,9 +781,21 @@ def _shared_output_prefix(outputs: list[dict[str, Any]]) -> str | None:
|
||||
_DIR_OVERRIDE_SWITCHES = ("-oc", "-o") # tools that accept a writable output directory
|
||||
|
||||
|
||||
def _is_dir_prefix(prefix: str) -> bool:
|
||||
"""Return True when *prefix* looks like a directory rather than a filename stem."""
|
||||
return "/" not in prefix and "." not in prefix and _PREFIX_TOKEN.fullmatch(prefix) is not None
|
||||
def _prefix_targets_directory(paths: list[str], prefix: str) -> bool:
|
||||
"""Return True when *prefix* matches a directory component of any path.
|
||||
|
||||
Checks the actual output paths rather than guessing from the prefix string:
|
||||
``meme_out/logo1.png`` has ``meme_out`` as a directory component, while
|
||||
``input_qc.tsv`` and ``Ft.blast`` do not — so filename stems are not mistaken
|
||||
for directory prefixes.
|
||||
"""
|
||||
for path in paths:
|
||||
for part in PurePosixPath(path).parts[:-1]: # 排除文件名
|
||||
if part.startswith(prefix) and (
|
||||
len(part) == len(prefix) or part[len(prefix)] in _PREFIX_DELIMITERS
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _replace_positional_prefix(command: str, old: str, new: str) -> str:
|
||||
@@ -1066,7 +1077,16 @@ def resolve_runtime_bindings(
|
||||
for fact_id in target.get("fact_ids", [])
|
||||
if isinstance(fact_id, str)
|
||||
}
|
||||
unpatchable_fact_ids = {
|
||||
fact_id
|
||||
for target in contract.get("unpatchable_targets", [])
|
||||
if isinstance(target, dict)
|
||||
for fact_id in target.get("fact_ids", [])
|
||||
if isinstance(fact_id, str)
|
||||
}
|
||||
output_bindings = []
|
||||
output_paths = [output["path"] for output in output_files]
|
||||
prefix_targets_dir = _prefix_targets_directory(output_paths, old_prefix)
|
||||
for output in output_files:
|
||||
original_path = output["path"]
|
||||
runtime_path = _runtime_input_owned_output(
|
||||
@@ -1075,9 +1095,17 @@ def resolve_runtime_bindings(
|
||||
allow_sidecar=output["fact_id"] not in direct_output_fact_ids,
|
||||
)
|
||||
if runtime_path is None:
|
||||
# 可重命名当且仅当:input-derived 前缀 / 目录前缀 / 有可 patch 的
|
||||
# 输出字面量。只有三者都不满足(硬编码且无法 patch,如
|
||||
# input_qc.tsv 动态拼接)才保持原名。
|
||||
can_rename = (
|
||||
bool(participating_paths)
|
||||
or prefix_targets_dir
|
||||
or output["fact_id"] not in unpatchable_fact_ids
|
||||
)
|
||||
runtime_path = (
|
||||
_replace_output_prefix(original_path, old_prefix, new_prefix)
|
||||
if _output_has_prefix(original_path, old_prefix)
|
||||
if can_rename and _output_has_prefix(original_path, old_prefix)
|
||||
else original_path
|
||||
)
|
||||
output_bindings.append(
|
||||
@@ -1781,43 +1809,39 @@ def render_runtime_package(
|
||||
):
|
||||
old = prefix["old_prefix"]
|
||||
new = prefix["new_prefix"]
|
||||
# Only rewrite when the prefix looks like a directory being renamed.
|
||||
# Check: every output whose runtime path differs has the prefix in
|
||||
# a directory component (not the filename). When that holds the
|
||||
# tool is writing into a fixed directory name and we can inject a
|
||||
# post-command rename.
|
||||
if any(
|
||||
b.get("transformed") for b in resolved.get("output_bindings", [])
|
||||
):
|
||||
# Prepend an output-dir override when the tool supports
|
||||
# ``-oc`` (MEME family) and the old prefix is a plain
|
||||
# single-segment directory.
|
||||
if _is_dir_prefix(old):
|
||||
# 目录前缀(如 meme_out/logo1.png 的 meme_out)→ -oc 注入。
|
||||
# 文件名 stem(如 input_qc.tsv)不是目录,不能注入 -oc。
|
||||
transformed_paths = [
|
||||
b["original_path"]
|
||||
for b in resolved.get("output_bindings", [])
|
||||
if b.get("transformed")
|
||||
]
|
||||
is_dir_prefix = _prefix_targets_directory(transformed_paths, old)
|
||||
if is_dir_prefix:
|
||||
actual_command = _inject_dir_prefix_override(
|
||||
command, old, new
|
||||
)
|
||||
# Also rewrite the inner command inside wrapper scripts
|
||||
# (e.g. ``bash run.sh`` wraps ``MCScanX ./Ft``).
|
||||
# Try positional-prefix replacement (for tools that take
|
||||
# the output prefix as a positional argument), then -oc/-o
|
||||
# injection (for MEME-family tools).
|
||||
for script in sorted(source_hashes):
|
||||
script_path = raw_code / script
|
||||
inner = script_path.read_text(
|
||||
encoding="utf-8", errors="replace"
|
||||
# 改写 shell 脚本里的命令(位置参数替换 / -oc 注入)。
|
||||
# 只处理 .sh 脚本:代码文件(.py 等)的输出路径由
|
||||
# _render_script_updates 负责,shlex 会破坏它们的语法。
|
||||
for script in sorted(source_hashes):
|
||||
if not script.endswith(".sh"):
|
||||
continue
|
||||
script_path = raw_code / script
|
||||
inner = script_path.read_text(
|
||||
encoding="utf-8", errors="replace"
|
||||
)
|
||||
rewritten = _replace_positional_prefix(inner, old, new)
|
||||
if rewritten == inner and is_dir_prefix:
|
||||
rewritten = _inject_dir_prefix_override(inner, old, new)
|
||||
if rewritten != inner:
|
||||
source_script_updates[script] = rewritten
|
||||
_symlink_prefix_data_files(
|
||||
old, new, work_dir, set(source_hashes)
|
||||
)
|
||||
rewritten = _replace_positional_prefix(inner, old, new)
|
||||
if rewritten == inner:
|
||||
rewritten = _inject_dir_prefix_override(inner, old, new)
|
||||
if rewritten != inner:
|
||||
source_script_updates[script] = rewritten
|
||||
# When positional-prefix rewrite changes a tool command
|
||||
# (e.g. MCScanX ./Ft → MCScanX ./output), some tools
|
||||
# expect data files matching the new prefix. Create
|
||||
# symlinks so the tool finds its inputs.
|
||||
_symlink_prefix_data_files(
|
||||
old, new, work_dir, set(source_hashes)
|
||||
)
|
||||
|
||||
for relative in sorted(source_hashes):
|
||||
destination = work_dir / relative
|
||||
@@ -1860,6 +1884,7 @@ def render_runtime_package(
|
||||
src = raw_data_root / relative
|
||||
dst = work_dir / relative
|
||||
if src.is_file() and not dst.exists():
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(src, dst)
|
||||
|
||||
# When a prefix rename is active, create symlinks so tools that
|
||||
|
||||
@@ -9,6 +9,7 @@ import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from typing import Any, Callable, Iterator
|
||||
from uuid import uuid4
|
||||
@@ -41,6 +42,34 @@ def _copy_tree(source: Path, destination: Path) -> None:
|
||||
shutil.copytree(source, destination, symlinks=True)
|
||||
|
||||
|
||||
def _conda_activated_env(runtime_env: str) -> dict[str, str]:
|
||||
"""Return the full environment after ``conda run -n <env>`` activation.
|
||||
|
||||
``conda run`` runs the env's activate.d/*.sh scripts (which set
|
||||
PYTHONPATH/LD_LIBRARY_PATH and may prepend PATH) in addition to the
|
||||
normal PATH/CONDA_* bookkeeping — more complete than just pointing
|
||||
PATH at env/bin.
|
||||
"""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
["conda", "run", "-n", runtime_env, "python", "-c",
|
||||
"import os,json;print(json.dumps(dict(os.environ)))"],
|
||||
capture_output=True, text=True, timeout=20,
|
||||
)
|
||||
except Exception:
|
||||
return {}
|
||||
if proc.returncode != 0:
|
||||
return {}
|
||||
try:
|
||||
data = json.loads(proc.stdout.strip())
|
||||
except ValueError:
|
||||
return {}
|
||||
if not isinstance(data, dict):
|
||||
return {}
|
||||
data.pop("_", None)
|
||||
return data
|
||||
|
||||
|
||||
class LightweightExtensionService:
|
||||
"""A small state machine that delegates all conversion/runtime facts to Phase A."""
|
||||
|
||||
@@ -119,6 +148,9 @@ class LightweightExtensionService:
|
||||
env["PATH"] = f"{conda_bin}:{existing_path}" if existing_path else conda_bin
|
||||
env["CONDA_PREFIX"] = conda_prefix
|
||||
env["CONDA_DEFAULT_ENV"] = runtime_env
|
||||
# 用 conda run 的官方激活结果覆盖:activate.d 设置的 PYTHONPATH/LD_LIBRARY_PATH
|
||||
# 以及 PATH 前置(如 MCScanX)都会被正确带进来;失败时保留上面的手动设置作为兜底。
|
||||
env.update(_conda_activated_env(runtime_env))
|
||||
return env
|
||||
|
||||
def _deepseek_candidate(self, facts: dict[str, Any], raw_code: Path) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
|
||||
@@ -700,11 +700,14 @@ def build_report_data(
|
||||
),
|
||||
]
|
||||
sections.extend(stage_sections)
|
||||
# The integrated conclusion has a fixed evidence contract. Do not take the
|
||||
# first arbitrary observations because plan ordering can displace one of
|
||||
# the three core facts requested for delivery.
|
||||
integrated_observations = [
|
||||
section.observations[0]
|
||||
for section in stage_sections
|
||||
if section.observations
|
||||
][:3]
|
||||
stage_sections_by_name[stage].observations[0]
|
||||
for stage in ("identify", "phylogeny", "reference_classification")
|
||||
if stage in stage_sections_by_name and stage_sections_by_name[stage].observations
|
||||
]
|
||||
sections.extend(
|
||||
[
|
||||
ReportSection(
|
||||
|
||||
@@ -0,0 +1,753 @@
|
||||
# gfa_core/i18n.py — 后端共享 i18n 模块
|
||||
"""
|
||||
共享翻译函数和字典,供后端所有模块使用。
|
||||
|
||||
Usage:
|
||||
from gfa_core.i18n import t, I18N_MAP
|
||||
|
||||
msg = t("session-created", lang) # 简单查找
|
||||
msg = t("stage-missing-pep", lang, stage=stage, sp=sp) # 参数替换
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# ── 翻译字典 ──────────────────────────────────────────
|
||||
# 格式:{ "key": {"zh": "中文文本", "en": "English text"} }
|
||||
# 新增条目按命名规范添加(见文件末尾规范注释)
|
||||
|
||||
I18N_MAP: dict[str, dict[str, str]] = {
|
||||
# ── 通用 ──
|
||||
"status-completed": {"zh": "✅ 所有分析阶段已完成!", "en": "✅ All analysis stages completed!"},
|
||||
"status-auto-plan": {"zh": "📋 自动规划剩余分析阶段", "en": "📋 Auto-planned remaining analysis stages"},
|
||||
"status-completed-label": {"zh": "**已完成**", "en": "**Completed**"},
|
||||
"status-none": {"zh": "无", "en": "None"},
|
||||
"status-pending": {"zh": "**待执行**", "en": "**Pending**"},
|
||||
"status-stage-count": {"zh": "个阶段)", "en": "stage(s))"},
|
||||
"confirm-execute-stages": {"zh": "请确认是否按顺序执行以上分析阶段?", "en": "Confirm to execute the above analysis stages in order?"},
|
||||
|
||||
# ── session / 会话 ──
|
||||
"session-created": {"zh": "会话创建成功", "en": "Conversation created"},
|
||||
"session-message-added": {"zh": "消息添加成功", "en": "Message added"},
|
||||
"session-title-updated": {"zh": "标题更新成功", "en": "Title updated"},
|
||||
"session-deleted": {"zh": "会话已删除", "en": "Conversation deleted"},
|
||||
"session-archived": {"zh": "会话已归档", "en": "Conversation archived"},
|
||||
"session-linked": {"zh": "已关联到分析任务", "en": "Linked to analysis task"},
|
||||
"session-not-found": {"zh": "会话不存在", "en": "Conversation not found"},
|
||||
"session-store-not-init": {"zh": "会话存储尚未初始化", "en": "Conversation store not initialized"},
|
||||
"session-user-not-found": {"zh": "用户不存在", "en": "User not found"},
|
||||
"session-invalid-pagination": {"zh": "无效的消息分页参数", "en": "Invalid message pagination"},
|
||||
|
||||
# ── 通用 HTTPException 模板 ──
|
||||
"err-create-session": {"zh": "创建会话失败: {exc}", "en": "Failed to create conversation: {exc}"},
|
||||
"err-add-message": {"zh": "添加消息失败: {exc}", "en": "Failed to add message: {exc}"},
|
||||
"err-list-conversations": {"zh": "获取会话列表失败: {exc}", "en": "Failed to list conversations: {exc}"},
|
||||
"err-get-messages": {"zh": "获取消息失败: {exc}", "en": "Failed to get messages: {exc}"},
|
||||
"err-update-title": {"zh": "更新标题失败: {exc}", "en": "Failed to update title: {exc}"},
|
||||
"err-delete-session": {"zh": "删除会话失败: {exc}", "en": "Failed to delete conversation: {exc}"},
|
||||
"err-archive-session": {"zh": "归档会话失败: {exc}", "en": "Failed to archive conversation: {exc}"},
|
||||
"err-link-job": {"zh": "关联任务失败: {exc}", "en": "Failed to link job: {exc}"},
|
||||
|
||||
# ── guidance 阶段指导(迁移自 orchestrator._MSG) ──
|
||||
"guidance-motif": {"zh": "请提供基因家族蛋白序列文件(.fa/.fasta),用于MEME保守基序预测。",
|
||||
"en": "Please provide gene family protein sequence file (.fa/.fasta) for MEME conserved motif prediction."},
|
||||
"guidance-genome-prepare": {"zh": "请提供config.xlsx及基因组/GFF/CDS/PEP等原始文件,用于完整数据准备。",
|
||||
"en": "Please provide config.xlsx and genome/GFF/CDS/PEP files for complete data preparation."},
|
||||
"guidance-domain": {"zh": "请提供基因家族蛋白序列文件,用于Pfam/InterPro结构域注释。",
|
||||
"en": "Please provide gene family protein sequence file for Pfam/InterPro domain annotation."},
|
||||
"guidance-structure": {"zh": "请提供基因家族的GFF注释文件和基因组序列。",
|
||||
"en": "Please provide gene family GFF annotation file and genome sequence."},
|
||||
"guidance-promoter": {"zh": "请提供基因家族成员的启动子序列(上游2000bp)或全基因组序列+GFF。",
|
||||
"en": "Please provide gene family promoter sequences (upstream 2000bp) or whole genome + GFF."},
|
||||
"guidance-synteny": {"zh": "请提供物种基因组序列(.fasta)和GFF注释文件。",
|
||||
"en": "Please provide species genome sequence (.fasta) and GFF annotation file."},
|
||||
"guidance-interspecies-synteny": {"zh": "请提供多个物种的基因组序列和GFF注释文件,以及物种间的同源关系。",
|
||||
"en": "Please provide genome sequences and GFF annotation files for multiple species, along with inter-species orthology."},
|
||||
"guidance-karyotype": {"zh": "请提供染色体长度信息和基因位置信息。",
|
||||
"en": "Please provide chromosome length and gene position information."},
|
||||
"guidance-ppi": {"zh": "请提供拟南芥同源基因映射文件(ortholog_pairs.txt,格式:拟南芥ID\\t苦荞ID),用于STRING数据库互作预测。如未提供,请先运行同源基因识别分析。",
|
||||
"en": "Please provide Arabidopsis ortholog mapping file (ortholog_pairs.txt, format: AtID\\tTargetID) for STRING interaction prediction. If unavailable, run ortholog identification first."},
|
||||
"guidance-ortholog": {"zh": "请提供目标物种和参考物种(如拟南芥At)的蛋白序列文件,用于同源基因识别。",
|
||||
"en": "Please provide target and reference species (e.g., Arabidopsis At) protein FASTA for ortholog identification."},
|
||||
"guidance-kaks": {"zh": "请提供物种间同源基因对(ortholog pairs,格式:基因A\\t基因B)或共线性分析结果。",
|
||||
"en": "Please provide inter-species ortholog pairs (format: GeneA\\tGeneB) or synteny analysis results."},
|
||||
"guidance-expression": {"zh": "请提供RNA-seq表达矩阵(FPKM/TPM/Count)或原始测序数据。",
|
||||
"en": "Please provide RNA-seq expression matrix (FPKM/TPM/Count) or raw sequencing data."},
|
||||
"guidance-enrichment": {"zh": "请提供差异表达基因列表(DEGs)或目标基因集。",
|
||||
"en": "Please provide differentially expressed gene list (DEGs) or target gene set."},
|
||||
"guidance-pep": {"zh": "请提供基因家族蛋白序列文件,用于理化性质分析。",
|
||||
"en": "Please provide gene family protein sequence file for physicochemical property analysis."},
|
||||
"guidance-combo": {"zh": "请提供基因家族HMM文件和蛋白序列文件,将自动执行标准11阶段分析。",
|
||||
"en": "Please provide gene family HMM file and protein FASTA; standard 11-stage analysis will be executed automatically."},
|
||||
"guidance-rna-seq-qc": {"zh": "请提供RNA-seq原始测序数据(FASTQ格式)。",
|
||||
"en": "Please provide RNA-seq raw sequencing data (FASTQ format)."},
|
||||
"guidance-rna-seq-mapping": {"zh": "请提供质控后的clean reads(FASTQ格式)和参考基因组。",
|
||||
"en": "Please provide QC-filtered clean reads (FASTQ format) and reference genome."},
|
||||
"guidance-rna-seq-quantification": {"zh": "请提供比对后的BAM文件或参考基因组注释GFF。",
|
||||
"en": "Please provide aligned BAM file or reference genome annotation GFF."},
|
||||
"guidance-rnaseq-sample-qc": {"zh": "请提供featureCounts计数矩阵和样本分组表,用于VST、PCA、样本相关性和聚类。",
|
||||
"en": "Please provide featureCounts count matrix and sample grouping table for VST, PCA, sample correlation and clustering."},
|
||||
"guidance-trend": {"zh": "请提供多时相表达矩阵(如不同发育阶段/处理时间点)。",
|
||||
"en": "Please provide multi-temporal expression matrix (e.g., different developmental stages/treatment time points)."},
|
||||
"guidance-rnaseq-wgcna": {"zh": "请提供VST/log2表达矩阵或count矩阵和样本性状表,用于共表达网络构建。",
|
||||
"en": "Please provide VST/log2 expression matrix or count matrix and sample trait table for co-expression network construction."},
|
||||
"guidance-rnaseq-go-enrich": {"zh": "请提供差异表达结果和gene2go背景文件,或先执行背景注释。",
|
||||
"en": "Please provide differential expression results and gene2go background file, or run background annotation first."},
|
||||
"guidance-rnaseq-kegg-enrich": {"zh": "请提供差异表达结果和gene2ko背景文件,或先执行背景注释。",
|
||||
"en": "Please provide differential expression results and gene2ko background file, or run background annotation first."},
|
||||
"guidance-rnaseq-fgsea": {"zh": "请提供差异表达排序统计量和gene2ko/KEGG背景,用于fgsea排序富集。",
|
||||
"en": "Please provide differential expression ranking statistics and gene2ko/KEGG background for fgsea ranked enrichment."},
|
||||
"guidance-rnaseq-candidate-family": {"zh": "请提供差异表达结果和目标物种蛋白FASTA;若已有InterProScan TSV会优先复用。",
|
||||
"en": "Please provide differential expression results and target species protein FASTA; existing InterProScan TSV will be reused if available."},
|
||||
"guidance-wgcna": {"zh": "请提供表达矩阵(样本数≥6),用于共表达网络构建。",
|
||||
"en": "Please provide expression matrix (samples ≥ 6) for co-expression network construction."},
|
||||
|
||||
# ── 意图识别 / 自然语言响应 ──
|
||||
"intent-rnaseq-execute": {"zh": "我将帮您执行RNA-seq", "en": "I will help you execute RNA-seq"},
|
||||
"intent-confirm-start": {"zh": "分析。请确认参数后开始。", "en": "analysis. Please confirm parameters to start."},
|
||||
"intent-rnaseq-async": {"zh": "RNA-seq分析将通过前端异步执行。", "en": "RNA-seq analysis will be executed asynchronously via frontend."},
|
||||
"intent-planning-v2-plan": {"zh": " 分析。PlanningAgentV2 已根据当前状态生成完整执行计划。", "en": " analysis. PlanningAgentV2 has generated a complete execution plan based on current state."},
|
||||
"intent-will-execute": {"zh": "我将帮您执行", "en": "I will help you execute"},
|
||||
"intent-analysis-suffix": {"zh": "分析。", "en": "analysis."},
|
||||
"intent-rnaseq-async-generic":{"zh": "RNA-seq {stage} 分析将通过前端异步执行。", "en": "RNA-seq {stage} analysis will be executed asynchronously via frontend."},
|
||||
"intent-upload-or-confirm": {"zh": "请上传进行", "en": "Please upload"},
|
||||
"intent-required-files": {"zh": "分析所需的文件,或确认使用系统内置数据。", "en": " required files, or confirm to use built-in data."},
|
||||
"intent-unknown": {"zh": "无法识别您的分析意图,请描述您想要执行的分析类型。", "en": "Unable to recognize your analysis intent. Please describe the type of analysis you want to perform."},
|
||||
"replan-guidance": {"zh": "该建议基于本次分析结果和最佳实践,旨在降低假阳性并补充关键证据链。请确认是否按此计划重新执行?",
|
||||
"en": "This recommendation is based on current analysis results and best practices, aiming to reduce false positives and strengthen the evidence chain. Confirm to re-execute per this plan?"},
|
||||
|
||||
# ── 文件操作 ──
|
||||
"file-not-found-in-job": {"zh": "没有在当前 job 的安全目录中找到文件:{filename}", "en": "File not found in current job directory: {filename}"},
|
||||
"file-read-failed": {"zh": "读取文件失败: {exc}", "en": "Failed to read file: {exc}"},
|
||||
"file-path-not-exist": {"zh": "路径不存在: {path}", "en": "Path does not exist: {path}"},
|
||||
"file-uploaded": {"zh": "✅ 已上传到 input/{job_id}/: {files}", "en": "✅ Uploaded to input/{job_id}/: {files}"},
|
||||
"file-select-required": {"zh": "请选择文件", "en": "Please select a file"},
|
||||
"file-no-preview": {"zh": "没有找到可预览的文件。", "en": "No previewable file found."},
|
||||
"file-binary-no-preview": {"zh": "无法预览此文件(可能是二进制/图片/PDF)", "en": "Cannot preview this file (binary/image/PDF)"},
|
||||
"file-large-truncated": {"zh": "\n\n... (文件较大,仅显示部分)", "en": "\n\n... (file too large, showing partial content)"},
|
||||
"file-loaded-count": {"zh": "已加载 {count} 个文件", "en": "Loaded {count} file(s)"},
|
||||
"file-load-failed": {"zh": "加载失败: {exc}", "en": "Load failed: {exc}"},
|
||||
"file-load-error-item": {"zh": "{path}: {exc}", "en": "{path}: {exc}"},
|
||||
|
||||
# ── 阶段名称(stage-name-{id},统一入口) ──
|
||||
"stage-name-wgs_qc": {"zh": "WGS质控", "en": "WGS QC"},
|
||||
"stage-name-wgs_index": {"zh": "WGS参考索引", "en": "WGS Reference Index"},
|
||||
"stage-name-wgs_align": {"zh": "WGS序列比对", "en": "WGS Alignment"},
|
||||
"stage-name-wgs_markdup": {"zh": "WGS去重复", "en": "WGS Mark Duplicates"},
|
||||
"stage-name-wgs_bam_qc": {"zh": "WGS BAM质控", "en": "WGS BAM QC"},
|
||||
"stage-name-wgs_haplotype_calling": {"zh": "WGS单样本变异检测", "en": "WGS Haplotype Calling"},
|
||||
"stage-name-wgs_gvcf_combine": {"zh": "WGS GVCF合并", "en": "WGS GVCF Combine"},
|
||||
"stage-name-wgs_joint_genotyping": {"zh": "WGS联合分型", "en": "WGS Joint Genotyping"},
|
||||
"stage-name-wgs_variant_select": {"zh": "WGS变异类型选择", "en": "WGS Variant Select"},
|
||||
"stage-name-wgs_variant_filter": {"zh": "WGS变异硬过滤", "en": "WGS Variant Hard Filter"},
|
||||
"stage-name-wgs_impute": {"zh": "WGS基因型填充", "en": "WGS Imputation"},
|
||||
"stage-name-wgs_maf_missing_filter": {"zh": "WGS MAF与缺失率过滤", "en": "WGS MAF & Missingness Filter"},
|
||||
"stage-name-pop_pca": {"zh": "群体PCA分析", "en": "Population PCA"},
|
||||
"stage-name-pop_phylogeny": {"zh": "群体系统树分析", "en": "Population Phylogeny"},
|
||||
"stage-name-pop_structure": {"zh": "群体结构分析", "en": "Population Structure"},
|
||||
"stage-name-pop_ld_decay": {"zh": "群体LD衰减分析", "en": "Population LD Decay"},
|
||||
"stage-name-pop_snp_density": {"zh": "SNP密度分析", "en": "SNP Density"},
|
||||
"stage-name-gwas_fmt": {"zh": "GWAS VCF排序与HapMap转换", "en": "GWAS VCF Sort & HapMap Convert"},
|
||||
"stage-name-gwas_kinship": {"zh": "GWAS TASSEL亲缘关系矩阵", "en": "GWAS TASSEL Kinship Matrix"},
|
||||
"stage-name-gwas_trait": {"zh": "GWAS表型数据准备", "en": "GWAS Trait Data Preparation"},
|
||||
"stage-name-gwas_cov": {"zh": "GWAS群体结构协变量准备", "en": "GWAS Population Covariate Preparation"},
|
||||
"stage-name-gwas_mlm": {"zh": "GWAS TASSEL MLM/CMLM", "en": "GWAS TASSEL MLM/CMLM"},
|
||||
"stage-name-gwas_plot": {"zh": "GWAS MLM结果绘图", "en": "GWAS MLM Result Plot"},
|
||||
"stage-name-gwas_emmax": {"zh": "GWAS EMMAX关联分析", "en": "GWAS EMMAX Association"},
|
||||
"stage-name-gwas_emplot": {"zh": "GWAS EMMAX结果绘图", "en": "GWAS EMMAX Result Plot"},
|
||||
"stage-name-gwas_glm": {"zh": "GWAS TASSEL GLM", "en": "GWAS TASSEL GLM"},
|
||||
"stage-name-identify": {"zh": "基因家族鉴定", "en": "Gene Family Identification"},
|
||||
"stage-name-identify_refine": {"zh": "修订鉴定成员", "en": "Refine Identify Members"},
|
||||
"stage-name-subfamily_confirm": {"zh": "确认亚家族分类", "en": "Confirm Subfamily Classification"},
|
||||
"stage-name-phylogeny": {"zh": "系统发育分析", "en": "Phylogenetic Analysis"},
|
||||
"stage-name-reference_classification": {"zh": "参考亚家族分类", "en": "Reference Subfamily Classification"},
|
||||
"stage-name-motif": {"zh": "Motif分析", "en": "Motif Analysis"},
|
||||
"stage-name-domain": {"zh": "结构域分析", "en": "Domain Analysis"},
|
||||
"stage-name-structure": {"zh": "基因结构分析", "en": "Gene Structure Analysis"},
|
||||
"stage-name-promoter": {"zh": "启动子分析", "en": "Promoter Analysis"},
|
||||
"stage-name-tfbs": {"zh": "TF结合位点预测", "en": "TF Binding Site Prediction"},
|
||||
"stage-name-synteny": {"zh": "种内共线性分析", "en": "Intra-species Synteny"},
|
||||
"stage-name-karyotype": {"zh": "核型分析", "en": "Karyotype Analysis"},
|
||||
"stage-name-interspecies_synteny": {"zh": "物种间共线性", "en": "Inter-species Synteny"},
|
||||
"stage-name-expression": {"zh": "表达分析", "en": "Expression Analysis"},
|
||||
"stage-name-kaks": {"zh": "Ka/Ks分析", "en": "Ka/Ks Analysis"},
|
||||
"stage-name-ppi": {"zh": "PPI分析", "en": "PPI Analysis"},
|
||||
"stage-name-pep": {"zh": "蛋白特征分析", "en": "Protein Feature Analysis"},
|
||||
"stage-name-ortholog": {"zh": "直系同源基因鉴定", "en": "Ortholog Identification"},
|
||||
"stage-name-msa": {"zh": "多序列比对", "en": "Multiple Sequence Alignment"},
|
||||
"stage-name-enrichment": {"zh": "富集分析", "en": "Enrichment Analysis"},
|
||||
"stage-name-combo": {"zh": "组合图", "en": "Combination Plot"},
|
||||
"stage-name-gene_family_report": {"zh": "基因家族结果报告", "en": "Gene Family Report"},
|
||||
"stage-name-genome_prepare": {"zh": "基因组准备", "en": "Genome Preparation"},
|
||||
"stage-name-ncbi_fetch_sequences": {"zh": "NCBI序列下载", "en": "NCBI Sequence Download"},
|
||||
"stage-name-sequence_extract_by_id": {"zh": "按ID提取序列", "en": "Extract Sequences by ID"},
|
||||
"stage-name-msa_region_plot": {"zh": "MSA区间绘图", "en": "MSA Region Plot"},
|
||||
"stage-name-synteny_plot_revise": {"zh": "种内共线性Circos重绘", "en": "Intra-species Synteny Circos Revise"},
|
||||
"stage-name-promoter_plot_revise": {"zh": "启动子自定义绘图", "en": "Promoter Custom Plot"},
|
||||
"stage-name-prepare_organize": {"zh": "数据文件整理", "en": "Organize Data Files"},
|
||||
"stage-name-prepare_normalize_gff": {"zh": "GFF ID规范化", "en": "GFF ID Normalization"},
|
||||
"stage-name-prepare_longest_transcript": {"zh": "提取最长转录本", "en": "Extract Longest Transcript"},
|
||||
"stage-name-prepare_extract_pep": {"zh": "从GFF提取PEP", "en": "Extract PEP from GFF"},
|
||||
"stage-name-prepare_extract_cds": {"zh": "从GFF提取CDS", "en": "Extract CDS from GFF"},
|
||||
"stage-name-prepare_filter_fasta": {"zh": "按ID提取FASTA", "en": "Extract FASTA by ID"},
|
||||
"stage-name-prepare_simplify_ncbi": {"zh": "NCBI ID简化", "en": "Simplify NCBI IDs"},
|
||||
"stage-name-rnaseq_qc": {"zh": "RNA-seq质控", "en": "RNA-seq QC"},
|
||||
"stage-name-rnaseq_align": {"zh": "RNA-seq比对", "en": "RNA-seq Alignment"},
|
||||
"stage-name-rnaseq_quant": {"zh": "RNA-seq定量", "en": "RNA-seq Quantification"},
|
||||
"stage-name-rnaseq_sample_qc": {"zh": "RNA-seq样本质量评估", "en": "RNA-seq Sample QC"},
|
||||
"stage-name-rnaseq_de": {"zh": "差异表达分析(自动生成FPKM表达图)", "en": "Differential Expression (auto FPKM plot)"},
|
||||
"stage-name-rnaseq_wgcna": {"zh": "RNA-seq WGCNA分析", "en": "RNA-seq WGCNA Analysis"},
|
||||
"stage-name-annotation": {"zh": "背景注释", "en": "Background Annotation"},
|
||||
"stage-name-annotation_go": {"zh": "GO背景自动注释", "en": "GO Background Auto-Annotation"},
|
||||
"stage-name-annotation_kegg": {"zh": "KEGG/KO背景自动注释", "en": "KEGG/KO Background Auto-Annotation"},
|
||||
"stage-name-rnaseq_go_enrich": {"zh": "RNA-seq GO富集分析", "en": "RNA-seq GO Enrichment"},
|
||||
"stage-name-rnaseq_kegg_enrich": {"zh": "RNA-seq KEGG富集分析", "en": "RNA-seq KEGG Enrichment"},
|
||||
"stage-name-rnaseq_enrich": {"zh": "RNA-seq富集分析", "en": "RNA-seq Enrichment"},
|
||||
"stage-name-rnaseq_fgsea": {"zh": "RNA-seq fgsea分析", "en": "RNA-seq fgsea Analysis"},
|
||||
"stage-name-rnaseq_candidate_family": {"zh": "RNA-seq候选家族筛选", "en": "RNA-seq Candidate Family Screening"},
|
||||
|
||||
# ── 任务状态 ──
|
||||
"task-waiting-resource": {"zh": "等待资源可用", "en": "Waiting for resources"},
|
||||
"task-waiting-dots": {"zh": "等待资源可用...", "en": "Waiting for resources..."},
|
||||
"task-executing-stage": {"zh": "正在执行 {stage}...", "en": "Executing {stage}..."},
|
||||
"task-submitted-celery": {"zh": "任务已提交到 Celery,等待 worker 获取资源", "en": "Task submitted to Celery, waiting for worker"},
|
||||
"task-queued": {"zh": "任务已进入资源调度队列", "en": "Task entered resource scheduling queue"},
|
||||
"task-submitted": {"zh": "任务已提交,等待 Celery worker 接收", "en": "Task submitted, waiting for Celery worker"},
|
||||
"task-running": {"zh": "任务运行中", "en": "Task running"},
|
||||
"task-completed": {"zh": "任务完成", "en": "Task completed"},
|
||||
"task-failed": {"zh": "任务失败", "en": "Task failed"},
|
||||
"task-not-found": {"zh": "任务不存在", "en": "Task not found"},
|
||||
"task-ended-cannot-cancel": {"zh": "任务已结束,无法取消", "en": "Task ended, cannot cancel"},
|
||||
"task-cancelled": {"zh": "任务已取消", "en": "Task cancelled"},
|
||||
"task-marked-cancel": {"zh": "任务已标记为取消", "en": "Task marked for cancellation"},
|
||||
"task-submitting-to-plantcare": {"zh": "正在提交序列到 PlantCARE...", "en": "Submitting sequences to PlantCARE..."},
|
||||
"task-running-mcscanx": {"zh": "正在进行共线性分析(MCScanX)...", "en": "Running synteny analysis (MCScanX)..."},
|
||||
"task-running-fastp": {"zh": "正在进行RNA-seq质控分析(fastp)... 这可能需要15-30分钟,请耐心等待...", "en": "Running RNA-seq QC (fastp)... this may take 15-30 minutes, please wait..."},
|
||||
"task-submitted-plantcare": {"zh": "已提交 PlantCARE,等待邮件结果(可能需要10-60分钟)...", "en": "Submitted to PlantCARE, waiting for email results (may take 10-60 min)..."},
|
||||
"task-waiting-plantcare-email": {"zh": "已提交 PlantCARE,正在等待邮件...", "en": "Submitted to PlantCARE, waiting for email..."},
|
||||
|
||||
# ── 执行状态/阶段消息 ──
|
||||
"exec-stage-failed": {"zh": "{stage} 执行失败", "en": "{stage} execution failed"},
|
||||
"exec-stage-timeout": {"zh": "{stage} 等待超时,最后状态: {last_status}", "en": "{stage} timed out, last status: {last_status}"},
|
||||
"exec-stage-submit-failed": {"zh": "{stage} 提交失败", "en": "{stage} submission failed"},
|
||||
"exec-ok-tree-revised": {"zh": "✅ 进化树图修订完成。", "en": "✅ Tree plot revision completed."},
|
||||
"exec-ok-synteny-circos": {"zh": "✅ 种内共线性 Circos 图修订完成。", "en": "✅ Synteny Circos plot revision completed."},
|
||||
"exec-ok-promoter-plot": {"zh": "✅ 启动子自定义绘图完成。", "en": "✅ Promoter custom plot completed."},
|
||||
"exec-ok-msa-region-plot": {"zh": "✅ MSA区间绘图完成。", "en": "✅ MSA region plot completed."},
|
||||
"exec-err-no-valid-output": {"zh": "❌ {stage} 未生成有效输出", "en": "❌ {stage} did not generate valid output"},
|
||||
"exec-ok-auto-annotation": {"zh": "✅ 自动背景注释已完成", "en": "✅ Auto background annotation completed"},
|
||||
|
||||
# ── execution context / preflight ──
|
||||
"preflight-missing-subfamily": {"zh": "执行前上下文检查失败:缺少确认或上传的亚家族标签", "en": "Pre-execution context check failed: missing confirmed or uploaded subfamily labels"},
|
||||
"preflight-semantic-failed": {"zh": "执行前语义预检失败", "en": "Pre-execution semantic preflight failed"},
|
||||
|
||||
# ── 清理 ──
|
||||
"cleanup-no-old-dirs": {"zh": "✅ 清理完成:未发现超过 {days} 天的任务目录", "en": "✅ Cleanup done: no directories older than {days} days found"},
|
||||
"cleanup-deleted": {"zh": "✅ 清理完成,已删除:\n", "en": "✅ Cleanup done, deleted:\n"},
|
||||
|
||||
# ── 亚家族 ──
|
||||
"subfamily-label-mismatch": {"zh": "{filename} 未覆盖当前 {species}/{family} 目标成员集合。", "en": "{filename} does not cover the current {species}/{family} target member set."},
|
||||
"subfamily-reconfirm": {"zh": "请重新确认亚家族标签,或检查当前成员集合是否已更新。", "en": "Please re-confirm subfamily labels, or check if the current member set has been updated."},
|
||||
"subfamily-missing-ref": {"zh": "缺少参考物种 {species}.{family}.subfamily。", "en": "Missing reference species {species}.{family}.subfamily."},
|
||||
"subfamily-upload-ref": {"zh": "请上传 {species}.{family}.subfamily 作为参考亚家族标签。", "en": "Please upload {species}.{family}.subfamily as reference subfamily labels."},
|
||||
"subfamily-no-identify": {"zh": "{species}/{family} 尚未完成基因家族成员鉴定,不能进行亚家族确认或传统基因家族分析。", "en": "{species}/{family} has not completed gene family member identification; subfamily confirmation or traditional gene family analysis cannot proceed."},
|
||||
"subfamily-run-identify": {"zh": "请先执行 identify。", "en": "Please run identify first."},
|
||||
"subfamily-predicted-unconfirmed": {"zh": "检测到 {species}/{family} 已有亚家族分类预测文件,但还没有保存确认标签。", "en": "Subfamily classification prediction file exists for {species}/{family}, but confirmed labels have not been saved."},
|
||||
"subfamily-open-confirm": {"zh": "请先打开亚家族确认表并保存。", "en": "Please open the subfamily confirmation table and save it."},
|
||||
"subfamily-not-classified": {"zh": "{species}/{family} 尚未进行目标物种+参考物种的亚家族分类。", "en": "{species}/{family} has not undergone target+reference species subfamily classification."},
|
||||
"subfamily-run-classification": {"zh": "请先执行亚家族分类并确认标签,或上传目标物种亚家族文件。", "en": "Please run subfamily classification and confirm labels, or upload target species subfamily file."},
|
||||
|
||||
# ── planning_v2 ──
|
||||
"planning-pop-vcf-needed": {"zh": "{stage_id} 需要在执行确认弹窗中选择群体 VCF", "en": "{stage_id} requires selecting a population VCF in the confirmation dialog"},
|
||||
"planning-pop-vcf-confirm": {"zh": "请确认群体VCF绝对路径;执行时会重新校验文件和科学身份。", "en": "Please confirm the absolute path of the population VCF; it will be re-validated during execution."},
|
||||
"planning-gwas-fmt-vcf-needed": {"zh": "gwas_fmt 需要在执行确认弹窗中选择外部 VCF", "en": "gwas_fmt requires selecting an external VCF in the confirmation dialog"},
|
||||
"planning-gwas-fmt-vcf-confirm": {"zh": "请在执行确认弹窗填写外部 VCF 的绝对路径;提交执行时仍会严格校验文件及 VCF 内容。", "en": "Please provide the absolute path of the external VCF in the confirmation dialog; strict validation will be enforced upon submission."},
|
||||
"planning-preflight-failed": {"zh": "❌ 预检验证失败: {error_msg}", "en": "❌ Preflight validation failed: {error_msg}"},
|
||||
"planning-preflight-passed": {"zh": "✅ 预检验证通过", "en": "✅ Preflight validation passed"},
|
||||
"planning-preflight-warnings": {"zh": " ({count}个警告)", "en": " ({count} warning(s))"},
|
||||
"planning-preflight-plan-summary": {"zh": " | 已生成执行计划: {stages}个阶段, 预估{time}分钟", "en": " | Execution plan generated: {stages} stage(s), estimated {time} minutes"},
|
||||
"planning-preflight-learning": {"zh": " | 已应用{count}个阶段的学习推荐参数", "en": " | Applied learning recommendations for {count} stage(s)"},
|
||||
"planning-single-wgs-manifest": {"zh": "确认该任务只有一套完整的 WGS 上游 Manifest", "en": "Confirm that this job has only one complete set of WGS upstream manifests"},
|
||||
"planning-context-index-failed": {"zh": "项目上下文索引构建失败: {exc}", "en": "Project context index build failed: {exc}"},
|
||||
"planning-context-index-fallback": {"zh": "将退回旧版文件检查逻辑;如重复误报缺输入,请检查 job 目录结构和 manifest", "en": "Falling back to legacy file check; if false positives persist, check job directory structure and manifests"},
|
||||
"planning-stage-not-executable": {"zh": "阶段 {stage_id} 暂未接入标准执行器", "en": "Stage {stage_id} is not yet registered with a standard executor"},
|
||||
"planning-stage-not-executable-sugg": {"zh": "请选择已接入执行器的分析阶段,或先为该阶段补充 pipeline_backend.run_stage 分支和标准 manifest", "en": "Please select a supported analysis stage, or add a pipeline_backend.run_stage branch and standard manifest for this stage first"},
|
||||
"planning-stage-completed": {"zh": "阶段 {stage_id} 已完成", "en": "Stage {stage_id} completed"},
|
||||
"planning-stage-skip": {"zh": "该阶段已完成,可跳过执行", "en": "Stage already completed; execution can be skipped"},
|
||||
"planning-dep-incomplete": {"zh": "依赖阶段 {dep} 未完成 (状态: {status})", "en": "Dependency {dep} incomplete (status: {status})"},
|
||||
"planning-dep-auto-add": {"zh": "将自动添加 {dep} 阶段到执行计划", "en": "Will auto-add {dep} to execution plan"},
|
||||
"planning-input-gwas-fmt-resolved": {"zh": "gwas_fmt 输入 external_vcf 已解析", "en": "gwas_fmt input external_vcf resolved"},
|
||||
"planning-input-resolved": {"zh": "{stage_id} 输入 {role} 已解析", "en": "{stage_id} input {role} resolved"},
|
||||
"planning-input-upstream": {"zh": "{stage_id} 所需输入 {role} ({kind}) 将由上游阶段 {producer} 生成", "en": "{stage_id} input {role} ({kind}) will be generated by upstream stage {producer}"},
|
||||
"planning-input-upstream-sugg": {"zh": "规划器会先执行 {producer},再执行 {stage_id}。", "en": "Planner will run {producer} before {stage_id}."},
|
||||
"planning-input-missing": {"zh": "缺少 {stage_id} 所需输入: {role} ({kind})", "en": "Missing required input for {stage_id}: {role} ({kind})"},
|
||||
"planning-input-missing-type": {"zh": "缺少 {type} 类型输入文件", "en": "Missing {type} type input file"},
|
||||
"planning-input-multi-target-spp": {"zh": "{stage_id} 检测到多个可用目标物种蛋白,需要确认目标物种", "en": "{stage_id} detected multiple available target species proteins; target species must be confirmed"},
|
||||
"planning-input-multi-target-sugg": {"zh": "请在执行确认弹窗中选择 target species;系统不应在 Multi prefix 下静默任选一个物种。", "en": "Please select target species in confirmation dialog; the system should not silently pick one under a multi-species prefix."},
|
||||
"planning-wgs-run-upstream": {"zh": "先执行上游 WGS 阶段,再重新绑定其 Manifest", "en": "Run upstream WGS stages first, then re-bind their manifests"},
|
||||
"planning-wgs-fix-manifest": {"zh": "修复 wgs_qc 或 wgs_index 权威 Manifest 后重试", "en": "Fix wgs_qc or wgs_index authoritative manifests and retry"},
|
||||
"planning-wgs-align-resolved": {"zh": "wgs_align 输入已从权威 Manifest 解析", "en": "wgs_align inputs resolved from authoritative manifests"},
|
||||
"planning-low-cpu": {"zh": "CPU 核数较少: {count}", "en": "Low CPU core count: {count}"},
|
||||
"planning-low-cpu-sugg": {"zh": "建议至少 4 核 CPU", "en": "Recommend at least 4 CPU cores"},
|
||||
"planning-low-disk": {"zh": "磁盘空间不足: {gb:.1f} GB", "en": "Insufficient disk space: {gb:.1f} GB"},
|
||||
"planning-low-disk-sugg": {"zh": "建议至少 10 GB 磁盘空间", "en": "Recommend at least 10 GB disk space"},
|
||||
"planning-learning-recommend": {"zh": "基于 {count} 次成功执行推荐", "en": "Recommendations based on {count} successful executions"},
|
||||
"planning-learning-insufficient": {"zh": "历史数据不足,使用默认参数", "en": "Insufficient historical data; using default parameters"},
|
||||
"planning-desc-tool": {"zh": "执行工具型任务:", "en": "Execute utility tasks: "},
|
||||
"planning-desc-repair": {"zh": "执行受控修复任务:{stages}", "en": "Execute controlled repair: {stages}"},
|
||||
"planning-desc-rnaseq": {"zh": "执行RNA-seq的{stages}分析", "en": "Execute RNA-seq {stages} analysis"},
|
||||
"planning-desc-rnaseq-de": {"zh": ";差异分析成功后会自动触发 Expression(使用FPKM) 绘图", "en": "; upon successful DE, Expression (FPKM-based) plotting will be auto-triggered"},
|
||||
"planning-desc-gene-family": {"zh": "执行{gene_family}基因家族的{stages}分析", "en": "Execute {gene_family} gene family {stages} analysis"},
|
||||
"planning-desc-unsupported": {"zh": "以下阶段暂未接入标准执行器: {stages}", "en": "These stages are not yet registered with standard executors: {stages}"},
|
||||
"planning-desc-all-done": {"zh": "所有阶段已完成", "en": "All stages completed"},
|
||||
"planning-input-target-proteome": {"zh": "请在 config.xlsx 中配置目标物种 pep,或上传目标物种蛋白 FASTA;若只有 genome+gff,请先执行数据准备。", "en": "Configure target species pep in config.xlsx, or upload target species protein FASTA; if only genome+gff is available, run data preparation first."},
|
||||
"planning-input-family-members-pep": {"zh": "请先完成 identify,或确认 final/{prefix}.final_members.pep.fa / scoped phylogeny 输入存在。", "en": "Complete identify first, or confirm final/{prefix}.final_members.pep.fa / scoped phylogeny input exists."},
|
||||
"planning-input-family-members-id": {"zh": "请先完成 identify,或提供 final_members.id。", "en": "Complete identify first, or provide final_members.id."},
|
||||
"planning-input-target-gff": {"zh": "请在 config.xlsx 中配置目标物种 GFF/GFF3,或先执行数据准备生成 {Species}_final.gff3。", "en": "Configure target species GFF/GFF3 in config.xlsx, or run data preparation to generate {Species}_final.gff3."},
|
||||
"planning-input-target-genome": {"zh": "请在 config.xlsx 中配置目标物种 genome FASTA,或先执行数据准备生成 {Species}.genome.fasta。", "en": "Configure target species genome FASTA in config.xlsx, or run data preparation to generate {Species}.genome.fasta."},
|
||||
"planning-input-tf-motif-db": {"zh": "请确认项目 data/tfbs/jaspar2026/combined/JASPAR2026_CORE_plants_non-redundant.meme 存在,或配置 PlantTFDB/JASPAR MEME 资源。", "en": "Confirm that data/tfbs/jaspar2026/combined/JASPAR2026_CORE_plants_non-redundant.meme exists, or configure PlantTFDB/JASPAR MEME resources."},
|
||||
"planning-input-tree": {"zh": "请先完成 phylogeny,或提供 final/{prefix}.tree.nwk。", "en": "Complete phylogeny first, or provide final/{prefix}.tree.nwk."},
|
||||
"planning-input-pfam-db": {"zh": "请确认项目 data/hmm/Pfam-A.hmm 及 hmmpress 索引存在。", "en": "Confirm that data/hmm/Pfam-A.hmm and hmmpress indexes exist."},
|
||||
# ── diagnosis ──
|
||||
"diag-gene2go-mismatch": {"zh": "当前 gene2go 背景与 DE 基因 ID 体系不一致。应提示用户确认本地自动注释,生成与当前蛋白/DE ID 匹配的 gene2go 后再做 GO 富集。", "en": "Current gene2go background does not match the DE gene ID system. Prompt the user to confirm local auto-annotation to generate gene2go matching current protein/DE IDs before GO enrichment."},
|
||||
"diag-max-iterations": {"zh": "已达到最大迭代次数({max_iterations}),停止自动修复。", "en": "Maximum iterations ({max_iterations}) reached; stopping auto-repair."},
|
||||
"diag-complete": {"zh": "🔍 诊断完成: {error_type}", "en": "🔍 Diagnosis complete: {error_type}"},
|
||||
"diag-unknown-error": {"zh": "无法自动诊断,请检查错误日志", "en": "Unable to auto-diagnose; please check error logs"},
|
||||
"diag-unknown-reason": {"zh": "未知错误不能自动修复,需要用户查看日志后确认下一步。", "en": "Unknown error cannot be auto-repaired; user needs to check logs and confirm next steps."},
|
||||
"diag-genome-prepare-running": {"zh": "genome_prepare 当前仍在运行,拒绝执行 job-local 修复以避免覆盖正在生成的产物。", "en": "genome_prepare is still running; refusing job-local fix to avoid overwriting in-progress outputs."},
|
||||
"diag-no-safe-fix": {"zh": "没有可安全副本修复的问题;需要用户手动处理或补充确认信息。", "en": "No issues can be safely fixed via copy; manual handling or additional confirmation is needed."},
|
||||
"diag-fix-prepared": {"zh": "已准备当前 job 的修复副本;尚未执行。", "en": "Fix copy prepared for current job; not yet executed."},
|
||||
"diag-stage-running-refuse": {"zh": "{job_id}/{stage} 当前仍在运行,拒绝执行修复副本以避免覆盖正在生成的产物。", "en": "{job_id}/{stage} is still running; refusing fix copy to avoid overwriting in-progress outputs."},
|
||||
"diag-cannot-build-fix-cmd": {"zh": "无法构建修复副本执行命令", "en": "Unable to build fix copy execution command"},
|
||||
"diag-fix-copy-success": {"zh": "修复副本执行完成", "en": "Fix copy execution completed"},
|
||||
"diag-fix-copy-failed": {"zh": "修复副本执行失败", "en": "Fix copy execution failed"},
|
||||
"diag-pep-not-found": {"zh": "未找到可清理的蛋白 FASTA:final/{prefix}.final_members.pep.fa", "en": "Cleanable protein FASTA not found: final/{prefix}.final_members.pep.fa"},
|
||||
"diag-pep-stop-codon-removed": {"zh": "已生成去除 '*' stop marker 的 job-local 蛋白 FASTA。后续 phylogeny/motif/domain/pep 将优先使用该文件。", "en": "Generated job-local protein FASTA with '*' stop markers removed. Subsequent phylogeny/motif/domain/pep will preferentially use this file."},
|
||||
"diag-gff-overlap-report": {"zh": "已生成 GFF/成员 ID 重叠诊断报告;该问题不能静默改 ID。", "en": "GFF/member ID overlap diagnostic report generated; this issue cannot be silently fixed by renaming IDs."},
|
||||
"diag-pfam-import-failed": {"zh": "无法导入 identify Pfam fallback 工具: {exc}", "en": "Failed to import identify Pfam fallback tool: {exc}"},
|
||||
"diag-pfam-infer-failed": {"zh": "未能从 issue/config/prefix 推断 Required_Pfams;请在 config.xlsx 的 Analysis.Required_Pfams 填写 PFxxxxx,或上传 HMM。", "en": "Could not infer Required_Pfams from issue/config/prefix; please fill in PFxxxxx in Analysis.Required_Pfams in config.xlsx, or upload an HMM."},
|
||||
"diag-pfam-local-failed": {"zh": "本地 Pfam/HMM fallback 失败: {exc}", "en": "Local Pfam/HMM fallback failed: {exc}"},
|
||||
"diag-pfam-hmm-generated": {"zh": "已基于 config/family/Pfam 本地资源生成 identify 可用的 job-local 查询 HMM。", "en": "Generated job-local query HMM usable for identify based on local config/family/Pfam resources."},
|
||||
"diag-ref-prot-infer-failed": {"zh": "无法从文件名推断 Species/Family: {name}", "en": "Cannot infer Species/Family from filename: {name}"},
|
||||
"diag-ref-prot-exists": {"zh": "标准家族参考蛋白已存在: {expected}", "en": "Standard family reference protein already exists: {expected}"},
|
||||
"diag-ref-prot-created": {"zh": "已创建标准家族参考蛋白 {name} ({link_type})。", "en": "Created standard family reference protein {name} ({link_type})."},
|
||||
"diag-manifest-exists": {"zh": "Manifest 已存在: {path}", "en": "Manifest already exists: {path}"},
|
||||
"diag-manifest-stage-running": {"zh": "{job_id}/{stage} 当前仍在运行,拒绝恢复 manifest。", "en": "{job_id}/{stage} is still running; refusing manifest restore."},
|
||||
"diag-manifest-not-supported": {"zh": "暂不支持 {stage} 的 manifest 自动恢复;需要先定义该阶段核心产物契约。", "en": "Auto manifest restore for {stage} is not yet supported; core product contract must be defined first."},
|
||||
"diag-manifest-insufficient-products": {"zh": "{stage} final 核心产物不足,不能安全恢复 manifest。", "en": "{stage} final core products are insufficient; manifest cannot be safely restored."},
|
||||
"diag-manifest-restored": {"zh": "已根据 final 核心产物恢复 {stage} manifest。", "en": "Restored {stage} manifest from final core products."},
|
||||
"diag-ortholog-running": {"zh": "{job_id}/ortholog 当前仍在运行,拒绝执行 intermediate 落盘修复。", "en": "{job_id}/ortholog is still running; refusing intermediate disk fix."},
|
||||
"diag-ortholog-no-reusable": {"zh": "ortholog/ 下没有可复用的 *_pairs.txt,无法执行 intermediate 落盘修复。", "en": "No reusable *_pairs.txt found under ortholog/; intermediate disk fix cannot be executed."},
|
||||
"diag-ortholog-fix-failed": {"zh": "ortholog intermediate 落盘修复失败: {exc}", "en": "Ortholog intermediate disk fix failed: {exc}"},
|
||||
"diag-ortholog-no-manifest": {"zh": "ortholog intermediate 落盘修复未生成 manifest", "en": "Ortholog intermediate disk fix did not generate a manifest"},
|
||||
"diag-gff-empty-not-found": {"zh": "未能定位空 final GFF。", "en": "Could not locate empty final GFF."},
|
||||
"diag-gff-no-nonempty-species": {"zh": "{species} 没有可用的非空原始 GFF。", "en": "{species} has no available non-empty original GFF."},
|
||||
"diag-gff-cannot-identify-species": {"zh": "无法从诊断证据中识别物种。", "en": "Cannot identify species from diagnostic evidence."},
|
||||
"diag-gff-missing-genome": {"zh": "{species} 缺少可用于 gffread 的 genome 或非空 GFF。", "en": "{species} lacks a genome or non-empty GFF usable for gffread."},
|
||||
"diag-gff-ready-for-gffread": {"zh": "已确认可用 genome+GFF,可在确认后执行 gffread 提取 pep/cds。", "en": "Confirmed usable genome+GFF; gffread can be executed after confirmation to extract pep/cds."},
|
||||
"diag-cmd-empty": {"zh": "缺少可执行命令。请在 fix_data.cmd 中提供原阶段命令,DiagnosisAgent 会替换为 patched script。", "en": "Missing executable command. Please provide the original stage command in fix_data.cmd; DiagnosisAgent will replace it with the patched script."},
|
||||
"diag-cmd-no-script-path": {"zh": "原命令中未找到阶段脚本路径,无法保证使用 patched script 执行。", "en": "Stage script path not found in original command; cannot guarantee execution with patched script."},
|
||||
"diag-fix-notification-title": {"zh": "🔧 自动修复通知", "en": "🔧 Auto-fix notification"},
|
||||
"diag-param-changed": {"zh": "参数变更:", "en": "Parameters changed:"},
|
||||
"diag-re-executing": {"zh": "正在重新执行...", "en": "Re-executing..."},
|
||||
"diag-llm-infer-failed": {"zh": "LLM 推断 {stage} 失败根因:{category}", "en": "LLM inference for {stage} failure root cause: {category}"},
|
||||
"diag-fix-cpu-reduce": {"zh": "将CPU从{current_cpu}减少到{new_cpu}", "en": "Reduce CPU from {current_cpu} to {new_cpu}"},
|
||||
"diag-fix-params-renamed": {"zh": "修复了{count}个参数名称", "en": "Fixed {count} parameter name(s)"},
|
||||
"diag-fix-missing-deps": {"zh": "需要前置执行依赖阶段: {deps}", "en": "Prerequisite dependency stages needed: {deps}"},
|
||||
# ── interpretation evidence risks ──
|
||||
"evidence-species-incomplete": {"zh": "存在未成功完成数据准备的物种,后续分析不能视为输入完整。", "en": "Some species have not completed data preparation; downstream analysis should not be considered input-complete."},
|
||||
"evidence-config-preflight-fail": {"zh": "config.xlsx 预检未通过,物种角色或输入文件需复核。", "en": "config.xlsx preflight check failed; species roles or input files need review."},
|
||||
"evidence-candidates-lt-final": {"zh": "manifest 中 candidates={candidates} 小于 final_members={final_members},报告不得表述为从{candidates}个候选筛出{final_members}个。", "en": "Manifest candidates={candidates} is less than final_members={final_members}; report must not claim {final_members} were screened from {candidates} candidates."},
|
||||
"evidence-id-count-mismatch": {"zh": "成员ID文件计数({id_count})与manifest final_members({final_members})不一致。", "en": "Member ID file count ({id_count}) does not match manifest final_members ({final_members})."},
|
||||
"evidence-fasta-count-mismatch": {"zh": "蛋白FASTA序列数({fasta_count})与manifest final_members({final_members})不一致。", "en": "Protein FASTA sequence count ({fasta_count}) does not match manifest final_members ({final_members})."},
|
||||
"evidence-tip-count-mismatch": {"zh": "Newick tip数({tips})与manifest序列数({seq_count})不一致,系统发育图或树文件需复核。", "en": "Newick tip count ({tips}) does not match manifest sequence count ({seq_count}); tree file needs review."},
|
||||
"evidence-msa-gap-high": {"zh": "MSA gap比例为{gap:.2f}%,深层拓扑和远缘分支解释需谨慎。", "en": "MSA gap ratio is {gap:.2f}%; deep topology and distant branch interpretation should be cautious."},
|
||||
"evidence-motif-txt-empty": {"zh": "motif_prot_txt 为空,无法从原始位置表验证每个基因的motif排列。", "en": "motif_prot_txt is empty; cannot verify per-gene motif arrangement from the raw position table."},
|
||||
"evidence-motif-low-coverage": {"zh": "最保守motif覆盖率约{coverage:.1%},不宜直接声称所有成员共享该motif。", "en": "Most conserved motif coverage is ~{coverage:.1%}; should not directly claim all members share this motif."},
|
||||
"evidence-structure-partial": {"zh": "structure 阶段为 partial_success,结构结论应标记为可用但不完整。", "en": "Structure stage is partial_success; structural conclusions should be marked as usable but incomplete."},
|
||||
"evidence-structure-coverage-note": {"zh": "structure_coverage_rate={coverage},manifest 的 genes_with_structure 可能按mRNA/转录本层级计数;报告应优先采用structure.tsv的基因级计数。", "en": "structure_coverage_rate={coverage}; manifest genes_with_structure may count at mRNA/transcript level; report should use gene-level counts from structure.tsv."},
|
||||
"evidence-expression-fewer-genes": {"zh": "表达矩阵包含{expr_genes}个家族基因,少于鉴定成员{family_genes}个;缺失示例: {missing}。", "en": "Expression matrix contains {expr_genes} family genes, fewer than {family_genes} identified members; missing examples: {missing}."},
|
||||
"evidence-domain-significant-parse-error": {"zh": "manifest记录显著hits为0,但domtblout按E<=1e-5解析得到{hits}个显著hits;当前domain manifest/report统计存在解析错误,应以原始domtblout复核。", "en": "Manifest records 0 significant hits, but domtblout parsing at E<=1e-5 yields {hits} significant hits; domain manifest/report stats have parsing errors; should verify against raw domtblout."},
|
||||
"evidence-domain-zero-significant": {"zh": "hmmscan产生{total}个raw hits,但显著hits为0;结构域结论应解释阈值、数据库和identify阶段SBP HMM的一致性。", "en": "hmmscan produced {total} raw hits, but significant hits are 0; domain conclusions should explain threshold, database, and identify-stage SBP HMM consistency."},
|
||||
"evidence-domain-sbp-few": {"zh": "SBP结构域仅在{sbp_count}/{query_count}个蛋白中达到显著阈值,短序列或截短成员需要单独标注。", "en": "SBP domain reaches significance in only {sbp_count}/{query_count} proteins; short sequences or truncated members need separate annotation."},
|
||||
"evidence-ref-class-no-confirmed": {"zh": "reference_classification 未形成可确认亚家族分类;需检查参考亚家族表、参考序列ID映射和联合FASTA。", "en": "reference_classification did not produce confirmable subfamily classification; check reference subfamily table, reference sequence ID mapping, and joint FASTA."},
|
||||
"evidence-promoter-coverage": {"zh": "启动子元件覆盖{genes_with}/{family_count}个家族成员,至少{missing}个成员缺少可解释启动子元件记录。", "en": "Promoter elements cover {genes_with}/{family_count} family members; at least {missing} members lack interpretable promoter element records."},
|
||||
"evidence-promoter-core-only": {"zh": "最高频元件为TATA/CAAT等核心启动子元件,不能单独作为胁迫或激素响应结论;需结合ABRE、MYB/MYC、LTR等功能元件和表达数据解释。", "en": "Top elements are core promoter elements (TATA/CAAT); cannot be used alone as stress or hormone response evidence; combine with ABRE, MYB/MYC, LTR, and expression data."},
|
||||
"evidence-synteny-background": {"zh": "synteny manifest 的 gene_count={background} 是全基因组背景规模,不应当解读为{prefix}家族成员数。", "en": "synteny manifest gene_count={background} is genome-wide background scale; should not be interpreted as {prefix} family member count."},
|
||||
"evidence-synteny-no-pairs": {"zh": "种内共线性文件未检测到涉及{family_label}成员的基因对,不能据此声称家族扩张由片段复制驱动。", "en": "Intra-species collinearity file detected no gene pairs involving {family_label} members; cannot claim family expansion driven by segmental duplication."},
|
||||
"evidence-karyotype-no-genes": {"zh": "核型阶段未定位到家族基因,不能进行染色体分布解释。", "en": "No family genes mapped in karyotype stage; chromosome distribution interpretation is not possible."},
|
||||
"evidence-interspecies-no-target-collinearity": {"zh": "种间MCScanX collinearity中未检测到{target_label} {family_label}成员参与的共线性基因对,当前结果只能说明全基因组目标-参考共线性背景,不能直接支撑{family_label}保守共线性结论。", "en": "No collinear gene pairs involving {target_label} {family_label} members detected in inter-species MCScanX; results only show genome-wide target-reference collinearity background, cannot support {family_label} conserved collinearity."},
|
||||
"evidence-interspecies-no-direct-pair": {"zh": "虽然检测到目标或参考{family_label}成员参与collinearity,但未发现二者在同一共线性基因对中直接配对;保守共线性结论应降级为候选线索。", "en": "Target or reference {family_label} members participate in collinearity, but no direct pairing in the same collinear gene pair found; conserved collinearity should be downgraded to candidate clue."},
|
||||
"evidence-ppi-zero": {"zh": "PPI阶段报告Input Genes或Total Interactions为0,不能作为{family_label}蛋白互作证据;需修正物种映射/同源转换后重跑。", "en": "PPI stage reports Input Genes or Total Interactions as 0; cannot serve as {family_label} protein interaction evidence; fix species mapping/homology conversion and re-run."},
|
||||
"evidence-pep-pi-outlier": {"zh": "蛋白pI出现超出生物物理常见范围的数值(avg_pi={avg_pi}, max_pi={max_pi}),pI算法或字段解析需要复核,不能直接发表为理化性质结论。", "en": "Protein pI values exceed common biophysical ranges (avg_pi={avg_pi}, max_pi={max_pi}); pI algorithm or field parsing needs review; cannot be directly published as physicochemical conclusions."},
|
||||
"evidence-kaks-saturated": {"zh": "检测到Ks=99或接近99的基因对。这通常是yn00同义替换饱和/估计边界信号;原始结果应保留,但该基因对的分化时间和选择压力解释需要在深度解读中降级为谨慎解释。", "en": "Ks=99 or near-99 gene pairs detected; this is typically yn00 synonymous saturation/boundary signal; keep raw results but downgrade divergence time and selection pressure interpretation to cautious."},
|
||||
"evidence-kaks-anomalous": {"zh": "mean_ks={mean_ks:.2f}且发散时间达到异常大数量级,Ka/Ks结果很可能存在序列配对、比对或单位换算问题;不能直接解释为正选择。", "en": "mean_ks={mean_ks:.2f} and divergence time is anomalously large; Ka/Ks results likely have sequence pairing, alignment, or unit conversion issues; cannot directly interpret as positive selection."},
|
||||
"evidence-kaks-all-positive": {"zh": "所有成功基因对均表现Ka/Ks>1时,应先排除饱和替换、远缘同源和CDS比对质量问题,再讨论正选择。", "en": "When all successful gene pairs show Ka/Ks>1, first exclude saturation, distant homology, and CDS alignment quality issues before discussing positive selection."},
|
||||
"evidence-rnaseq-qc-zero-reads": {"zh": "QC manifest中所有样本raw_reads/clean_reads为0;FastQC/Cutadapt产物存在时说明流程执行完成,但reads计数字段未被解析,报告中不要据此声称测序量为0。", "en": "All sample raw_reads/clean_reads are 0 in QC manifest; FastQC/Cutadapt products exist meaning the pipeline completed, but read count fields were not parsed; do not claim zero sequencing in reports."},
|
||||
"evidence-rnaseq-align-zero": {"zh": "Align manifest中total_reads/aligned_reads为0;应优先查看HISAT2日志判断真实比对率,不能把0作为生物学或数据质量结论。", "en": "Align manifest total_reads/aligned_reads are 0; check HISAT2 logs for real alignment rate; cannot use 0 as biological or data quality conclusion."},
|
||||
"evidence-rnaseq-sample-qc-note": {"zh": "样本数为{samples},PCA/相关性可以用于质控,但离群样本判断需要结合原始QC、比对率和实验批次。", "en": "{samples} samples; PCA/correlation can be used for QC, but outlier judgment needs combined original QC, alignment rate, and experimental batch."},
|
||||
"evidence-rnaseq-wgcna-note": {"zh": "WGCNA样本数为{samples},可以作为候选模块探索,但模块-性状相关和hub基因不宜作为强因果结论。", "en": "WGCNA has {samples} samples; can be used for candidate module exploration, but module-trait correlations and hub genes should not be treated as strong causal conclusions."},
|
||||
|
||||
# ── main.py user-visible messages ──
|
||||
"main-auto-annotation-both-missing": {"zh": "当前 GO/KEGG 富集背景缺失或与 DE 基因 ID 不匹配,需要启动本地自动注释。", "en": "GO/KEGG enrichment background missing or mismatched with DE gene IDs; need to start local auto-annotation."},
|
||||
"main-auto-annotation-kegg-missing": {"zh": "当前 KEGG 富集背景缺失,需要启动本地自动注释生成 gene2ko。", "en": "KEGG enrichment background missing; need to start local auto-annotation to generate gene2ko."},
|
||||
"main-auto-annotation-go-mismatch": {"zh": "当前 GO 背景与 DE 基因 ID 不匹配,需要启动本地自动注释生成匹配的 gene2go。", "en": "GO background does not match DE gene IDs; need to start local auto-annotation to generate matching gene2go."},
|
||||
"main-auto-annotation-go-missing": {"zh": "当前 GO 富集背景缺失,需要启动本地自动注释生成 gene2go。", "en": "GO enrichment background missing; need to start local auto-annotation to generate gene2go."},
|
||||
"main-family-conflict": {"zh": "基因家族冲突:请求为 {requested},但当前 job config.xlsx 的 Analysis.Family 为 {config_family}。请修改 config 或新建 job 后再运行。", "en": "Gene family conflict: requested {requested}, but current job config.xlsx Analysis.Family is {config_family}. Please modify config or create a new job."},
|
||||
"main-family-parse-error": {"zh": "基因家族解析错误:{requested} 是物种代码,不是 gene family。", "en": "Gene family parse error: {requested} is a species code, not a gene family."},
|
||||
"main-identify-select-target": {"zh": "请选择本次 identify 要鉴定的物种;reference 仅作为参考来源,不参与目标鉴定。", "en": "Please select the species to identify in this run; reference species are for reference only and will not be identified."},
|
||||
"main-species-unresolved": {"zh": "{role} 物种名称无法从 config.xlsx 映射到 Species_Code: {unresolved};请在 config 中补充 Species_Code 及拉丁名/别名后再执行。", "en": "{role} species names could not be mapped to Species_Code from config.xlsx: {unresolved}; please add Species_Code, Latin name, or alias in config before running."},
|
||||
"main-species-confirmed": {"zh": "已根据本次亚家族分类请求确定待分类物种;reference 仅作为参考来源。", "en": "Classification target species confirmed based on this subfamily classification request; reference species are for reference only."},
|
||||
"main-species-not-confirmed": {"zh": "基因家族分析尚未确认目标物种;请先核对并确认 config.xlsx。", "en": "Gene family analysis has not confirmed target species; please check and confirm config.xlsx first."},
|
||||
"main-role-target": {"zh": "目标物种", "en": "Target species"},
|
||||
"main-role-reference": {"zh": "参考物种", "en": "Reference species"},
|
||||
"main-role-comparative": {"zh": "比较物种", "en": "Comparative species"},
|
||||
"main-role-analysis-target": {"zh": "本轮分析对象", "en": "Analysis target"},
|
||||
"main-role-not-configured": {"zh": "未登记", "en": "Not configured"},
|
||||
"main-species-role-conflict": {"zh": "{species}:config.xlsx 为{config_role},本轮为{requested_role}", "en": "{species}: config.xlsx = {config_role}, current run = {requested_role}"},
|
||||
"main-species-internal-conflict": {"zh": "检测到 config.xlsx 内部物种角色冲突:{details}。请发送打开 config.xlsx,修正并重新确认物种角色后再继续。", "en": "Detected internal species role conflict in config.xlsx: {details}. Please open config.xlsx, correct and re-confirm species roles."},
|
||||
"main-species-role-drift": {"zh": "检测到系统意图解析发生角色漂移:{details}。config.xlsx 未被修改,系统已阻止按错误范围执行,请重新提交当前分析请求。", "en": "Detected role drift in intent parsing: {details}. config.xlsx unchanged; execution blocked. Please re-submit current analysis request."},
|
||||
"main-species-inconsistent-role": {"zh": "检测到本轮请求与项目物种角色不一致:{details}。系统已阻止执行,请按 config.xlsx 中已确认的角色重新描述分析范围;如需修改项目物种角色,请发送打开 config.xlsx。", "en": "Detected inconsistency between current request and project species roles: {details}. Execution blocked. Please re-describe analysis scope per confirmed config.xlsx roles; to change project roles, open config.xlsx."},
|
||||
"main-subfamily-missing-labels": {"zh": "请求要求使用亚家族标签,但缺少 {sp}.{family}.confirmed_subfamily.tsv 或 {sp}.{family}.subfamily。", "en": "Request requires subfamily labels, but {sp}.{family}.confirmed_subfamily.tsv or {sp}.{family}.subfamily is missing."},
|
||||
"main-subfamily-missing-suggestion": {"zh": "请先完成参考亚家族分类并保存确认标签,或上传目标物种亚家族文件。", "en": "Please complete reference subfamily classification and save confirmed labels, or upload target species subfamily file."},
|
||||
"main-stage-missing-pep": {"zh": "{stage} 缺少 {sp} 的基因家族成员蛋白文件。", "en": "{stage} is missing gene family member protein file for {sp}."},
|
||||
"main-stage-missing-pep-sugg": {"zh": "请先完成 {sp}/{family} 的 identify,生成 Target.{sp}.{family}.final_members.pep.fa。", "en": "Please complete identify for {sp}/{family} first to generate Target.{sp}.{family}.final_members.pep.fa."},
|
||||
"main-unknown-label": {"zh": "未知", "en": "Unknown"},
|
||||
"main-unknown-species": {"zh": "未知物种", "en": "Unknown species"},
|
||||
"main-species-role-conflict": {"zh": "{species}:config.xlsx 为{config_role},本轮为{requested_role}", "en": "{species}: config.xlsx = {config_role}, current run = {requested_role}"},
|
||||
"main-species-internal-conflict-prefix": {"zh": "检测到 config.xlsx 内部物种角色冲突:", "en": "Detected internal species role conflict in config.xlsx: "},
|
||||
"main-species-role-drift-prefix": {"zh": "检测到系统意图解析发生角色漂移:", "en": "Detected role drift in intent parsing: "},
|
||||
"main-species-inconsistent-role-prefix": {"zh": "检测到本轮请求与项目物种角色不一致:", "en": "Detected inconsistency between current request and project species roles: "},
|
||||
"main-open-config-suggestion": {"zh": "如需修改项目物种角色,请发送打开 config.xlsx。", "en": "To modify project species roles, please send open config.xlsx."},
|
||||
"main-species-unresolved": {"zh": "{role} 物种名称无法从 config.xlsx 映射到 Species_Code: {unresolved};请在 config 中补充 Species_Code 及拉丁名/别名后再执行。", "en": "{role} species names could not be mapped to Species_Code from config.xlsx: {unresolved}; please add Species_Code, Latin name, or alias in config before running."},
|
||||
"main-intent-ncbi-download": {"zh": "我理解你想从 NCBI 下载序列;输入为 {query}。", "en": "I understand you want to download sequences from NCBI; input: {query}."},
|
||||
"main-intent-clean-fasta": {"zh": "我理解你想清理FASTA序列ID/header;", "en": "I understand you want to clean FASTA sequence IDs/headers; "},
|
||||
"main-intent-extract-by-id": {"zh": "我理解你想按ID从本地FASTA提取序列;", "en": "I understand you want to extract sequences from local FASTA by ID; "},
|
||||
"main-intent-msa-conserved": {"zh": "我理解你想自动寻找 MSA 保守窗口并绘图;", "en": "I understand you want to automatically find MSA conserved windows and plot; "},
|
||||
"main-intent-msa-region": {"zh": "我理解你想对 MSA alignment 指定区间进行局部绘图。", "en": "I understand you want to plot a specific region of the MSA alignment."},
|
||||
"main-intent-msa-region-prompt": {"zh": "我理解你想对 MSA alignment 第 ... 列进行局部绘图。", "en": "I understand you want to plot specific columns of the MSA alignment."},
|
||||
"main-intent-promoter-custom": {"zh": "我理解你想基于已有 PlantCARE/promoter 结果进行启动子自定义筛选绘图,不重新跑启动子预测。", "en": "I understand you want to create custom promoter plots from existing PlantCARE/promoter results without re-running prediction."},
|
||||
"main-intent-extract-process": {"zh": "我理解你想按ID从本地FASTA提取或处理序列。", "en": "I understand you want to extract or process sequences from local FASTA by ID."},
|
||||
"main-intent-unknown": {"zh": "抱歉,我没有理解你想做什么分析。请说明要执行的分析阶段(鉴定、进化树、Motif、结构域、基因结构等)。", "en": "Sorry, I did not understand what analysis you want. Please specify the analysis stage (identification, phylogeny, Motif, domain, gene structure, etc.)."},
|
||||
"main-preflight-multi-target-spp": {"zh": "Identify 阶段检测到多个目标物种蛋白,需要确认 target species,不自动开始执行。", "en": "Identify stage detected multiple target species proteins; target species must be confirmed before auto-starting execution."},
|
||||
"main-preflight-select-target": {"zh": "请在弹窗中选择 target species;系统会据此自动生成内部 prefix。", "en": "Please select target species in the dialog; the system will auto-generate the internal prefix accordingly."},
|
||||
"main-preflight-check-config": {"zh": "请检查 config.xlsx、input、final 或先运行可生成该输入的上游阶段。", "en": "Please check config.xlsx, input, final, or run the upstream stage that generates this input first."},
|
||||
"main-preflight-check-genome": {"zh": "请检查 config.xlsx 中目标物种 genome FASTA,或先运行 genome_prepare 生成目标物种 genome 输入。", "en": "Please check target species genome FASTA in config.xlsx, or run genome_prepare first to generate target genome input."},
|
||||
"main-preflight-check-tfbs": {"zh": "请确认 data/tfbs/jaspar2026/combined/JASPAR2026_CORE_plants_non-redundant.meme 存在,或配置 PlantTFDB/JASPAR MEME 资源。", "en": "Please confirm data/tfbs/jaspar2026/combined/JASPAR2026_CORE_plants_non-redundant.meme exists, or configure PlantTFDB/JASPAR MEME resources."},
|
||||
"main-ext-stage-build-failed": {"zh": "拓展分析阶段构建失败: {exc}", "en": "Extension stage build failed: {exc}"},
|
||||
"main-ext-stage-trial-failed": {"zh": "拓展分析阶段试运行失败: {exc}", "en": "Extension stage trial run failed: {exc}"},
|
||||
"main-ext-stage-confirm-failed": {"zh": "拓展分析阶段确认失败: {exc}", "en": "Extension stage confirmation failed: {exc}"},
|
||||
"main-ext-stage-validate-params-failed": {"zh": "拓展分析阶段参数检查失败: {exc}", "en": "Extension stage parameter validation failed: {exc}"},
|
||||
"main-ext-stage-validate-failed": {"zh": "拓展分析阶段验证失败: {exc}", "en": "Extension stage validation failed: {exc}"},
|
||||
"main-ext-stage-repair-failed": {"zh": "拓展分析阶段修订失败: {exc}", "en": "Extension stage repair failed: {exc}"},
|
||||
"main-ext-stage-repair-candidates-failed": {"zh": "拓展分析阶段修订候选读取失败: {exc}", "en": "Extension stage repair candidate read failed: {exc}"},
|
||||
"main-ext-stage-repair-promote-failed": {"zh": "拓展分析阶段修订候选发布失败: {exc}", "en": "Extension stage repair candidate publish failed: {exc}"},
|
||||
"main-ext-stage-publish-failed": {"zh": "拓展分析阶段发布失败: {exc}", "en": "Extension stage publish failed: {exc}"},
|
||||
"main-ext-stage-frontend-gate-failed": {"zh": "拓展分析阶段前端门控记录失败: {exc}", "en": "Extension stage frontend gate record failed: {exc}"},
|
||||
"main-ext-stage-frontend-e2e-gate-failed": {"zh": "拓展分析阶段前端展示门控失败: {exc}", "en": "Extension stage frontend e2e gate failed: {exc}"},
|
||||
"main-ext-stage-nl-runtime-gate-failed": {"zh": "拓展分析阶段自然语言运行门控失败: {exc}", "en": "Extension stage NL runtime gate failed: {exc}"},
|
||||
"main-ext-stage-nl-runtime-repair-failed": {"zh": "拓展分析阶段自然语言运行门控修订失败: {exc}", "en": "Extension stage NL runtime gate repair failed: {exc}"},
|
||||
"main-ext-stage-execute-failed": {"zh": "拓展分析阶段执行失败: {exc}", "en": "Extension stage execution failed: {exc}"},
|
||||
"main-methodology-module-unavailable": {"zh": "方法学配置模块不可用,请检查 GF_MAS_ROOT", "en": "Methodology configuration module unavailable; please check GF_MAS_ROOT"},
|
||||
"main-methodology-config-read-failed": {"zh": "读取方法学配置失败: {e}", "en": "Failed to read methodology configuration: {e}"},
|
||||
"main-methodology-rag-trigger-failed": {"zh": "无法触发 RAG MinerU+LLM 方法学提取: {e}", "en": "Failed to trigger RAG MinerU+LLM methodology extraction: {e}"},
|
||||
"main-methodology-task-list-failed": {"zh": "无法读取 RAG 方法学任务列表: {e}", "en": "Failed to read RAG methodology task list: {e}"},
|
||||
"main-methodology-task-status-failed": {"zh": "无法读取 RAG 方法学任务状态: {e}", "en": "Failed to read RAG methodology task status: {e}"},
|
||||
"main-methodology-config-failed": {"zh": "无法读取 RAG 方法学配置: {e}", "en": "Failed to read RAG methodology configuration: {e}"},
|
||||
"main-methodology-plan-failed": {"zh": "无法读取 RAG 方法学方案: {e}", "en": "Failed to read RAG methodology plan: {e}"},
|
||||
"main-methodology-sandbox-example-failed": {"zh": "无法读取 RAG 方法学 sandbox 示例: {e}", "en": "Failed to read RAG methodology sandbox example: {e}"},
|
||||
"main-methodology-sandbox-build-failed": {"zh": "无法构建 RAG 方法学 sandbox: {e}", "en": "Failed to build RAG methodology sandbox: {e}"},
|
||||
"main-methodology-sandbox-query-failed": {"zh": "无法查询 RAG 方法学 sandbox: {e}", "en": "Failed to query RAG methodology sandbox: {e}"},
|
||||
"main-methodology-plan-not-found": {"zh": "方法学方案文件不存在: {path}", "en": "Methodology plan file not found: {path}"},
|
||||
"main-methodology-plan-read-failed": {"zh": "无法读取 RAG 方法学方案: {exc}", "en": "Failed to read RAG methodology plan: {exc}"},
|
||||
"main-methodology-missing-plan": {"zh": "缺少 plan、plan_path、methodology_job_id 或已绑定 job plan", "en": "Missing plan, plan_path, methodology_job_id, or already bound job plan"},
|
||||
"main-methodology-no-orchestrator": {"zh": "方法学执行不再使用独立后端编排器;请通过 /api/agent/plan 生成多智能体规划后再逐阶段提交 /api/execute。", "en": "Methodology execution no longer uses a standalone backend orchestrator; please generate a multi-agent plan via /api/agent/plan and submit stages via /api/execute."},
|
||||
# ── main.py general messages ──
|
||||
"main-file-read-failed": {"zh": "读取文件失败: {exc}", "en": "Failed to read file: {exc}"},
|
||||
"main-file-not-found-in-job": {"zh": "没有在当前 job 的安全目录中找到文件:{filename}", "en": "File not found in current job directory: {filename}"},
|
||||
"main-no-fasta-header": {"zh": "没有从 {filename} 读取到 FASTA header。", "en": "No FASTA header read from {filename}."},
|
||||
"main-no-preview-file": {"zh": "没有找到可预览的文件。", "en": "No previewable files found."},
|
||||
"main-no-result-prefix": {"zh": "没有识别到结果前缀,例如 At.SPL。", "en": "Could not identify result prefix, e.g. At.SPL."},
|
||||
"main-no-ncbi-result": {"zh": "没有在当前 workspace 中找到 {prefix} 的 NCBI 下载结果。", "en": "No NCBI download result found for {prefix} in current workspace."},
|
||||
"main-annotation-just-ended": {"zh": "自动背景注释进程刚结束,等待后端写入最终注释状态。", "en": "Auto background annotation process just ended; waiting for backend to write final annotation status."},
|
||||
"main-annotation-syncing": {"zh": "自动背景注释状态同步中,正在等待后端最终结果...", "en": "Auto background annotation status syncing; waiting for backend final result..."},
|
||||
"main-annotation-interrupted": {"zh": "自动背景注释已中断:progress 仍为 running,但没有检测到对应的运行进程。请重新执行该阶段。", "en": "Auto background annotation interrupted: progress is still running but no corresponding process detected. Please re-run this stage."},
|
||||
"main-annotation-failed": {"zh": "自动背景注释失败", "en": "Auto background annotation failed"},
|
||||
"main-annotation-done-continue": {"zh": "自动背景注释已完成,正在继续执行富集分析...", "en": "Auto background annotation completed; continuing with enrichment analysis..."},
|
||||
"main-annotation-progress": {"zh": "自动背景注释进行中:{processed}/{total},已注释 {annotated},失败 {failed}", "en": "Auto background annotation in progress: {processed}/{total}, annotated {annotated}, failed {failed}"},
|
||||
"main-waiting-resource": {"zh": "等待资源可用", "en": "Waiting for resource"},
|
||||
"main-waiting-resource-dots": {"zh": "等待资源可用...", "en": "Waiting for resource..."},
|
||||
"main-submitting-plantcare": {"zh": "正在提交序列到 PlantCARE...", "en": "Submitting sequences to PlantCARE..."},
|
||||
"main-running-mcscanx": {"zh": "正在进行共线性分析(MCScanX)...", "en": "Running synteny analysis (MCScanX)..."},
|
||||
"main-plantcare-waiting": {"zh": "已提交 PlantCARE,等待邮件结果(可能需要10-60分钟)...", "en": "Submitted to PlantCARE; waiting for email results (may take 10-60 minutes)..."},
|
||||
"main-plantcare-waiting-email": {"zh": "已提交 PlantCARE,正在等待邮件...", "en": "Submitted to PlantCARE; waiting for email..."},
|
||||
"main-task-queued": {"zh": "任务已进入资源调度队列", "en": "Task entered resource scheduling queue"},
|
||||
"main-task-submitted-celery": {"zh": "任务已提交到 Celery,等待 worker 获取资源", "en": "Task submitted to Celery; waiting for worker"},
|
||||
"main-task-done": {"zh": "任务完成", "en": "Task completed"},
|
||||
"main-task-failed": {"zh": "任务失败", "en": "Task failed"},
|
||||
"main-task-submitted-wait": {"zh": "任务已提交,等待 Celery worker 接收", "en": "Task submitted; waiting for Celery worker"},
|
||||
"main-task-running": {"zh": "任务运行中", "en": "Task running"},
|
||||
"main-task-not-exist": {"zh": "任务不存在", "en": "Task not found"},
|
||||
"main-task-ended": {"zh": "任务已结束,无法取消", "en": "Task ended; cannot cancel"},
|
||||
"main-task-cancelled": {"zh": "任务已取消", "en": "Task cancelled"},
|
||||
"main-task-marked-cancel": {"zh": "任务已标记为取消", "en": "Task marked for cancellation"},
|
||||
"main-ok-tree-revised": {"zh": "✅ 进化树图修订完成。", "en": "✅ Tree plot revision completed."},
|
||||
"main-ok-synteny-circos": {"zh": "✅ 种内共线性 Circos 图修订完成。", "en": "✅ Synteny Circos plot revision completed."},
|
||||
"main-ok-promoter-plot": {"zh": "✅ 启动子自定义绘图完成。", "en": "✅ Promoter custom plot completed."},
|
||||
"main-ok-msa-region-plot": {"zh": "✅ MSA区间绘图完成。", "en": "✅ MSA region plot completed."},
|
||||
"main-ok-stage-done": {"zh": "✅ {stage} 分析已完成", "en": "✅ {stage} analysis completed"},
|
||||
"main-err-stage-no-output": {"zh": "❌ {stage} 未生成有效输出", "en": "❌ {stage} did not generate valid output"},
|
||||
"main-ok-annotation-done": {"zh": "✅ 自动背景注释已完成", "en": "✅ Auto background annotation completed"},
|
||||
"main-gwas-need-fmt": {"zh": "先完成同一 job、同一 genotype scope 的 gwas_fmt 阶段。", "en": "Complete the gwas_fmt stage under the same job and genotype scope first."},
|
||||
"main-gwas-select-phenotype": {"zh": "请选择 phenotype 的样本列和性状列。", "en": "Please select phenotype sample column and trait column."},
|
||||
"main-gwas-admixture-k": {"zh": "ADMIXTURE K 必须至少为2。", "en": "ADMIXTURE K must be at least 2."},
|
||||
"main-gwas-qfam-no-source": {"zh": "Q/FAM 缺少可验证的源 genotype scope。", "en": "Q/FAM lacks verifiable source genotype scope."},
|
||||
"main-gwas-qfam-mismatch": {"zh": "Q/FAM 与当前 genotype scope 不一致。", "en": "Q/FAM does not match current genotype scope."},
|
||||
"main-gwas-need-four-upstream": {"zh": "先完成同一 job 和精确 scientific scope 的四个 GWAS 上游阶段。", "en": "Complete the four GWAS upstream stages under the same job and exact scientific scope first."},
|
||||
"main-gwas-need-mlm": {"zh": "先完成同一 job 和精确五范围的 gwas_mlm 阶段。", "en": "Complete the gwas_mlm stage under the same job and exact five-scope first."},
|
||||
"main-gwas-need-fmt-trait": {"zh": "先完成同一 job 和精确 genotype/phenotype scope 的 gwas_fmt 与 gwas_trait 阶段。", "en": "Complete gwas_fmt and gwas_trait stages under the same job and exact genotype/phenotype scope first."},
|
||||
"main-gwas-need-emmax": {"zh": "先完成同一 job 和精确 scientific scope 的 gwas_emmax 阶段。", "en": "Complete the gwas_emmax stage under the same job and exact scientific scope first."},
|
||||
"main-gwas-need-fmt-trait-cov": {"zh": "先完成同一 job 和精确 scientific scope 的 gwas_fmt、gwas_trait 与 gwas_cov 阶段。", "en": "Complete gwas_fmt, gwas_trait, and gwas_cov stages under the same job and exact scientific scope first."},
|
||||
"main-diag-closed": {"zh": "DiagnosisAgent 已关闭", "en": "DiagnosisAgent closed"},
|
||||
"main-diag-failed": {"zh": "诊断失败: {e}", "en": "Diagnosis failed: {e}"},
|
||||
"main-diag-job-stage-failed": {"zh": "Job/stage 诊断失败: {e}", "en": "Job/stage diagnosis failed: {e}"},
|
||||
"main-diag-no-auto-fix": {"zh": "该诊断没有可自动执行的修复动作,已清理确认请求。", "en": "This diagnosis has no auto-executable fix actions; confirmation request cleared."},
|
||||
"main-diag-old-confirm-cleared": {"zh": "该阶段已有成功 manifest,旧诊断确认请求已清理。", "en": "Stage already has a successful manifest; old diagnosis confirmation request cleared."},
|
||||
"main-diag-no-auto-fix-manual": {"zh": "该诊断结果没有可自动执行的修复动作;请按建议人工处理或重新运行已修复的流程。", "en": "No auto-executable fix actions for this diagnosis; please handle manually per suggestions or re-run the fixed pipeline."},
|
||||
"main-diag-fix-applied": {"zh": "修复方案已应用,将在下次轮询时继续执行", "en": "Fix applied; will continue execution on next poll"},
|
||||
"main-diag-fix-skipped": {"zh": "已跳过修复,将在下次轮询时继续下一阶段", "en": "Fix skipped; will continue to next stage on next poll"},
|
||||
"main-autofill-no-samples": {"zh": "样品表中没有可用于设计审查的样本。", "en": "No samples available for design review in sample table."},
|
||||
"main-autofill-no-config": {"zh": "未找到 config.autofill.draft.xlsx 或 config.xlsx", "en": "config.autofill.draft.xlsx or config.xlsx not found"},
|
||||
"main-autofill-no-samples-yaml": {"zh": "未找到 samples.autofill.draft.yaml 或 samples.yaml", "en": "samples.autofill.draft.yaml or samples.yaml not found"},
|
||||
"main-ident-no-members-id": {"zh": "未找到 identified_members.id,无法生成成员确认表。", "en": "identified_members.id not found; cannot generate member confirmation table."},
|
||||
"main-ident-no-interpro": {"zh": "未找到 InterProScan domain_review.tsv,已使用 identified_members + HMM/BLAST 证据生成成员确认表。", "en": "InterProScan domain_review.tsv not found; using identified_members + HMM/BLAST evidence to generate member confirmation table."},
|
||||
"main-subfamily-confirm-note": {"zh": "确认后会生成 confirmed_subfamily.tsv;后续树图、structure、motif、domain、combo、promoter 会优先使用确认后的亚家族标签。", "en": "Upon confirmation, confirmed_subfamily.tsv will be generated; subsequent tree, structure, motif, domain, combo, and promoter will use confirmed subfamily labels."},
|
||||
"main-subfamily-confirmed": {"zh": "亚家族标签已确认。联合确认会同步拆分生成各目标物种的 confirmed_subfamily.tsv;成员集合变化时需要重跑 phylogeny。", "en": "Subfamily labels confirmed. Joint confirmation will split and generate confirmed_subfamily.tsv per target species; phylogeny must be re-run when member sets change."},
|
||||
"main-promoter-no-filtered": {"zh": "未找到已有 promoter filtered 表,请先完成启动子分析。", "en": "No existing promoter filtered table found; please complete promoter analysis first."},
|
||||
"main-promoter-read-failed": {"zh": "读取启动子元件失败: {exc}", "en": "Failed to read promoter elements: {exc}"},
|
||||
"main-ref-downloading": {"zh": "参考物种注释包仍在后台下载", "en": "Reference species annotation package still downloading in background"},
|
||||
"main-ref-downloading-bg": {"zh": "正在后台下载参考物种注释包", "en": "Downloading reference species annotation package in background"},
|
||||
"main-ref-download-started": {"zh": "参考物种注释包下载任务已启动", "en": "Reference species annotation package download started"},
|
||||
"main-ref-download-done": {"zh": "参考物种注释包下载完成", "en": "Reference species annotation package download completed"},
|
||||
"main-ref-download-incomplete": {"zh": "参考物种注释包下载未满足必需文件", "en": "Reference species annotation package download missing required files"},
|
||||
"main-ref-download-submitted": {"zh": "参考物种注释包下载任务已提交", "en": "Reference species annotation package download submitted"},
|
||||
"main-old-rnaseq-deprecated": {"zh": "旧版 /api/rnaseq/run 需要 stage 和 sample;当前 RNA-seq 流程请使用 /api/execute。", "en": "Legacy /api/rnaseq/run requires stage and sample; for current RNA-seq pipeline, please use /api/execute."},
|
||||
"main-done": {"zh": "完成", "en": "Done"},
|
||||
"main-path-not-exist": {"zh": "路径不存在: {path}", "en": "Path does not exist: {path}"},
|
||||
"main-files-loaded": {"zh": "已加载 {count} 个文件", "en": "{count} file(s) loaded"},
|
||||
"main-load-failed": {"zh": "加载失败: {e}", "en": "Load failed: {e}"},
|
||||
"main-load-file-error": {"zh": "{path}: {e}", "en": "{path}: {e}"},
|
||||
"main-kg-html-not-found": {"zh": "kg.html not found", "en": "kg.html not found"},
|
||||
"main-kg-omics-not-found": {"zh": "knowledge_graph_with_omics.json not found", "en": "knowledge_graph_with_omics.json not found"},
|
||||
"main-kg-json-not-found": {"zh": "knowledge_graph.json not found", "en": "knowledge_graph.json not found"},
|
||||
"main-kg-json-env": {"zh": "未找到知识图谱 JSON。请设置环境变量 KNOWLEDGE_GRAPH_JSON...", "en": "Knowledge graph JSON not found. Please set environment variable KNOWLEDGE_GRAPH_JSON..."},
|
||||
"main-kg-read-failed": {"zh": "读取知识图谱文件失败: {exc}", "en": "Failed to read knowledge graph file: {exc}"},
|
||||
"main-rag-qa-failed": {"zh": "gene-family-mas-v2 QA 查询失败: {e}。请检查 GENE_FAMILY_MAS_API/DUAL_KB_QA_BASE_URL 指向的服务是否已启动。", "en": "gene-family-mas-v2 QA query failed: {e}. Please check whether the service pointed to by GENE_FAMILY_MAS_API/DUAL_KB_QA_BASE_URL is running."},
|
||||
"main-meth-data-not-ready": {"zh": "方法学方案数据未就绪,请先确认 config.xlsx。", "en": "Methodology plan data not ready; please confirm config.xlsx first."},
|
||||
"main-meth-species-unmapped": {"zh": "方法学方案中的物种名称未能映射到 config.xlsx 的 Species_Code,已阻止生成执行计划。", "en": "Species names in methodology plan could not be mapped to config.xlsx Species_Code; execution plan generation blocked."},
|
||||
"main-meth-no-required-pfams": {"zh": "方法学 identify 缺少 requiredPfams,未启动普通 Pfam 推断。", "en": "Methodology identify missing requiredPfams; generic Pfam inference not started."},
|
||||
"main-meth-reproduce": {"zh": "按已保存方法学方案复现文献流程", "en": "Reproduce literature pipeline per saved methodology plan"},
|
||||
"main-meth-no-job": {"zh": "当前请求未传入有效 job_id,或当前 job 未绑定 methodology_plan.json...", "en": "Current request has no valid job_id, or current job is not bound to methodology_plan.json..."},
|
||||
"main-meth-no-plan": {"zh": "当前 job 缺少可验证的基因家族分析计划,无法生成或打开正式报告。", "en": "Current job lacks a verifiable gene family analysis plan; cannot generate or open formal report."},
|
||||
"main-meth-report-action": {"zh": "生成或打开基因家族结果报告", "en": "Generate or open gene family result report"},
|
||||
"main-unspecified": {"zh": "未指定", "en": "Unspecified"},
|
||||
"main-stage-completed-fastp": {"zh": "正在进行RNA-seq质控分析(fastp)... 这可能需要15-30分钟,请耐心等待...", "en": "Running RNA-seq QC (fastp)... this may take 15-30 minutes, please wait..."},
|
||||
"main-detected-queue-recovery": {"zh": "检测到队列 {queue_name} 曾无 worker,已尝试恢复;任务仍在等待 worker 接收。", "en": "Queue {queue_name} previously had no workers; recovery attempted; task still waiting for worker."},
|
||||
"main-fasta-extracted": {"zh": "已从 {filename} 的 FASTA header 中抽取 {title}。", "en": "Extracted {title} from FASTA headers in {filename}."},
|
||||
"main-fasta-count": {"zh": "共 {count} 个;未识别到该字段的 header 数:{missing}。", "en": "Total {count}; unrecognized field headers: {missing}."},
|
||||
"main-source-file": {"zh": "来源文件:{path}", "en": "Source file: {path}"},
|
||||
"main-ncbi-result-found": {"zh": "已找到 {prefix} 的 NCBI 下载结果,workspace: {workspace}。", "en": "Found NCBI download results for {prefix}, workspace: {workspace}."},
|
||||
"main-gene-id-count": {"zh": "共 {count} 个基因 ID:", "en": "Total {count} gene IDs:"},
|
||||
"main-file-columns-preview": {"zh": "{filename} 的前 {cols} 列:\n\n{body}\n\n来源文件:{path}", "en": "First {cols} columns of {filename}:\n\n{body}\n\nSource file: {path}"},
|
||||
"main-file-lines-preview": {"zh": "{filename} 的前 {count} 行:\n\n```text\n{body}\n```\n\n来源文件:{path}", "en": "First {count} lines of {filename}:\n\n```text\n{body}\n```\n\nSource file: {path}"},
|
||||
# ── pipeline_backend ──
|
||||
"pipe-parse-ref-assembly": {"zh": "正在解析推荐参考物种装配", "en": "Parsing recommended reference species assembly"},
|
||||
"pipe-download-ncbi-ref": {"zh": "正在下载 NCBI 参考注释包", "en": "Downloading NCBI reference annotation package"},
|
||||
"pipe-decompress-ref": {"zh": "正在解压参考 PEP/CDS", "en": "Decompressing reference PEP/CDS"},
|
||||
"pipe-validate-pep-cds": {"zh": "正在校验 PEP/CDS ID 对齐", "en": "Validating PEP/CDS ID alignment"},
|
||||
"pipe-ref-download-done": {"zh": "参考物种注释包下载完成", "en": "Reference species annotation package download completed"},
|
||||
"pipe-subfamily-need-identify": {"zh": "❌ 亚家族证据阶段需要先完成 identify 成员初筛确认,生成 curated_members 后才能继续:{exc}", "en": "❌ Subfamily evidence stage requires identify member screening and curated_members generation first: {exc}"},
|
||||
"pipe-need-subfamily-confirm": {"zh": "❌ 常规基因家族分析需要先完成亚家族确认,生成 final_members 后才能继续:{exc}", "en": "❌ Standard gene family analysis requires subfamily confirmation and final_members generation first: {exc}"},
|
||||
"pipe-no-member-pep": {"zh": "❌ 未找到可用的成员蛋白 FASTA。", "en": "❌ No usable member protein FASTA found."},
|
||||
"pipe-cleanup-no-old": {"zh": "✅ 清理完成:未发现超过 {days} 天的任务目录", "en": "✅ Cleanup done: no directories older than {days} days found"},
|
||||
"pipe-cleanup-done": {"zh": "✅ 清理完成,已删除:\n", "en": "✅ Cleanup done, deleted:\n"},
|
||||
"pipe-select-file": {"zh": "请选择文件", "en": "Please select a file"},
|
||||
"pipe-upload-done": {"zh": "✅ 已上传到 input/{job_id}/: {files}", "en": "✅ Uploaded to input/{job_id}/: {files}"},
|
||||
"pipe-file-not-exist": {"zh": "文件不存在", "en": "File not found"},
|
||||
"pipe-file-truncated": {"zh": "\n\n... (文件较大,仅显示部分)", "en": "\n\n... (file too large, showing partial content)"},
|
||||
"pipe-file-binary-preview": {"zh": "无法预览此文件(可能是二进制/图片/PDF)", "en": "Cannot preview this file (binary/image/PDF)"},
|
||||
# ── report ──
|
||||
"report-no-section-boundary": {"zh": "报告模板没有章节边界:{path}", "en": "Report template has no section boundaries: {path}"},
|
||||
"report-unreplaced-placeholders": {"zh": "报告模板仍含未替换占位符", "en": "Report template still contains unreplaced placeholders"},
|
||||
"report-no-results": {"zh": "暂无可生成的基因家族分析结果", "en": "No gene family analysis results available for report generation"},
|
||||
"report-title-with-prefix": {"zh": "{prefix} 基因家族分析结果解读报告", "en": "{prefix} Gene Family Analysis Interpretation Report"},
|
||||
"report-table-header": {"zh": "| 分析阶段 | 运行状态 | 模板支持 | 纳入报告 |", "en": "| Analysis Stage | Status | Template Support | Included |"},
|
||||
"report-status-success": {"zh": "成功", "en": "Success"},
|
||||
"report-status-partial": {"zh": "部分完成", "en": "Partial"},
|
||||
"report-status-failed": {"zh": "失败", "en": "Failed"},
|
||||
"report-status-pending": {"zh": "未完成", "en": "Pending"},
|
||||
"report-unspecified": {"zh": "未指定", "en": "Unspecified"},
|
||||
"report-api-orchestrated": {"zh": "API 编排", "en": "API Orchestrated"},
|
||||
"report-fallback-script": {"zh": "候补脚本", "en": "Fallback Script"},
|
||||
"report-is": {"zh": "是", "en": "Yes"},
|
||||
"report-no": {"zh": "否", "en": "No"},
|
||||
"report-fig-meaning": {"zh": "结果图展示该阶段的主要计算产物。", "en": "The result figure shows the main computational products of this stage."},
|
||||
"report-fig-reading": {"zh": "结合图题、坐标轴、图例和成员标签阅读。", "en": "Read in conjunction with figure title, axes, legend, and member labels."},
|
||||
"report-fig-result": {"zh": "本阶段代表结果见图,具体事实以本节确定性数据为准。", "en": "Representative results are shown in the figure; specific facts should reference the definitive data in this section."},
|
||||
# ── interpretation submodules ──
|
||||
"interp-pattern-member-confidence": {"zh": "家族成员集合可信度", "en": "Gene family member set confidence"},
|
||||
"interp-pattern-core-domain": {"zh": "核心结构域支持", "en": "Core domain support"},
|
||||
"interp-pattern-domain-motif": {"zh": "结构域与motif共同支持保守性", "en": "Domain and motif jointly support conservation"},
|
||||
"interp-pattern-expression": {"zh": "表达调控线索", "en": "Expression regulation clues"},
|
||||
"interp-claim-member-traceable": {"zh": "当前成员集合具有可追溯的家族鉴定证据。", "en": "The current member set has traceable family identification evidence."},
|
||||
"interp-claim-no-expansion": {"zh": "目标物种发生家族扩张或收缩。", "en": "Target species underwent family expansion or contraction."},
|
||||
"interp-validate-boundary-members": {"zh": "复核边界成员的HMM/BLAST来源和蛋白序列完整性。", "en": "Review HMM/BLAST sources and protein sequence integrity of boundary members."},
|
||||
"interp-claim-domain-support": {"zh": "成员集合具有核心结构域层面的支持。", "en": "Member set has core domain-level support."},
|
||||
"interp-claim-no-full-conservation": {"zh": "所有成员结构域完全保守。", "en": "All member domains are fully conserved."},
|
||||
"interp-validate-truncation": {"zh": "检查未检出核心结构域的成员是否为截短或注释异常。", "en": "Check whether members without detected core domains are truncated or annotation anomalies."},
|
||||
"interp-claim-domain-motif-conservation": {"zh": "结构域和motif共同支持蛋白架构保守性。", "en": "Domain and motif jointly support protein architecture conservation."},
|
||||
"interp-critic-no-local-evidence": {"zh": "未绑定本地证据,不能进入本项目主要发现。", "en": "Not bound to local evidence; cannot enter main findings of this project."},
|
||||
"interp-critic-high-risk": {"zh": "存在high级风险,结论可信度已自动降级。", "en": "High-level risk exists; conclusion confidence has been auto-downgraded."},
|
||||
"interp-critic-weak-evidence": {"zh": "当前证据不足以支撑强机制动词,需改为候选或一致性表述。", "en": "Current evidence insufficient for strong mechanistic language; use candidate or consistency phrasing."},
|
||||
"interp-report-supported": {"zh": "可表述为:{claim_text};该表述由本地证据支撑...", "en": "Can be stated as: {claim_text}; supported by local evidence..."},
|
||||
"interp-report-candidate": {"zh": "可表述为:{claim_text};该结论应写成候选或筛查线索...", "en": "Can be stated as: {claim_text}; this conclusion should be phrased as candidate or screening clue..."},
|
||||
"interp-report-lit-background": {"zh": "可表述为:{claim_text};相关文献可用于交代生物学背景...", "en": "Can be stated as: {claim_text}; relevant literature can be used for biological background..."},
|
||||
"interp-narr-local-evidence": {"zh": " 本地证据:{evidence_text}。", "en": " Local evidence: {evidence_text}."},
|
||||
"interp-narr-boundary": {"zh": " 边界:{limitation}。", "en": " Boundary: {limitation}."},
|
||||
"interp-narr-next-step": {"zh": " 下一步:{next_step}。", "en": " Next step: {next_step}."},
|
||||
"interp-narr-conclusion-boundary": {"zh": "当前结论必须限制在本地证据支持范围内。", "en": "Current conclusions must be limited to locally supported evidence."},
|
||||
"interp-narr-candidate-discovery": {"zh": "候选科学发现", "en": "Candidate scientific discoveries"},
|
||||
"interp-slot-object-scope": {"zh": "本阶段分析对象和规模", "en": "Objects and scale of this stage"},
|
||||
"interp-slot-key-features": {"zh": "关键数据特征", "en": "Key data features"},
|
||||
"interp-slot-patterns": {"zh": "组内/组间模式或例外", "en": "Within/between-group patterns or exceptions"},
|
||||
"interp-slot-relationships": {"zh": "与前后阶段的关系", "en": "Relationship with preceding and following stages"},
|
||||
"interp-slot-supported-claims": {"zh": "可支持的解释", "en": "Supported interpretations"},
|
||||
"interp-slot-boundaries": {"zh": "不能越界的部分", "en": "Boundaries that should not be crossed"},
|
||||
"interp-unresolved": {"zh": "未解析", "en": "Unresolved"},
|
||||
"interp-no-local-evidence": {"zh": "未找到可追溯本地证据;仅能报告阶段状态,不能形成结果解读。", "en": "No traceable local evidence found; only stage status can be reported, no result interpretation."},
|
||||
"interp-status-restricted": {"zh": "阶段状态为 {status},结果解读应标注为受限。", "en": "Stage status is {status}; result interpretation should be marked as restricted."},
|
||||
"interp-missing-subtask-products": {"zh": "{status} 阶段缺少完整子任务产物,不能按完全成功阶段写作。", "en": "{status} stage lacks complete subtask products; cannot be written as fully successful stage."},
|
||||
"interp-missing-newick": {"zh": "缺少可核验的 Newick tip 计数或分支支持信息。", "en": "Missing verifiable Newick tip counts or branch support information."},
|
||||
"interp-missing-motif-sites": {"zh": "缺少每个 motif 的位点分布表,不能写成员级 motif 排列。", "en": "Missing per-motif site distribution table; cannot write member-level motif arrangement."},
|
||||
"interp-motif-partial": {"zh": "motif 阶段存在部分成功,seqlogo 或辅助图件等子任务应作为缺口处理。", "en": "Motif stage partially succeeded; subtasks such as seqlogo or auxiliary figures should be treated as gaps."},
|
||||
"interp-missing-domain": {"zh": "缺少可解析的主要结构域分布。", "en": "Missing parseable major domain distribution."},
|
||||
"interp-missing-structure": {"zh": "缺少结构原始表基因级计数,组内 exon/intron 模式应降级。", "en": "Missing gene-level counts from raw structure table; within-group exon/intron patterns should be downgraded."},
|
||||
"interp-generic-template": {"zh": "该阶段使用通用模板,不能替代领域特异 Results 解读。", "en": "This stage uses a generic template; cannot substitute for domain-specific Results interpretation."},
|
||||
"interp-missing-evidence": {"zh": "缺少可追溯 evidence 条目,只能描述 manifest/指标层结果。", "en": "Missing traceable evidence entries; only manifest/metric-level results can be described."},
|
||||
"interp-status-downgrade": {"zh": "阶段状态为 {status},需要降级解释。", "en": "Stage status is {status}; interpretation needs downgrading."},
|
||||
"interp-high-risk-conflict": {"zh": "存在 high risk 证据冲突,不能写成可发表结果。", "en": "High-risk evidence conflicts exist; cannot be written as publishable results."},
|
||||
"interp-risk-caveat": {"zh": "存在风险提示,解释需保留边界。", "en": "Risk caveats exist; interpretation must retain boundaries."},
|
||||
}
|
||||
|
||||
# ── 兼容映射:旧 _MSG 中文 key → 新 key ──
|
||||
_LEGACY_MSG_MAP: dict[str, str] = {
|
||||
"✅ 所有分析阶段已完成!": "status-completed",
|
||||
"📋 自动规划剩余分析阶段": "status-auto-plan",
|
||||
"**已完成**": "status-completed-label",
|
||||
"无": "status-none",
|
||||
"**待执行**": "status-pending",
|
||||
"个阶段)": "status-stage-count",
|
||||
"请确认是否按顺序执行以上分析阶段?": "confirm-execute-stages",
|
||||
"我将帮您执行RNA-seq": "intent-rnaseq-execute",
|
||||
"分析。请确认参数后开始。": "intent-confirm-start",
|
||||
"RNA-seq分析将通过前端异步执行。": "intent-rnaseq-async",
|
||||
" 分析。PlanningAgentV2 已根据当前状态生成完整执行计划。": "intent-planning-v2-plan",
|
||||
"我将帮您执行": "intent-will-execute",
|
||||
"分析。": "intent-analysis-suffix",
|
||||
"请提供基因家族蛋白序列文件(.fa/.fasta),用于MEME保守基序预测。": "guidance-motif",
|
||||
"请提供config.xlsx及基因组/GFF/CDS/PEP等原始文件,用于完整数据准备。": "guidance-genome-prepare",
|
||||
"请提供基因家族蛋白序列文件,用于Pfam/InterPro结构域注释。": "guidance-domain",
|
||||
"请提供基因家族的GFF注释文件和基因组序列。": "guidance-structure",
|
||||
"请提供基因家族成员的启动子序列(上游2000bp)或全基因组序列+GFF。": "guidance-promoter",
|
||||
"请提供物种基因组序列(.fasta)和GFF注释文件。": "guidance-synteny",
|
||||
"请提供多个物种的基因组序列和GFF注释文件,以及物种间的同源关系。": "guidance-interspecies-synteny",
|
||||
"请提供染色体长度信息和基因位置信息。": "guidance-karyotype",
|
||||
"请提供拟南芥同源基因映射文件(ortholog_pairs.txt,格式:拟南芥ID\\t苦荞ID),用于STRING数据库互作预测。如未提供,请先运行同源基因识别分析。": "guidance-ppi",
|
||||
"请提供目标物种和参考物种(如拟南芥At)的蛋白序列文件,用于同源基因识别。": "guidance-ortholog",
|
||||
"请提供物种间同源基因对(ortholog pairs,格式:基因A\\t基因B)或共线性分析结果。": "guidance-kaks",
|
||||
"请提供RNA-seq表达矩阵(FPKM/TPM/Count)或原始测序数据。": "guidance-expression",
|
||||
"请提供差异表达基因列表(DEGs)或目标基因集。": "guidance-enrichment",
|
||||
"请提供基因家族蛋白序列文件,用于理化性质分析。": "guidance-pep",
|
||||
"请提供基因家族HMM文件和蛋白序列文件,将自动执行标准11阶段分析。": "guidance-combo",
|
||||
"请提供RNA-seq原始测序数据(FASTQ格式)。": "guidance-rna-seq-qc",
|
||||
"请提供质控后的clean reads(FASTQ格式)和参考基因组。": "guidance-rna-seq-mapping",
|
||||
"请提供比对后的BAM文件或参考基因组注释GFF。": "guidance-rna-seq-quantification",
|
||||
"请提供featureCounts计数矩阵和样本分组表,用于VST、PCA、样本相关性和聚类。": "guidance-rnaseq-sample-qc",
|
||||
"请提供多时相表达矩阵(如不同发育阶段/处理时间点)。": "guidance-trend",
|
||||
"请提供VST/log2表达矩阵或count矩阵和样本性状表,用于共表达网络构建。": "guidance-rnaseq-wgcna",
|
||||
"请提供差异表达结果和gene2go背景文件,或先执行背景注释。": "guidance-rnaseq-go-enrich",
|
||||
"请提供差异表达结果和gene2ko背景文件,或先执行背景注释。": "guidance-rnaseq-kegg-enrich",
|
||||
"请提供差异表达排序统计量和gene2ko/KEGG背景,用于fgsea排序富集。": "guidance-rnaseq-fgsea",
|
||||
"请提供差异表达结果和目标物种蛋白FASTA;若已有InterProScan TSV会优先复用。": "guidance-rnaseq-candidate-family",
|
||||
"请提供表达矩阵(样本数≥6),用于共表达网络构建。": "guidance-wgcna",
|
||||
"你是BotanicalBaseAgent基因家族分析系统的知识助手。基于提供的文献资料,生成专业、详细的回答。": "knowledge-system-prompt",
|
||||
"📚 基于知识库的回答:": "knowledge-answer-label",
|
||||
"**参考文献**:": "knowledge-refs-label",
|
||||
"(来源: 本地LLM回答 + RAG辅助检索)": "knowledge-local-rag-source",
|
||||
"⚠️ (本地LLM回答,未使用RAG文献检索)": "knowledge-local-no-rag",
|
||||
"找到": "knowledge-found",
|
||||
"条相关文献,正在生成本地LLM回答...": "knowledge-generating",
|
||||
"未找到相关文献,生成一般回答": "knowledge-no-refs",
|
||||
"你是一个专业的基因家族分析意图识别器。": "intent-recognizer-prompt",
|
||||
"你是一个基因家族分析专家。用户想要进行基因家族分析。": "intent-expert-prompt",
|
||||
}
|
||||
|
||||
|
||||
# ── API ──────────────────────────────────────────────
|
||||
|
||||
def t(key: str, lang: str = "zh", **params) -> str:
|
||||
"""获取翻译文本,支持参数替换.
|
||||
|
||||
Args:
|
||||
key: I18N_MAP 中的 key
|
||||
lang: "zh" 或 "en"
|
||||
**params: 替换 {param_name} 占位符
|
||||
|
||||
Returns:
|
||||
翻译后的文本;若 key 不存在则返回 "⚠key"
|
||||
"""
|
||||
entry = I18N_MAP.get(key, {})
|
||||
text = entry.get(lang) or entry.get("zh") or f"⚠{key}"
|
||||
if params:
|
||||
text = text.format(**params)
|
||||
return text
|
||||
|
||||
|
||||
def t_legacy(zh_text: str, lang: str = "zh") -> str:
|
||||
"""兼容旧 orchestrator._l() 的调用方式:用中文文本查找翻译.
|
||||
|
||||
优先查 _LEGACY_MSG_MAP → 新 I18N_MAP,找不到做部分匹配,最后回退到原中文。
|
||||
"""
|
||||
if lang == "zh":
|
||||
return zh_text
|
||||
|
||||
new_key = _LEGACY_MSG_MAP.get(zh_text)
|
||||
if new_key:
|
||||
return t(new_key, lang)
|
||||
|
||||
# fallback: partial match
|
||||
for legacy_zh, mapped_key in _LEGACY_MSG_MAP.items():
|
||||
if legacy_zh in zh_text:
|
||||
translated_part = t(mapped_key, lang)
|
||||
return zh_text.replace(legacy_zh, translated_part)
|
||||
|
||||
return zh_text
|
||||
|
||||
|
||||
def validate_i18n_coverage() -> list[str]:
|
||||
"""返回所有缺少 en 翻译的 key."""
|
||||
return [k for k, v in I18N_MAP.items() if "en" not in v]
|
||||
|
||||
|
||||
# ── Key 命名规范 ──
|
||||
# 阶段名称: stage-name-{id}
|
||||
# 状态消息: status-{state}
|
||||
# 会话: session-{action}
|
||||
# 文件操作: file-{action}
|
||||
# 错误/验证: error-{context} / validation-{context}
|
||||
# 诊断/修复: diag-{context} / fix-{context}
|
||||
# 报告: report-{stage}-{type}
|
||||
# GWAS: gwas-{action}
|
||||
# 方法学: methodology-{action}
|
||||
# 亚家族: subfamily-{action}
|
||||
# 扩展阶段: extension-{action}
|
||||
# 解读证据: evidence-{stage}-{type}
|
||||
@@ -23,10 +23,7 @@ from gfa_core.mas.mutation_surface import ToolSurfaceEntry
|
||||
from gfa_core.mas.stage_capabilities import get_stage_capability
|
||||
from gfa_core.mas.tools.stage_status import SUCCESS_STATUSES
|
||||
from gfa_core.mas.tools.stage_tools import stage_registry
|
||||
from gfa_core.mas.workflow_catalog import (
|
||||
load_canonical_workflow_contract,
|
||||
load_rnaseq_business_context,
|
||||
)
|
||||
from gfa_core.mas.workflow_catalog import load_rnaseq_business_context
|
||||
from gfa_core.mas.workflow_models import GoalLedger
|
||||
|
||||
|
||||
@@ -56,6 +53,7 @@ class ConversationToolRegistry:
|
||||
*,
|
||||
workflow_id: str | None = None,
|
||||
guidance: bool = False,
|
||||
language: str = "zh",
|
||||
) -> str:
|
||||
text = str(query or "").strip()
|
||||
if self._analysis_handoff_resolver is None:
|
||||
@@ -65,6 +63,7 @@ class ConversationToolRegistry:
|
||||
text,
|
||||
workflow_id=workflow_id,
|
||||
guidance=guidance,
|
||||
language=language,
|
||||
)
|
||||
or ""
|
||||
).strip()
|
||||
@@ -267,6 +266,7 @@ class RecognizeAnalysisIntentArgs(BaseModel):
|
||||
query: str
|
||||
job_id: str = ""
|
||||
prefix: str = ""
|
||||
language: str = "zh"
|
||||
|
||||
_job_id_is_local = field_validator("job_id")(_validate_job_id)
|
||||
_prefix_is_literal = field_validator("prefix")(_validate_prefix)
|
||||
@@ -299,6 +299,7 @@ class BoundRecognizeAnalysisIntentArgs(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
query: str
|
||||
prefix: str = ""
|
||||
language: str = "zh"
|
||||
_prefix_is_literal = field_validator("prefix")(_validate_prefix)
|
||||
|
||||
|
||||
@@ -1607,9 +1608,11 @@ def build_plantai_conversation_tools(
|
||||
*,
|
||||
workflow_id: str | None = None,
|
||||
guidance: bool = False,
|
||||
language: str = "zh",
|
||||
) -> str:
|
||||
"""Adapt MainAgent memory to the original IntentAgent text contract."""
|
||||
text = str(query or "").strip()
|
||||
en = language == "en"
|
||||
if (
|
||||
text.startswith("执行数据准备(genome_prepare)")
|
||||
and "当前业务动作已由数据准备契约唯一确定" in text
|
||||
@@ -1619,15 +1622,15 @@ def build_plantai_conversation_tools(
|
||||
return text
|
||||
terms = goal_ledger.user_terms
|
||||
context: list[str] = []
|
||||
canonical_request = ""
|
||||
family = str(terms.get("gene_family") or "").strip()
|
||||
if family:
|
||||
context.append(f"基因家族 {family}")
|
||||
for key, label in (
|
||||
("target_species", "目标物种"),
|
||||
("reference_species", "参考物种"),
|
||||
("comparative_species", "比较物种"),
|
||||
context.append(f"Gene family: {family}" if en else f"基因家族 {family}")
|
||||
for key, label_zh, label_en in (
|
||||
("target_species", "目标物种", "Target species"),
|
||||
("reference_species", "参考物种", "Reference species"),
|
||||
("comparative_species", "比较物种", "Comparative species"),
|
||||
):
|
||||
label = label_en if en else label_zh
|
||||
values = terms.get(key) or ()
|
||||
if isinstance(values, (list, tuple)):
|
||||
cleaned = [
|
||||
@@ -1647,21 +1650,39 @@ def build_plantai_conversation_tools(
|
||||
if mode == "whole_genome":
|
||||
enabled_designs = "、".join(business_context["enabled_designs"])
|
||||
disabled_designs = "、".join(business_context["disabled_designs"])
|
||||
context.append(
|
||||
f"RNA-seq业务模式 全基因组;{experiment_design} 是实验设计权威输入;"
|
||||
f"可解释设计为 {enabled_designs},{disabled_designs} 本轮禁用;"
|
||||
"设计或 batch 降级不得静默发生"
|
||||
)
|
||||
if en:
|
||||
context.append(
|
||||
f"RNA-seq mode: whole genome; {experiment_design} is the authoritative experimental design input; "
|
||||
f"interpretable designs: {enabled_designs}, {disabled_designs} disabled this round; "
|
||||
"design or batch downgrades must not happen silently"
|
||||
)
|
||||
else:
|
||||
context.append(
|
||||
f"RNA-seq业务模式 全基因组;{experiment_design} 是实验设计权威输入;"
|
||||
f"可解释设计为 {enabled_designs},{disabled_designs} 本轮禁用;"
|
||||
"设计或 batch 降级不得静默发生"
|
||||
)
|
||||
else:
|
||||
context.append(
|
||||
"RNA-seq业务模式 基因家族成员;使用已成功的全基因组差异表达结果"
|
||||
"和精确确认的基因家族成员进行投影;不得自动选择首个成员文件,"
|
||||
"不得隐式重跑差异表达"
|
||||
)
|
||||
if en:
|
||||
context.append(
|
||||
"RNA-seq mode: gene family member; use successful whole-genome differential expression results "
|
||||
"and precisely confirmed gene family members for projection; must not auto-select the first member file, "
|
||||
"must not implicitly re-run differential expression"
|
||||
)
|
||||
else:
|
||||
context.append(
|
||||
"RNA-seq业务模式 基因家族成员;使用已成功的全基因组差异表达结果"
|
||||
"和精确确认的基因家族成员进行投影;不得自动选择首个成员文件,"
|
||||
"不得隐式重跑差异表达"
|
||||
)
|
||||
elif rna_route == "whole_genome":
|
||||
context.append("RNA-seq模式 全基因组")
|
||||
context.append(
|
||||
"RNA-seq mode: whole genome" if en else "RNA-seq模式 全基因组"
|
||||
)
|
||||
elif rna_route == "family_member":
|
||||
context.append("RNA-seq模式 基因家族成员")
|
||||
context.append(
|
||||
"RNA-seq mode: gene family member" if en else "RNA-seq模式 基因家族成员"
|
||||
)
|
||||
job_id = bound_job_id or str(terms.get("active_job_id") or "").strip()
|
||||
formal_config_exists = bool(
|
||||
job_id and (project_root / "input" / job_id / "config.xlsx").is_file()
|
||||
@@ -1671,7 +1692,6 @@ def build_plantai_conversation_tools(
|
||||
or "config.xlsx 已确认" in goal_ledger.approved_decisions
|
||||
or formal_config_exists
|
||||
)
|
||||
genome_prepare_observed = False
|
||||
subfamily_confirm_observed = False
|
||||
if job_id:
|
||||
final_dir = project_root / "workspace" / job_id / "final"
|
||||
@@ -1682,8 +1702,6 @@ def build_plantai_conversation_tools(
|
||||
continue
|
||||
stage = str(manifest.get("stage") or "").strip()
|
||||
status = manifest.get("status")
|
||||
if stage == "genome_prepare" and status in {"success", "partial_success"}:
|
||||
genome_prepare_observed = True
|
||||
manifest_prefix = str(manifest.get("prefix") or "").strip()
|
||||
family_matches = not family or bool(
|
||||
re.search(
|
||||
@@ -1698,11 +1716,20 @@ def build_plantai_conversation_tools(
|
||||
and family_matches
|
||||
):
|
||||
subfamily_confirm_observed = True
|
||||
genome_prepare_observed = False
|
||||
canonical_request = None
|
||||
if config_confirmed and not genome_prepare_observed:
|
||||
context.append(
|
||||
"正式 config.xlsx 已确认;这只表示配置已核对,不等于数据准备已经完成,"
|
||||
"当前 job 尚无数据准备完成证据;具体分析意图和阶段仍由现有分析系统判断"
|
||||
)
|
||||
if en:
|
||||
context.append(
|
||||
"Formal config.xlsx confirmed; this indicates configuration has been verified, "
|
||||
"not that data preparation is complete. No data preparation completion evidence in current job; "
|
||||
"specific analysis intent and stages still determined by the existing analysis system"
|
||||
)
|
||||
else:
|
||||
context.append(
|
||||
"正式 config.xlsx 已确认;这只表示配置已核对,不等于数据准备已经完成,"
|
||||
"当前 job 尚无数据准备完成证据;具体分析意图和阶段仍由现有分析系统判断"
|
||||
)
|
||||
if guidance and workflow_id in {
|
||||
"data_preparation",
|
||||
"gene_family",
|
||||
@@ -1717,19 +1744,34 @@ def build_plantai_conversation_tools(
|
||||
)
|
||||
actions = {item.id: item for item in contract.actions}
|
||||
target_stage = actions[transition.action_id].target_stage
|
||||
canonical_request = (
|
||||
f"执行数据准备({target_stage})\n"
|
||||
"当前业务动作已由数据准备契约唯一确定"
|
||||
)
|
||||
if en:
|
||||
canonical_request = (
|
||||
f"Execute data preparation ({target_stage})\n"
|
||||
"Business action uniquely determined by data preparation contract"
|
||||
)
|
||||
else:
|
||||
canonical_request = (
|
||||
f"执行数据准备({target_stage})\n"
|
||||
"当前业务动作已由数据准备契约唯一确定"
|
||||
)
|
||||
if family and subfamily_confirm_observed:
|
||||
context.append(
|
||||
"当前 job 的 subfamily_confirm 已成功完成,家族成员和亚家族分类表已确认,"
|
||||
"这是当前项目状态事实;具体分析意图和阶段仍由现有分析系统判断"
|
||||
)
|
||||
if en:
|
||||
context.append(
|
||||
"Current job's subfamily_confirm completed successfully; "
|
||||
"family members and subfamily classification table confirmed. "
|
||||
"This is a current project state fact; specific analysis intent and stages still determined by the existing analysis system"
|
||||
)
|
||||
else:
|
||||
context.append(
|
||||
"当前 job 的 subfamily_confirm 已成功完成,家族成员和亚家族分类表已确认,"
|
||||
"这是当前项目状态事实;具体分析意图和阶段仍由现有分析系统判断"
|
||||
)
|
||||
if canonical_request:
|
||||
return canonical_request
|
||||
if not context:
|
||||
return text
|
||||
if en:
|
||||
return f"{text}\nConfirmed context: {'; '.join(context)}."
|
||||
return f"{text}\n已确认上下文:{';'.join(context)}。"
|
||||
|
||||
registry.set_analysis_handoff_resolver(intent_query_with_confirmed_context)
|
||||
@@ -1746,7 +1788,7 @@ def build_plantai_conversation_tools(
|
||||
receiver="intent_v2",
|
||||
message_type="recognize_intent",
|
||||
payload={
|
||||
"query": intent_query_with_confirmed_context(args.query),
|
||||
"query": intent_query_with_confirmed_context(args.query, language=args.language),
|
||||
"job_id": job_id,
|
||||
"prefix": args.prefix,
|
||||
"workspace": str(project_root / "workspace" / job_id) if job_id else "",
|
||||
|
||||
@@ -0,0 +1,718 @@
|
||||
"""DeepSeek-backed, evidence-locked gene-family summary reports.
|
||||
|
||||
The LLM is used only to turn verified local results into readable Chinese. All
|
||||
numbers, result boundaries and figure names come from local manifests and the
|
||||
existing evidence interpreter. The report, prompt evidence and API metadata
|
||||
are persisted separately so a generated claim can be audited later.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Optional
|
||||
|
||||
from gfa_core.mas.agents.interpretation_evidence import (
|
||||
build_evidence_report,
|
||||
extract_stage_evidence,
|
||||
)
|
||||
from gfa_core.mas.llm.base import LLMMessage
|
||||
from gfa_core.mas.llm.providers.deepseek import DeepSeekProvider
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[3]
|
||||
DEFAULT_FRAMEWORK = PROJECT_ROOT / "prompts" / "gene_family_summary_report_framework_zh.md"
|
||||
IMAGE_SUFFIXES = {".png", ".pdf", ".svg", ".jpg", ".jpeg"}
|
||||
STAGE_HINTS = [
|
||||
"reference_classification",
|
||||
"subfamily_confirm",
|
||||
"interspecies_synteny",
|
||||
"phylogeny",
|
||||
"identify",
|
||||
"motif",
|
||||
"domain",
|
||||
"structure",
|
||||
"combo",
|
||||
"promoter",
|
||||
"tfbs",
|
||||
"synteny",
|
||||
"karyotype",
|
||||
"ortholog",
|
||||
"kaks",
|
||||
"pep",
|
||||
]
|
||||
|
||||
|
||||
class DeepSeekSummaryError(RuntimeError):
|
||||
"""Raised when an evidence-locked report cannot be generated."""
|
||||
|
||||
|
||||
def _read_json(path: Path) -> Dict[str, Any]:
|
||||
try:
|
||||
value = json.loads(path.read_text(encoding="utf-8"))
|
||||
return value if isinstance(value, dict) else {}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _load_env_file(path: Path) -> Dict[str, str]:
|
||||
values: Dict[str, str] = {}
|
||||
if not path.is_file():
|
||||
return values
|
||||
for raw in path.read_text(encoding="utf-8", errors="ignore").splitlines():
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, value = line.split("=", 1)
|
||||
values[key.strip()] = value.strip().strip('"').strip("'")
|
||||
return values
|
||||
|
||||
|
||||
def load_deepseek_settings(*, model_override: str = "") -> Dict[str, str]:
|
||||
values: Dict[str, str] = {}
|
||||
for env_path in (
|
||||
PROJECT_ROOT / ".env.v15_deepseek",
|
||||
PROJECT_ROOT / ".env",
|
||||
PROJECT_ROOT / "config" / ".env",
|
||||
):
|
||||
for key, value in _load_env_file(env_path).items():
|
||||
values.setdefault(key, value)
|
||||
api_key = (
|
||||
os.environ.get("DEEPSEEK_API_KEY", "").strip()
|
||||
or os.environ.get("NEW_API_KEY", "").strip()
|
||||
or values.get("DEEPSEEK_API_KEY", "").strip()
|
||||
or values.get("NEW_API_KEY", "").strip()
|
||||
)
|
||||
if not api_key:
|
||||
raise DeepSeekSummaryError("DeepSeek API key is not configured (DEEPSEEK_API_KEY or NEW_API_KEY).")
|
||||
base_url = (
|
||||
os.environ.get("DEEPSEEK_BASE_URL", "").strip()
|
||||
or os.environ.get("NEW_API_BASE_URL", "").strip()
|
||||
or values.get("DEEPSEEK_BASE_URL", "").strip()
|
||||
or values.get("NEW_API_BASE_URL", "").strip()
|
||||
or "https://api.deepseek.com"
|
||||
)
|
||||
model = (
|
||||
model_override.strip()
|
||||
or os.environ.get("DEEPSEEK_MODEL", "").strip()
|
||||
or os.environ.get("NEW_API_MODEL", "").strip()
|
||||
or values.get("DEEPSEEK_MODEL", "").strip()
|
||||
or values.get("NEW_API_MODEL", "").strip()
|
||||
or "deepseek-chat"
|
||||
)
|
||||
return {"api_key": api_key, "base_url": base_url, "model": model}
|
||||
|
||||
|
||||
def select_primary_scope(summaries: Iterable[Dict[str, Any]], requested: str = "") -> str:
|
||||
if str(requested or "").strip():
|
||||
return str(requested).strip()
|
||||
counts: Dict[str, int] = defaultdict(int)
|
||||
for item in summaries:
|
||||
scope = str(item.get("scope_id") or "").strip()
|
||||
stage = str(item.get("stage") or "").strip()
|
||||
if scope and stage not in {"summary", "genome_prepare", "config_autofill"}:
|
||||
counts[scope] += 1
|
||||
if not counts:
|
||||
return ""
|
||||
return max(
|
||||
counts,
|
||||
key=lambda scope: (
|
||||
int("target_plus_reference" in scope),
|
||||
int("subfamily_evidence" not in scope),
|
||||
counts[scope],
|
||||
scope,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _infer_stage(name: str, role: str = "") -> str:
|
||||
text = f"{role} {name}".lower()
|
||||
if "tfbs" in text or "motif_source" in text or "gene_motif" in text:
|
||||
return "tfbs"
|
||||
if "interspecies" in text:
|
||||
return "interspecies_synteny"
|
||||
if "circular_tree" in text or ".tree." in text or text.endswith("tree.png"):
|
||||
return "phylogeny"
|
||||
if "protparam" in text or "tmhmm" in text or "pep_" in text:
|
||||
return "pep"
|
||||
for stage in STAGE_HINTS:
|
||||
if stage in text:
|
||||
return stage
|
||||
return "other"
|
||||
|
||||
|
||||
def _logical_figure_key(path: Path) -> str:
|
||||
return path.stem.lower()
|
||||
|
||||
|
||||
def collect_figure_inventory(
|
||||
final_dir: Path,
|
||||
summaries: Iterable[Dict[str, Any]],
|
||||
*,
|
||||
scope_id: str = "",
|
||||
) -> List[Dict[str, Any]]:
|
||||
artifact_map: Dict[str, Dict[str, str]] = {}
|
||||
for item in summaries:
|
||||
item_scope = str(item.get("scope_id") or "")
|
||||
if scope_id and item_scope and item_scope != scope_id:
|
||||
# Foundational target-only figures are not reused in a reference scope.
|
||||
continue
|
||||
stage = str(item.get("stage") or "")
|
||||
for artifact in item.get("artifact_rows") or []:
|
||||
raw = str(artifact.get("path") or "")
|
||||
if not raw:
|
||||
continue
|
||||
path = Path(raw)
|
||||
if path.suffix.lower() not in IMAGE_SUFFIXES:
|
||||
continue
|
||||
artifact_map[path.name] = {
|
||||
"stage": stage or _infer_stage(path.name, str(artifact.get("artifact") or "")),
|
||||
"role": str(artifact.get("artifact") or ""),
|
||||
}
|
||||
|
||||
candidates: List[Path] = []
|
||||
if final_dir.is_dir():
|
||||
candidates.extend(
|
||||
path for path in final_dir.iterdir()
|
||||
if path.is_file()
|
||||
and path.suffix.lower() in IMAGE_SUFFIXES
|
||||
and ".bak" not in path.name.lower()
|
||||
)
|
||||
|
||||
grouped: Dict[str, List[Path]] = defaultdict(list)
|
||||
for path in candidates:
|
||||
grouped[_logical_figure_key(path)].append(path)
|
||||
|
||||
inventory: List[Dict[str, Any]] = []
|
||||
seen_content: Dict[str, str] = {}
|
||||
preference = {".png": 0, ".jpg": 1, ".jpeg": 1, ".svg": 2, ".pdf": 3}
|
||||
for _, paths in sorted(grouped.items()):
|
||||
paths.sort(key=lambda p: (preference.get(p.suffix.lower(), 9), p.name))
|
||||
preferred = paths[0]
|
||||
try:
|
||||
digest = hashlib.sha256(preferred.read_bytes()).hexdigest()
|
||||
except OSError:
|
||||
digest = ""
|
||||
if digest and digest in seen_content:
|
||||
continue
|
||||
if digest:
|
||||
seen_content[digest] = preferred.name
|
||||
mapped = artifact_map.get(preferred.name, {})
|
||||
role = mapped.get("role", "")
|
||||
stage = mapped.get("stage", "") or _infer_stage(preferred.name, role)
|
||||
inventory.append({
|
||||
"figure_id": f"FIG-{len(inventory) + 1:03d}",
|
||||
"file": preferred.name,
|
||||
"stage": stage,
|
||||
"artifact_role": role,
|
||||
"available_formats": sorted({p.suffix.lower().lstrip(".") for p in paths}),
|
||||
"marker": f"[[FIG:{preferred.name}]]",
|
||||
})
|
||||
return inventory
|
||||
|
||||
|
||||
def _foundation_identify(final_dir: Path, prefix: str) -> Dict[str, Any]:
|
||||
candidates: List[Path] = []
|
||||
exact = final_dir / f"{prefix}.identify.manifest.json"
|
||||
if exact.is_file():
|
||||
candidates.append(exact)
|
||||
candidates.extend(
|
||||
path for path in final_dir.glob(f"{prefix}*.identify.manifest.json")
|
||||
if path not in candidates
|
||||
)
|
||||
for path in candidates:
|
||||
manifest = _read_json(path)
|
||||
if manifest.get("stage") == "identify" and manifest.get("status") in {"success", "partial_success"}:
|
||||
bundle = extract_stage_evidence("identify", manifest)
|
||||
return {
|
||||
"manifest": path.name,
|
||||
"scope_id": manifest.get("scope_id", ""),
|
||||
"status": manifest.get("status", ""),
|
||||
"metrics": manifest.get("metrics", {}),
|
||||
"parsed": bundle.get("parsed", {}),
|
||||
"evidence": bundle.get("evidence", []),
|
||||
"risks": bundle.get("risks", []),
|
||||
}
|
||||
return {}
|
||||
|
||||
|
||||
def _compact_report(report: Dict[str, Any]) -> Dict[str, Any]:
|
||||
evidence_cards = report.get("evidence_cards") or []
|
||||
return {
|
||||
"metadata": report.get("metadata", {}),
|
||||
"source_summary": report.get("source_summary", {}),
|
||||
"stage_flow": report.get("stage_flow", {}),
|
||||
"key_findings": report.get("key_findings", []),
|
||||
"structured_result_interpretations": report.get("structured_result_interpretations", {}),
|
||||
"grouped_result_interpretations": report.get("grouped_result_interpretations", {}),
|
||||
"scientific_discoveries": report.get("scientific_discoveries", []),
|
||||
"risk_flags": report.get("risk_flags", []),
|
||||
"stage_reviews": report.get("stage_reviews", []),
|
||||
"publication_claims": report.get("publication_claims", []),
|
||||
"major_claims": report.get("major_claims", []),
|
||||
"next_steps": report.get("next_steps", []),
|
||||
"evidence_cards": evidence_cards[:100],
|
||||
}
|
||||
|
||||
|
||||
def build_summary_evidence(
|
||||
*,
|
||||
final_dir: Path,
|
||||
prefix: str,
|
||||
job_id: str,
|
||||
summaries: List[Dict[str, Any]],
|
||||
scope_id: str,
|
||||
) -> Dict[str, Any]:
|
||||
report = build_evidence_report(
|
||||
final_dir,
|
||||
prefix,
|
||||
job_id=job_id,
|
||||
scope_id=scope_id,
|
||||
analysis_route="target_plus_reference" if "target_plus_reference" in scope_id else "",
|
||||
)
|
||||
if not report:
|
||||
raise DeepSeekSummaryError(f"No evidence report could be built for scope: {scope_id or 'default'}")
|
||||
evidence = _compact_report(report)
|
||||
included = list((evidence.get("metadata") or {}).get("included_stages") or [])
|
||||
identify = _foundation_identify(final_dir, prefix)
|
||||
if identify and "identify" not in included:
|
||||
evidence["foundation_identify"] = identify
|
||||
included.insert(0, "identify")
|
||||
identify_metrics = identify.get("metrics") or {}
|
||||
final_members = identify_metrics.get("final_members")
|
||||
candidate_members = identify_metrics.get("candidate_members")
|
||||
hmm_members = identify_metrics.get("hmm_members")
|
||||
blast_members = identify_metrics.get("blast_members")
|
||||
evidence.setdefault("structured_result_interpretations", {})["identify"] = {
|
||||
"stage": "identify",
|
||||
"stage_name": "基因家族成员鉴定",
|
||||
"status": identify.get("status") or "success",
|
||||
"interpretation_level": "publishable_result",
|
||||
"result_interpretation": (
|
||||
f"目标物种全蛋白集共有 {identify_metrics.get('total_proteins', 'N/A')} 条蛋白;"
|
||||
f"HMM 搜索得到 {hmm_members} 个命中,BLAST 搜索记录 {blast_members} 个命中,"
|
||||
f"联合筛选形成 {candidate_members} 个候选,最终确认 {final_members} 个家族成员。"
|
||||
"HMM、BLAST、候选集和最终集的关系必须按本地记录描述,不能自行写成取交集或取并集。"
|
||||
),
|
||||
"evidence_basis": identify.get("evidence") or [],
|
||||
"limitations": [
|
||||
"鉴定证据来自目标物种 target_only 基础清单,并作为 target_plus_reference 后续分析的成员基础。",
|
||||
"鉴定成功不能直接证明所有成员具有相同功能。",
|
||||
],
|
||||
"boundary": "不得把全基因组蛋白数写成家族成员数,也不得猜测联合筛选的集合运算方式。",
|
||||
}
|
||||
metadata = evidence["metadata"]
|
||||
metadata["included_stages"] = included
|
||||
metadata["included_stages_for_deepseek"] = included
|
||||
missing = list(evidence["metadata"].get("missing_expected_stages") or [])
|
||||
missing = [x for x in missing if x != "identify"]
|
||||
metadata["missing_expected_stages"] = missing
|
||||
metadata["missing_expected_stages_for_deepseek"] = missing
|
||||
expected = list(metadata.get("expected_stages") or [])
|
||||
metadata["completed_expected_stages"] = [stage for stage in expected if stage in included]
|
||||
metadata["stage_count"] = len(included)
|
||||
metadata["is_final"] = bool(expected) and not missing
|
||||
stage_flow = evidence.setdefault("stage_flow", {})
|
||||
stages = list(stage_flow.get("stages") or [])
|
||||
stages.insert(0, {
|
||||
"id": "identify",
|
||||
"name": "基因家族成员鉴定",
|
||||
"status": identify.get("status") or "success",
|
||||
"key_result": f"最终确认 {final_members} 个成员",
|
||||
})
|
||||
stage_flow["stages"] = stages
|
||||
stage_flow["visualization"] = " -> ".join(item.get("id", "") for item in stages)
|
||||
source_summary = evidence.setdefault("source_summary", {})
|
||||
source_summary["manifest_count"] = int(source_summary.get("manifest_count") or 0) + 1
|
||||
evidence["figures"] = collect_figure_inventory(final_dir, summaries, scope_id=scope_id)
|
||||
return evidence
|
||||
|
||||
|
||||
def _system_prompt() -> str:
|
||||
return """你是植物基因家族分析结果报告作者。你只能使用用户提供的本地证据,不能补造数字、成员、分组、功能或文献。
|
||||
任务是把机器结果写成普通研究人员能直接看懂的中文报告。少用专业词;首次出现的专业词必须用一句日常语言解释。
|
||||
每张图必须逐图解读,但不能假装看到了证据包未提供的像素细节。若只有文件名和阶段事实,应明确写“结合该阶段统计进行解释,具体成员级图形细节需打开原图复核”。
|
||||
所有关联性、预测性结果都必须保留边界,不得使用“证明、决定、必然、直接调控”等过强措辞。
|
||||
风险清单优先级高于一般解释。高风险阶段不得写成可靠生物学结论。
|
||||
输出只允许完整 Markdown 正文,不要输出代码围栏、JSON、写作说明或思考过程。"""
|
||||
|
||||
|
||||
def _user_prompt(*, framework: str, evidence: Dict[str, Any], job_id: str, prefix: str, scope_id: str) -> str:
|
||||
figures = evidence.get("figures") or []
|
||||
return f"""请为真实任务生成最终中文报告。
|
||||
|
||||
任务编号:{job_id}
|
||||
分析前缀:{prefix}
|
||||
分析范围:{scope_id or '默认范围'}
|
||||
|
||||
硬性要求:
|
||||
1. 按“鉴定→系统发育与分类→motif→domain→基因结构→组合图→启动子/TFBS→种内与种间共线性→染色体分布→ortholog→PEP→综合结论”的顺序组织。
|
||||
2. 首页给出普通读者能看懂的一页式摘要,并列出完成阶段、缺失阶段和最重要风险。
|
||||
3. 对 figures 中的每个逻辑图写一个小节,并原样保留对应 marker,例如 [[FIG:xxx.png]]。共有 {len(figures)} 个逻辑图;同一图的 PDF/PNG 不重复解释。
|
||||
4. 每个图小节写:图用来回答什么、从本任务事实能看出什么、通俗解释、不能说明什么。没有足够的成员级数值时必须直说,不能编造。
|
||||
5. 核型阶段存在高风险时,必须明确说明不能据图得出染色体分布结论。
|
||||
6. 种内共线性没有检测到家族成员基因对时,不能写成片段复制驱动家族扩张。
|
||||
7. 最终成员数以 identify 事实为准;全基因组背景数量不能写成家族成员数。
|
||||
8. 报告末尾给出候选成员筛选框架、主要限制、下一步验证建议和“读者最需要记住的五句话”。
|
||||
9. 不要保留任何 {{{{占位符}}}}、TODO 或空表格;没有数据的内容直接写“本任务未提供”。
|
||||
10. 语言以短句为主,结论先行。专业词后立即补通俗解释。
|
||||
|
||||
以下是报告结构参考。它决定覆盖范围和表达方式,不代表其中的示例数字是本任务事实:
|
||||
|
||||
--- 框架开始 ---
|
||||
{framework}
|
||||
--- 框架结束 ---
|
||||
|
||||
以下 JSON 是本任务唯一允许使用的事实来源:
|
||||
|
||||
--- 本地证据开始 ---
|
||||
{json.dumps(evidence, ensure_ascii=False, separators=(',', ':'))}
|
||||
--- 本地证据结束 ---
|
||||
"""
|
||||
|
||||
|
||||
def _strip_fence(text: str) -> str:
|
||||
value = str(text or "").strip()
|
||||
if value.startswith("```"):
|
||||
value = re.sub(r"^```(?:markdown|md)?\s*", "", value, flags=re.I)
|
||||
value = re.sub(r"\s*```$", "", value)
|
||||
return value.strip()
|
||||
|
||||
|
||||
def _validate_report(report: str, *, job_id: str, figures: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
markers = [str(item.get("marker") or "") for item in figures]
|
||||
missing = [marker for marker in markers if marker and marker not in report]
|
||||
heading_count = len(re.findall(r"(?m)^##+\s+", report))
|
||||
placeholders = sorted(set(re.findall(r"\{\{[^{}]+\}\}", report)))
|
||||
return {
|
||||
"character_count": len(report),
|
||||
"heading_count": heading_count,
|
||||
"job_id_present": job_id in report,
|
||||
"figure_count": len(markers),
|
||||
"covered_figure_count": len(markers) - len(missing),
|
||||
"missing_figure_markers": missing,
|
||||
"placeholders": placeholders,
|
||||
"passed": bool(len(report) >= 4000 and heading_count >= 8 and job_id in report and not missing and not placeholders),
|
||||
}
|
||||
|
||||
|
||||
def _supplement_prompt(missing: List[str], evidence: Dict[str, Any]) -> str:
|
||||
wanted = [item for item in evidence.get("figures") or [] if item.get("marker") in set(missing)]
|
||||
return f"""上一版报告漏掉了部分图。只输出一个 Markdown 章节,标题为“## 补充图表解读”。
|
||||
必须逐项原样保留 marker,并为每项写:图的用途、本地证据允许的解释、不能得出的结论。
|
||||
不得补造图中具体成员、颜色、分组或数值。
|
||||
|
||||
缺失图:
|
||||
{json.dumps(wanted, ensure_ascii=False)}
|
||||
|
||||
可用阶段解释和风险:
|
||||
{json.dumps({'structured': evidence.get('structured_result_interpretations', {}), 'risks': evidence.get('risk_flags', [])}, ensure_ascii=False)}
|
||||
"""
|
||||
|
||||
|
||||
def _items_for_stages(items: Iterable[Any], stages: set[str]) -> List[Any]:
|
||||
selected: List[Any] = []
|
||||
for item in items or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
item_stages = item.get("stages") or [item.get("stage")]
|
||||
if isinstance(item_stages, str):
|
||||
item_stages = [item_stages]
|
||||
if stages.intersection(str(stage) for stage in item_stages if stage):
|
||||
selected.append(item)
|
||||
return selected
|
||||
|
||||
|
||||
def _group_prompt(
|
||||
*,
|
||||
title: str,
|
||||
stages: List[str],
|
||||
evidence: Dict[str, Any],
|
||||
figures: List[Dict[str, Any]],
|
||||
) -> str:
|
||||
stage_set = set(stages)
|
||||
structured = evidence.get("structured_result_interpretations") or {}
|
||||
payload = {
|
||||
"stages": stages,
|
||||
"structured_result_interpretations": {
|
||||
stage: structured.get(stage) for stage in stages if structured.get(stage)
|
||||
},
|
||||
"key_findings": _items_for_stages(evidence.get("key_findings") or [], stage_set),
|
||||
"risk_flags": _items_for_stages(evidence.get("risk_flags") or [], stage_set),
|
||||
"stage_reviews": _items_for_stages(evidence.get("stage_reviews") or [], stage_set),
|
||||
"evidence_cards": _items_for_stages(evidence.get("evidence_cards") or [], stage_set)[:40],
|
||||
"figures": figures,
|
||||
}
|
||||
if "identify" in stage_set and evidence.get("foundation_identify"):
|
||||
payload["foundation_identify"] = evidence["foundation_identify"]
|
||||
return f"""请只撰写报告章节“## {title}”,不要写整篇报告,也不要重复首页信息。
|
||||
覆盖阶段:{', '.join(stages)}。
|
||||
|
||||
要求:
|
||||
- 先用一段通俗文字说明这些阶段得到了什么,再按图逐项解释。
|
||||
- 共有 {len(figures)} 个逻辑图。每张图必须建立三级标题,并原样保留 marker。
|
||||
- 每个图写“图的用途—本任务可确认的结果—通俗解释—结论边界”。
|
||||
- 没有成员级或像素级事实时,必须写需要打开原图复核,不能猜颜色、成员和具体分组。
|
||||
- 风险提示优先。高风险结果必须明确降级。
|
||||
- 本章节最后写一个“阶段小结”,只总结本地证据。
|
||||
- 只输出 Markdown 章节正文。
|
||||
|
||||
本地证据:
|
||||
{json.dumps(payload, ensure_ascii=False, separators=(',', ':'))}
|
||||
"""
|
||||
|
||||
|
||||
def _overview_prompt(*, evidence: Dict[str, Any], job_id: str, prefix: str, scope_id: str) -> str:
|
||||
payload = {
|
||||
"metadata": evidence.get("metadata", {}),
|
||||
"source_summary": evidence.get("source_summary", {}),
|
||||
"stage_flow": evidence.get("stage_flow", {}),
|
||||
"foundation_identify": evidence.get("foundation_identify", {}),
|
||||
"risk_flags": evidence.get("risk_flags", []),
|
||||
}
|
||||
return f"""请撰写基因家族结果报告的开头部分,只输出以下三个二级章节:
|
||||
## 一、项目与数据范围
|
||||
## 二、一页式结论摘要
|
||||
## 三、流程完成情况与可信度
|
||||
|
||||
任务编号:{job_id}
|
||||
前缀:{prefix}
|
||||
范围:{scope_id}
|
||||
|
||||
使用短句和通俗中文。摘要写5—8条最重要结论。必须写清最终成员数、完成阶段、缺失阶段、主要风险和结论边界。
|
||||
禁止补造证据;不能把全基因组背景数写成家族成员数。只输出 Markdown 章节正文。
|
||||
|
||||
本地证据:
|
||||
{json.dumps(payload, ensure_ascii=False, separators=(',', ':'))}
|
||||
"""
|
||||
|
||||
|
||||
def _conclusion_prompt(*, evidence: Dict[str, Any]) -> str:
|
||||
payload = {
|
||||
"scientific_discoveries": evidence.get("scientific_discoveries", []),
|
||||
"publication_claims": evidence.get("publication_claims", []),
|
||||
"major_claims": evidence.get("major_claims", []),
|
||||
"risk_flags": evidence.get("risk_flags", []),
|
||||
"next_steps": evidence.get("next_steps", []),
|
||||
"foundation_identify": evidence.get("foundation_identify", {}),
|
||||
}
|
||||
return f"""请撰写基因家族结果报告的结尾,只输出以下四个二级章节:
|
||||
## 十、候选成员与多证据综合
|
||||
## 十一、主要限制
|
||||
## 十二、下一步验证建议
|
||||
## 十三、读者最需要记住的五句话
|
||||
|
||||
候选成员没有足够成员级证据时,不要强行编排名;改为给出可执行筛选规则和待补证据。
|
||||
把预测和相关结果写成线索,不得写成已证明功能。高风险问题必须进入主要限制并排在修复建议前面。
|
||||
只输出 Markdown 章节正文。
|
||||
|
||||
本地证据:
|
||||
{json.dumps(payload, ensure_ascii=False, separators=(',', ':'))}
|
||||
"""
|
||||
|
||||
|
||||
def _call_section(
|
||||
provider: DeepSeekProvider,
|
||||
prompt: str,
|
||||
*,
|
||||
max_tokens: int,
|
||||
) -> tuple[str, Dict[str, Any]]:
|
||||
response = provider.call(
|
||||
[LLMMessage(role="system", content=_system_prompt()), LLMMessage(role="user", content=prompt)],
|
||||
temperature=0.12,
|
||||
max_tokens=max_tokens,
|
||||
disable_thinking=True,
|
||||
)
|
||||
text = _strip_fence(response.content)
|
||||
if len(text) < 300 or not re.search(r"(?m)^##\s+", text):
|
||||
raise DeepSeekSummaryError(
|
||||
f"DeepSeek returned an incomplete section (finish_reason={response.finish_reason}, chars={len(text)}): {text[:500]}"
|
||||
)
|
||||
return text, {
|
||||
"provider": response.provider,
|
||||
"model": response.model,
|
||||
"usage": response.usage or {},
|
||||
"finish_reason": response.finish_reason,
|
||||
}
|
||||
|
||||
|
||||
def generate_deepseek_summary(
|
||||
*,
|
||||
final_dir: Path,
|
||||
prefix: str,
|
||||
job_id: str,
|
||||
summaries: List[Dict[str, Any]],
|
||||
scope_id: str = "",
|
||||
framework_path: Optional[Path] = None,
|
||||
model: str = "",
|
||||
max_tokens: int = 16000,
|
||||
) -> Dict[str, Any]:
|
||||
final_dir = final_dir.resolve()
|
||||
effective_scope = select_primary_scope(summaries, scope_id)
|
||||
framework_file = (framework_path or DEFAULT_FRAMEWORK).expanduser().resolve()
|
||||
if not framework_file.is_file():
|
||||
raise DeepSeekSummaryError(f"Report framework not found: {framework_file}")
|
||||
evidence = build_summary_evidence(
|
||||
final_dir=final_dir,
|
||||
prefix=prefix,
|
||||
job_id=job_id,
|
||||
summaries=summaries,
|
||||
scope_id=effective_scope,
|
||||
)
|
||||
settings = load_deepseek_settings(model_override=model)
|
||||
provider = DeepSeekProvider(
|
||||
api_key=settings["api_key"],
|
||||
model=settings["model"],
|
||||
temperature=0.15,
|
||||
max_tokens=max_tokens,
|
||||
api_base=settings["base_url"],
|
||||
)
|
||||
started = time.time()
|
||||
framework = framework_file.read_text(encoding="utf-8", errors="ignore")
|
||||
framework_outline = "\n".join(
|
||||
line for line in framework.splitlines() if line.startswith("## ")
|
||||
)
|
||||
section_records: List[Dict[str, Any]] = []
|
||||
overview, call_meta = _call_section(
|
||||
provider,
|
||||
_overview_prompt(
|
||||
evidence=evidence,
|
||||
job_id=job_id,
|
||||
prefix=prefix,
|
||||
scope_id=effective_scope,
|
||||
) + "\n\n报告框架章节纲要:\n" + framework_outline,
|
||||
max_tokens=min(max_tokens, 4500),
|
||||
)
|
||||
section_records.append({"section": "overview", **call_meta})
|
||||
|
||||
figures = evidence.get("figures") or []
|
||||
group_specs = [
|
||||
(
|
||||
"四、成员鉴定、系统发育与分类",
|
||||
["identify", "phylogeny", "reference_classification", "subfamily_confirm"],
|
||||
),
|
||||
(
|
||||
"五、保守片段、功能区域、基因结构与组合证据",
|
||||
["motif", "domain", "structure", "combo"],
|
||||
),
|
||||
(
|
||||
"六、启动子与 TFBS 调控线索",
|
||||
["promoter", "tfbs"],
|
||||
),
|
||||
(
|
||||
"七、基因组分布、共线性与跨物种关系",
|
||||
["synteny", "interspecies_synteny", "karyotype", "ortholog", "kaks"],
|
||||
),
|
||||
(
|
||||
"八、候选蛋白 PEP 特征",
|
||||
["pep"],
|
||||
),
|
||||
(
|
||||
"九、其他结果图",
|
||||
["other"],
|
||||
),
|
||||
]
|
||||
sections = [overview]
|
||||
covered_stages: set[str] = set()
|
||||
for title, stages in group_specs:
|
||||
group_figures = [item for item in figures if str(item.get("stage") or "other") in set(stages)]
|
||||
structured = evidence.get("structured_result_interpretations") or {}
|
||||
has_stage_evidence = any(structured.get(stage) for stage in stages) or (
|
||||
"identify" in stages and evidence.get("foundation_identify")
|
||||
)
|
||||
if not group_figures and not has_stage_evidence:
|
||||
continue
|
||||
section_text, call_meta = _call_section(
|
||||
provider,
|
||||
_group_prompt(title=title, stages=stages, evidence=evidence, figures=group_figures),
|
||||
max_tokens=min(max_tokens, 6500),
|
||||
)
|
||||
sections.append(section_text)
|
||||
section_records.append({
|
||||
"section": title,
|
||||
"stages": stages,
|
||||
"figure_count": len(group_figures),
|
||||
**call_meta,
|
||||
})
|
||||
covered_stages.update(stages)
|
||||
|
||||
conclusion, call_meta = _call_section(
|
||||
provider,
|
||||
_conclusion_prompt(evidence=evidence),
|
||||
max_tokens=min(max_tokens, 5000),
|
||||
)
|
||||
sections.append(conclusion)
|
||||
section_records.append({"section": "conclusion", **call_meta})
|
||||
report = (
|
||||
f"# {prefix} 基因家族分析结果通俗解读报告\n\n"
|
||||
f"任务编号:`{job_id}` \n"
|
||||
f"分析范围:`{effective_scope or 'default'}` \n"
|
||||
f"生成方式:DeepSeek API 分段生成;本地证据锁定 \n\n"
|
||||
+ "\n\n".join(sections)
|
||||
)
|
||||
validation = _validate_report(report, job_id=job_id, figures=evidence.get("figures") or [])
|
||||
supplement_usage: Dict[str, Any] = {}
|
||||
if validation["missing_figure_markers"]:
|
||||
supplement = provider.call(
|
||||
[
|
||||
LLMMessage(role="system", content=_system_prompt()),
|
||||
LLMMessage(role="user", content=_supplement_prompt(validation["missing_figure_markers"], evidence)),
|
||||
],
|
||||
temperature=0.1,
|
||||
max_tokens=min(max_tokens, 8000),
|
||||
disable_thinking=True,
|
||||
)
|
||||
supplement_text = _strip_fence(supplement.content)
|
||||
if supplement_text:
|
||||
report = report.rstrip() + "\n\n" + supplement_text + "\n"
|
||||
supplement_usage = supplement.usage or {}
|
||||
section_records.append({
|
||||
"section": "missing_figure_supplement",
|
||||
"usage": supplement_usage,
|
||||
"finish_reason": supplement.finish_reason,
|
||||
})
|
||||
validation = _validate_report(report, job_id=job_id, figures=evidence.get("figures") or [])
|
||||
if not validation["passed"]:
|
||||
raise DeepSeekSummaryError(
|
||||
"DeepSeek report failed validation: "
|
||||
+ json.dumps(validation, ensure_ascii=False)
|
||||
)
|
||||
|
||||
scope_token = re.sub(r"[^0-9A-Za-z_.-]+", "-", effective_scope).strip(".-")
|
||||
name_root = f"{prefix}.{scope_token}" if scope_token else prefix
|
||||
report_path = final_dir / f"{name_root}.summary_report.md"
|
||||
evidence_path = final_dir / f"{name_root}.summary.evidence.json"
|
||||
metadata_path = final_dir / f"{name_root}.summary.deepseek.json"
|
||||
evidence_path.write_text(json.dumps(evidence, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
report_path.write_text(report.rstrip() + "\n", encoding="utf-8")
|
||||
metadata = {
|
||||
"schema": "deepseek_summary_generation_v1",
|
||||
"status": "success",
|
||||
"generated_at": datetime.now().isoformat(),
|
||||
"job_id": job_id,
|
||||
"prefix": prefix,
|
||||
"scope_id": effective_scope,
|
||||
"provider": "deepseek",
|
||||
"model": settings["model"],
|
||||
"base_url": settings["base_url"],
|
||||
"duration_seconds": round(time.time() - started, 3),
|
||||
"section_calls": section_records,
|
||||
"supplement_usage": supplement_usage,
|
||||
"framework": str(framework_file),
|
||||
"evidence_file": str(evidence_path),
|
||||
"report_file": str(report_path),
|
||||
"validation": validation,
|
||||
}
|
||||
metadata_path.write_text(json.dumps(metadata, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
return {
|
||||
"scope_id": effective_scope,
|
||||
"report_path": report_path,
|
||||
"evidence_path": evidence_path,
|
||||
"metadata_path": metadata_path,
|
||||
"metadata": metadata,
|
||||
}
|
||||
@@ -52,6 +52,7 @@ STAGE_SCRIPT_MAP = {
|
||||
"annotation_kegg": "run_annotation.py",
|
||||
"rnaseq_wgcna": "run_rnaseq_wgcna.py",
|
||||
"rnaseq_candidate_family": "run_rnaseq_candidate_family.py",
|
||||
"rnaseq_report": "run_rnaseq_report.py",
|
||||
"rnaseq_sample_qc": "run_rnaseq_sample_qc.py",
|
||||
"genome_prepare": "run_genome_prepare.py",
|
||||
"genome_prepare_v2": "run_genome_prepare.py",
|
||||
@@ -341,6 +342,11 @@ DIAGNOSIS_STAGE_COVERAGE = {
|
||||
"postflight": ["candidate_family_ranking", "evidence_matrix", "recommendation", "rag_queries", "rag_references"],
|
||||
"failure": ["annotation_missing", "id_mapping"],
|
||||
},
|
||||
"rnaseq_report": {
|
||||
"preflight": ["successful_rnaseq_manifest", "prefix_scope_identity"],
|
||||
"postflight": ["report_markdown", "evidence_snapshot", "coverage_status"],
|
||||
"failure": ["no_successful_rnaseq_evidence", "scope_mismatch", "artifact_missing"],
|
||||
},
|
||||
"annotation": {
|
||||
"preflight": ["protein_background", "tool_availability"],
|
||||
"postflight": ["gene2go", "gene2ko", "annotation_coverage"],
|
||||
@@ -390,48 +396,57 @@ class DiagnosisAgent(BaseAgent):
|
||||
ERROR_PATTERNS = {
|
||||
"missing_input_file": {
|
||||
"patterns": [r"文件不存在", r"FileNotFoundError", r"No such file", r"找不到文件"],
|
||||
"suggestion": "检查输入文件路径是否正确,确保文件已上传到input目录",
|
||||
"suggestion_zh": "检查输入文件路径是否正确,确保文件已上传到input目录",
|
||||
"suggestion_en": "Check if input file path is correct; ensure files are uploaded to the input directory",
|
||||
"auto_fix": False
|
||||
},
|
||||
"hmm_not_found": {
|
||||
"patterns": [r"required_pfams is empty", r"No HMM file", r"找不到HMM"],
|
||||
"suggestion": "请上传HMM模型文件或指定--required-pfams参数",
|
||||
"suggestion_zh": "请上传HMM模型文件或指定--required-pfams参数",
|
||||
"suggestion_en": "Please upload HMM model file or specify --required-pfams parameter",
|
||||
"auto_fix": False
|
||||
},
|
||||
"blast_db_error": {
|
||||
"patterns": [r"BLAST Database error", r"makeblastdb failed"],
|
||||
"suggestion": "Identify BLAST 固定用目标物种全蛋白 FASTA 建库;请优先检查目标全蛋白 FASTA 是否存在、非空且为有效蛋白FASTA格式,再检查参考家族蛋白 query 文件。",
|
||||
"suggestion_zh": "Identify BLAST 固定用目标物种全蛋白 FASTA 建库;请优先检查目标全蛋白 FASTA 是否存在、非空且为有效蛋白FASTA格式,再检查参考家族蛋白 query 文件。",
|
||||
"suggestion_en": "Identify BLAST uses target species full protein FASTA for database; check that target FASTA exists, is non-empty and valid, then check reference family protein query file.",
|
||||
"auto_fix": False
|
||||
},
|
||||
"memory_error": {
|
||||
"patterns": [r"MemoryError", r"out of memory", r"内存不足"],
|
||||
"suggestion": "减少--cpu参数值或分批处理,建议使用--cpu 4",
|
||||
"suggestion_zh": "减少--cpu参数值或分批处理,建议使用--cpu 4",
|
||||
"suggestion_en": "Reduce --cpu parameter or use batch processing; recommend --cpu 4",
|
||||
"auto_fix": True,
|
||||
"fix_action": "reduce_cpu"
|
||||
},
|
||||
"rnaseq_library_strandness_error": {
|
||||
"patterns": [r"library_strandness", r"链特异性", r"rna-strandness", r"featureCounts.*-s"],
|
||||
"suggestion": "请在 samples.yaml 的 rnaseq.library_strandness 或每个 samples.<sample>.library_strandness 中明确填写 non_stranded / forward / reverse;混合链特异性样本当前不能用单次 featureCounts 定量。",
|
||||
"suggestion_zh": "请在 samples.yaml 的 rnaseq.library_strandness 或每个 samples.<sample>.library_strandness 中明确填写 non_stranded / forward / reverse;混合链特异性样本当前不能用单次 featureCounts 定量。",
|
||||
"suggestion_en": "Please specify non_stranded / forward / reverse in samples.yaml rnaseq.library_strandness or each samples.<sample>.library_strandness; mixed strand-specific samples cannot be quantified in a single featureCounts run.",
|
||||
"auto_fix": False
|
||||
},
|
||||
"timeout_error": {
|
||||
"patterns": [r"Timeout", r"timed out", r"超时"],
|
||||
"suggestion": "增加超时时间或检查输入数据大小",
|
||||
"suggestion_zh": "增加超时时间或检查输入数据大小",
|
||||
"suggestion_en": "Increase timeout or check input data size",
|
||||
"auto_fix": False
|
||||
},
|
||||
"conda_env_error": {
|
||||
"patterns": [r"conda", r"Environment", r"找不到命令"],
|
||||
"suggestion": "检查conda环境是否正确激活,确保工具已安装",
|
||||
"suggestion_zh": "检查conda环境是否正确激活,确保工具已安装",
|
||||
"suggestion_en": "Check that conda environment is activated and tools are installed",
|
||||
"auto_fix": False
|
||||
},
|
||||
"permission_error": {
|
||||
"patterns": [r"Permission denied", r"权限不足"],
|
||||
"suggestion": "检查文件权限,确保有读写权限",
|
||||
"suggestion_zh": "检查文件权限,确保有读写权限",
|
||||
"suggestion_en": "Check file permissions; ensure read/write access",
|
||||
"auto_fix": False
|
||||
},
|
||||
"invalid_parameter": {
|
||||
"patterns": [r"unrecognized arguments", r"invalid parameter", r"参数错误"],
|
||||
"suggestion": "检查参数名称是否正确,注意使用短横线而非下划线",
|
||||
"suggestion_zh": "检查参数名称是否正确,注意使用短横线而非下划线",
|
||||
"suggestion_en": "Check parameter name; use hyphens instead of underscores",
|
||||
"auto_fix": True,
|
||||
"fix_action": "fix_parameter_format"
|
||||
},
|
||||
@@ -448,7 +463,8 @@ class DiagnosisAgent(BaseAgent):
|
||||
r"RuntimeError: MAFFT失败.*No such file",
|
||||
r"RuntimeError:.*失败.*No such file or directory"
|
||||
],
|
||||
"suggestion": "生物信息学工具未安装或未添加到PATH。建议:1) 激活conda环境: conda activate test4py311 2) 安装缺失工具: conda install -c bioconda mafft blast iqtree hmmer",
|
||||
"suggestion_zh": "生物信息学工具未安装或未添加到PATH。建议:1) 激活conda环境: conda activate test4py311 2) 安装缺失工具: conda install -c bioconda mafft blast iqtree hmmer",
|
||||
"suggestion_en": "Bioinformatics tools not installed or not in PATH. Suggestions: 1) activate conda environment: conda activate test4py311 2) install missing tools: conda install -c bioconda mafft blast iqtree hmmer",
|
||||
"auto_fix": False,
|
||||
"severity": "high"
|
||||
},
|
||||
@@ -461,7 +477,8 @@ class DiagnosisAgent(BaseAgent):
|
||||
r"requires.*stage.*not completed",
|
||||
r"dependency.*not satisfied"
|
||||
],
|
||||
"suggestion": "前置分析阶段未完成,系统将自动先执行依赖阶段",
|
||||
"suggestion_zh": "前置分析阶段未完成,系统将自动先执行依赖阶段",
|
||||
"suggestion_en": "Prerequisite analysis stage not completed; system will execute dependency stage first automatically",
|
||||
"auto_fix": True,
|
||||
"fix_action": "execute_dependencies",
|
||||
"severity": "high"
|
||||
@@ -601,6 +618,7 @@ class DiagnosisAgent(BaseAgent):
|
||||
prepare_patch_workspace: bool = False,
|
||||
scope_id: str = "",
|
||||
params: Optional[Dict[str, Any]] = None,
|
||||
language: str = "zh",
|
||||
) -> Dict[str, Any]:
|
||||
"""只读审计某个 job/stage 的产物、manifest、日志和输入配置。
|
||||
|
||||
@@ -930,6 +948,7 @@ class DiagnosisAgent(BaseAgent):
|
||||
gate: str = "preflight",
|
||||
project_root: Path = PROJECT_ROOT,
|
||||
params: Optional[Dict[str, Any]] = None,
|
||||
language: str = "zh",
|
||||
) -> Dict[str, Any]:
|
||||
"""Global QA gate for stage preflight/postflight/failure.
|
||||
|
||||
@@ -4490,10 +4509,10 @@ class DiagnosisAgent(BaseAgent):
|
||||
except Exception as exc:
|
||||
print(f"[DiagnosisAgent] Failed to remember LLM diagnosis case: {exc}")
|
||||
|
||||
def _match_error_pattern(self, error_text: str) -> Optional[Dict]:
|
||||
def _match_error_pattern(self, error_text: str, language: str = "zh") -> Optional[Dict]:
|
||||
"""匹配错误模式"""
|
||||
error_lower = error_text.lower()
|
||||
|
||||
|
||||
priority_order = [
|
||||
"tool_not_found",
|
||||
"dependency_failed",
|
||||
@@ -4507,15 +4526,16 @@ class DiagnosisAgent(BaseAgent):
|
||||
"permission_error",
|
||||
"timeout_error"
|
||||
]
|
||||
|
||||
|
||||
for error_type in priority_order:
|
||||
if error_type in self.ERROR_PATTERNS:
|
||||
pattern_info = self.ERROR_PATTERNS[error_type]
|
||||
for pattern in pattern_info["patterns"]:
|
||||
if re.search(pattern, error_text, re.IGNORECASE):
|
||||
suggestion_key = "suggestion_en" if language == "en" else "suggestion_zh"
|
||||
return {
|
||||
"error_type": error_type,
|
||||
"suggestion": pattern_info["suggestion"],
|
||||
"suggestion": pattern_info.get(suggestion_key, pattern_info.get("suggestion_zh", "")),
|
||||
"auto_fix": pattern_info.get("auto_fix", False),
|
||||
"fix_action": pattern_info.get("fix_action"),
|
||||
"severity": pattern_info.get("severity", "medium")
|
||||
|
||||
@@ -4,7 +4,7 @@ IntentAgent V2 - LLM驱动的智能意图识别
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
from typing import Dict, List, Optional, Any
|
||||
from typing import Any, Dict, List, Mapping, Optional
|
||||
from .base import AgentMessage, AgentState, BaseAgent, QueryMode
|
||||
from gfa_core.mas.llm import LLMManager
|
||||
from gfa_core.mas.stage_catalog import (
|
||||
@@ -74,6 +74,7 @@ class IntentAgentV2(BaseAgent):
|
||||
"rnaseq_de": "RNA-seq差异表达分析 - 从原始FASTQ数据开始,进行质控→比对→定量→差异表达分析(DESeq2)",
|
||||
"rnaseq_enrich": "RNA-seq富集分析 - GO/KEGG通路富集分析(RNA-seq流程最后一步)",
|
||||
"rnaseq_candidate_family": "RNA-seq候选家族筛选 - 整合DEG、InterPro/Pfam和GO/KEGG富集,输出候选基因家族排名",
|
||||
"rnaseq_report": "RNA-seq结果报告 - 从当前成功阶段生成可追溯的正式Markdown报告",
|
||||
"enrichment": "富集分析 - 基于基因家族鉴定结果进行GO/KEGG通路富集分析",
|
||||
"ncbi_fetch_sequences": "NCBI序列下载 - 根据GeneID、基因名或基因list下载protein/CDS序列"
|
||||
}
|
||||
@@ -152,6 +153,82 @@ class IntentAgentV2(BaseAgent):
|
||||
|
||||
def __init__(self, llm_manager: LLMManager):
|
||||
super().__init__(llm_manager)
|
||||
|
||||
@staticmethod
|
||||
def _apply_structured_goal_context(
|
||||
intent: Dict[str, Any],
|
||||
goal_context: Mapping[str, Any] | None,
|
||||
) -> Dict[str, Any]:
|
||||
if not isinstance(goal_context, Mapping):
|
||||
return intent
|
||||
terms = goal_context.get("user_terms")
|
||||
if not isinstance(terms, Mapping):
|
||||
return intent
|
||||
|
||||
def species_list(value: Any) -> List[str]:
|
||||
if isinstance(value, str):
|
||||
values = value.split(",")
|
||||
elif isinstance(value, (list, tuple, set)):
|
||||
values = value
|
||||
else:
|
||||
values = ()
|
||||
return list(
|
||||
dict.fromkeys(str(item).strip() for item in values if str(item).strip())
|
||||
)
|
||||
|
||||
normalized = dict(intent)
|
||||
params = dict(normalized.get("params") or {})
|
||||
targets = species_list(
|
||||
params.get("targetSpecies") or normalized.get("target_species")
|
||||
) or species_list(terms.get("target_species"))
|
||||
references = species_list(
|
||||
params.get("referenceSpecies") or normalized.get("reference_species")
|
||||
) or species_list(terms.get("reference_species"))
|
||||
comparatives = species_list(
|
||||
params.get("comparativeSpecies") or normalized.get("comparative_species")
|
||||
) or species_list(terms.get("comparative_species"))
|
||||
references = [species for species in references if species not in targets]
|
||||
comparatives = [
|
||||
species
|
||||
for species in comparatives
|
||||
if species not in targets and species not in references
|
||||
]
|
||||
family = str(
|
||||
normalized.get("gene_family") or terms.get("gene_family") or ""
|
||||
).strip()
|
||||
scope_mode = (
|
||||
"target_plus_reference_plus_comparative"
|
||||
if references and comparatives
|
||||
else "target_plus_reference"
|
||||
if references
|
||||
else "target_plus_comparative"
|
||||
if comparatives
|
||||
else "target_only"
|
||||
)
|
||||
params.update(
|
||||
{
|
||||
"targetSpecies": targets,
|
||||
"referenceSpecies": references,
|
||||
"comparativeSpecies": comparatives,
|
||||
"target_species": targets,
|
||||
"scopeMode": scope_mode,
|
||||
}
|
||||
)
|
||||
normalized.update(
|
||||
{
|
||||
"params": params,
|
||||
"gene_family": family,
|
||||
"target_species": targets,
|
||||
"reference_species": references,
|
||||
"comparative_species": comparatives,
|
||||
"scopeMode": scope_mode,
|
||||
}
|
||||
)
|
||||
if targets:
|
||||
normalized["species"] = targets[0] if len(targets) == 1 else "Multi"
|
||||
if normalized.get("species") and family:
|
||||
normalized["prefix"] = f"{normalized['species']}.{family}"
|
||||
return normalized
|
||||
|
||||
async def execute(self, state: AgentState, message: AgentMessage) -> AgentState:
|
||||
"""
|
||||
@@ -237,6 +314,11 @@ class IntentAgentV2(BaseAgent):
|
||||
print(f"[IntentAgentV2] Protected species from parser: {base_intent['species']}")
|
||||
intent_result.setdefault("task_kind", "pipeline")
|
||||
intent_result.setdefault("taskType", intent_result.get("task_kind", "pipeline"))
|
||||
|
||||
intent_result = self._apply_structured_goal_context(
|
||||
intent_result,
|
||||
message.payload.get("goal_context"),
|
||||
)
|
||||
|
||||
# 保留原始 stages,不过滤(留给 PlanningAgentV2 处理)
|
||||
original_stages = intent_result.get("stages", [])
|
||||
@@ -440,6 +522,22 @@ class IntentAgentV2(BaseAgent):
|
||||
has_rnaseq_marker = any(kw in query_lower for kw in ["rnaseq", "rna-seq", "rna seq", "转录组"])
|
||||
explicit_gene_family_context = any(kw in query_lower for kw in ["基因家族", "家族表达", "表达量", "热图", "heatmap"])
|
||||
|
||||
if has_rnaseq_marker and any(
|
||||
keyword in query_lower
|
||||
for keyword in ["生成结果报告", "打开结果报告", "结果报告", "生成报告", "打开报告"]
|
||||
):
|
||||
action = "open" if "打开" in query_lower else "generate"
|
||||
return {
|
||||
"gene_family": "",
|
||||
"species": "",
|
||||
"stages": ["rnaseq_report"],
|
||||
"primary_stage": "rnaseq_report",
|
||||
"description": "打开或生成当前RNA-seq正式结果报告",
|
||||
"requirements": ["至少一个成功的RNA-seq阶段manifest"],
|
||||
"params": {"rnaseqReportAction": action},
|
||||
"confidence": 0.99,
|
||||
}
|
||||
|
||||
stage_force_rules = [
|
||||
("rnaseq_candidate_family", ["候选家族", "候选基因家族", "家族排名", "tf注释", "tf family", "tffamily", "pfam注释", "deg注释", "差异基因注释"], "执行RNA-seq候选家族筛选"),
|
||||
("rnaseq_fgsea", ["fgsea", "gsea", "排序富集", "基因集富集"], "执行RNA-seq fgsea排序富集分析"),
|
||||
@@ -693,8 +791,17 @@ class IntentAgentV2(BaseAgent):
|
||||
intent_data["stages"] = (
|
||||
["rnaseq_de", "rnaseq_go_enrich", "rnaseq_kegg_enrich", "rnaseq_fgsea", "rnaseq_wgcna"]
|
||||
if has_rnaseq_keyword
|
||||
else ["identify"]
|
||||
else []
|
||||
)
|
||||
if not intent_data["stages"]:
|
||||
intent_data.update(
|
||||
{
|
||||
"description": "未能可靠识别要执行的分析阶段",
|
||||
"needs_clarification": True,
|
||||
"clarification_type": "analysis_stage",
|
||||
"clarification_question": "请说明要执行的具体分析阶段",
|
||||
}
|
||||
)
|
||||
|
||||
# 根据依赖关系补充缺失的阶段(关键修复:确保依赖展开)
|
||||
original_stages = intent_data["stages"]
|
||||
@@ -975,11 +1082,8 @@ class IntentAgentV2(BaseAgent):
|
||||
has_rnaseq_keyword = any(
|
||||
kw in query_lower for kw in ["rnaseq", "rna-seq", "rna seq", "转录组", "fastq"]
|
||||
)
|
||||
stages = (
|
||||
["rnaseq_de", "rnaseq_go_enrich", "rnaseq_kegg_enrich", "rnaseq_fgsea", "rnaseq_wgcna"]
|
||||
if has_rnaseq_keyword
|
||||
else ["identify"]
|
||||
)
|
||||
if has_rnaseq_keyword:
|
||||
stages = ["rnaseq_de", "rnaseq_go_enrich", "rnaseq_kegg_enrich", "rnaseq_fgsea", "rnaseq_wgcna"]
|
||||
|
||||
# 补充依赖(使用 stage_catalog 的主规划依赖链)
|
||||
original_stages = stages.copy()
|
||||
@@ -989,18 +1093,31 @@ class IntentAgentV2(BaseAgent):
|
||||
gene_family = self._extract_gene_family_fallback(query)
|
||||
species = "Multi" if any(kw in query.lower() for kw in ["基因家族成员鉴定", "基因家族鉴定", "identify"]) else ""
|
||||
|
||||
return {
|
||||
result = {
|
||||
"gene_family": gene_family,
|
||||
"species": species,
|
||||
"prefix": f"{species}.{gene_family}",
|
||||
"stages": stages,
|
||||
"description": f"执行{gene_family}基因家族的{', '.join(stages)}分析",
|
||||
"description": (
|
||||
f"执行{gene_family}基因家族的{', '.join(stages)}分析"
|
||||
if stages
|
||||
else "未能可靠识别要执行的分析阶段"
|
||||
),
|
||||
"requirements": [],
|
||||
"confidence": 0.7,
|
||||
"confidence": 0.7 if stages else 0.0,
|
||||
"task_kind": "pipeline",
|
||||
"taskType": "pipeline",
|
||||
"mode": "fallback"
|
||||
}
|
||||
if not stages:
|
||||
result.update(
|
||||
{
|
||||
"needs_clarification": True,
|
||||
"clarification_type": "analysis_stage",
|
||||
"clarification_question": "请说明要执行的具体分析阶段",
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def create_intent_agent_v2(llm_manager):
|
||||
|
||||
@@ -44,7 +44,7 @@ from gfa_core.wgs_upstream_manifest import (
|
||||
)
|
||||
|
||||
|
||||
GF_TARGET_ONLY_STAGE_PARAMS = {"promoter", "synteny", "karyotype", "pep", "tfbs"}
|
||||
GF_TARGET_ONLY_STAGE_PARAMS = {"promoter", "synteny", "karyotype", "pep", "tfbs", "ppi"}
|
||||
GF_REFERENCE_STAGE_PARAMS = {
|
||||
"phylogeny",
|
||||
"motif",
|
||||
@@ -54,7 +54,6 @@ GF_REFERENCE_STAGE_PARAMS = {
|
||||
"interspecies_synteny",
|
||||
"ortholog",
|
||||
"kaks",
|
||||
"ppi",
|
||||
}
|
||||
GF_SCOPE_IDENTITY_KEYS = (
|
||||
"scopeFile",
|
||||
@@ -377,6 +376,7 @@ class PlanningAgentV2(BaseAgent):
|
||||
"rnaseq_fgsea": [],
|
||||
"rnaseq_wgcna": [],
|
||||
"rnaseq_candidate_family": [],
|
||||
"rnaseq_report": [],
|
||||
}
|
||||
|
||||
# 各阶段依赖 (来自PreFlight)
|
||||
@@ -1958,17 +1958,6 @@ class PlanningAgentV2(BaseAgent):
|
||||
recommended_params = learning_recommendations.get("params", {})
|
||||
print(f"[PlanningAgentV2] 应用Learning推荐参数: {recommended_params}")
|
||||
|
||||
if "ppi" in stage_ids:
|
||||
intent_params = dict(intent_params or {})
|
||||
ppi_species = str(intent_params.get("ppiSpecies") or intent_params.get("ppi_species") or "").strip().lower()
|
||||
if ppi_species and ppi_species != "auto" and not intent_params.get("referenceSpecies"):
|
||||
ppi_ref_species = {"ath": ["At"], "osa": ["Os"], "zma": ["Zm"], "gma": ["Gm"], "sly": ["Sl"]}.get(ppi_species, [])
|
||||
if ppi_ref_species:
|
||||
intent_params["referenceSpecies"] = ppi_ref_species
|
||||
intent_params["reference_species"] = ppi_ref_species
|
||||
intent_params["refSpeciesList"] = ppi_ref_species
|
||||
intent_params.setdefault("scopeMode", "target_plus_reference")
|
||||
|
||||
for i, stage_id in enumerate(stage_ids):
|
||||
info = self.STAGE_INFO.get(stage_id, {"name": stage_id, "estimated_time": 10, "cpu": 8})
|
||||
deps = self.DEPENDENCIES.get(stage_id, [])
|
||||
|
||||
@@ -54,6 +54,7 @@ class RouterAgent(BaseAgent):
|
||||
"history": history,
|
||||
"session_id": str(payload.get("session_id") or state.session_id),
|
||||
"job_id": str(payload.get("job_id") or state.job_id),
|
||||
"language": str(payload.get("language") or "zh"),
|
||||
}
|
||||
if "goal_ledger" in payload:
|
||||
runtime_context["goal_ledger"] = payload.get("goal_ledger")
|
||||
|
||||
+1006
-234
File diff suppressed because it is too large
Load Diff
@@ -21,6 +21,35 @@ def _is_beneath(path: Path, parent: Path) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def _resolve_job_layout(root: Path, job_id: str) -> tuple[Path, Path, Path]:
|
||||
logical_input_job = root / "input" / job_id
|
||||
logical_workspace = root / "workspace" / job_id
|
||||
input_root = (root / "input").resolve(strict=False)
|
||||
workspace_root = (root / "workspace").resolve(strict=False)
|
||||
input_job = logical_input_job.resolve(strict=False)
|
||||
job_workspace = logical_workspace.resolve(strict=False)
|
||||
if _is_beneath(input_job, input_root) and _is_beneath(
|
||||
job_workspace,
|
||||
workspace_root,
|
||||
):
|
||||
return root, input_job, job_workspace
|
||||
|
||||
if not (logical_input_job.is_symlink() and logical_workspace.is_symlink()):
|
||||
if not _is_beneath(input_job, input_root):
|
||||
raise ValueError("input job path escapes project root")
|
||||
raise ValueError("workspace job path escapes project root")
|
||||
|
||||
input_job = logical_input_job.resolve(strict=True)
|
||||
job_workspace = logical_workspace.resolve(strict=True)
|
||||
linked_root = input_job.parents[1]
|
||||
if (
|
||||
input_job != linked_root / "input" / job_id
|
||||
or job_workspace != linked_root / "workspace" / job_id
|
||||
):
|
||||
raise ValueError("linked job paths do not share one project root")
|
||||
return linked_root, input_job, job_workspace
|
||||
|
||||
|
||||
def _read_manifest(path: Path) -> dict[str, Any]:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
if type(payload) is not dict:
|
||||
@@ -263,12 +292,7 @@ def build_confirmation_snapshot(
|
||||
if not isinstance(job_id, str) or not _JOB_ID.fullmatch(job_id):
|
||||
raise ValueError("invalid job_id")
|
||||
root = Path(project_root).expanduser().resolve(strict=True)
|
||||
input_job = (root / "input" / job_id).resolve(strict=False)
|
||||
job_workspace = (root / "workspace" / job_id).resolve(strict=False)
|
||||
if not _is_beneath(input_job, (root / "input").resolve(strict=False)):
|
||||
raise ValueError("input job path escapes project root")
|
||||
if not _is_beneath(job_workspace, (root / "workspace").resolve(strict=False)):
|
||||
raise ValueError("workspace job path escapes project root")
|
||||
root, input_job, job_workspace = _resolve_job_layout(root, job_id)
|
||||
final_dir = job_workspace / "final"
|
||||
|
||||
result: dict[str, Any] = {
|
||||
|
||||
@@ -251,7 +251,13 @@ class ProjectContextIndexer:
|
||||
self.workspace_root = self.project_root / "workspace"
|
||||
self.allow_legacy_manifest_migration = allow_legacy_manifest_migration
|
||||
|
||||
def build(self, job_id: str, prefix: str = "") -> ProjectContextIndex:
|
||||
def build(
|
||||
self,
|
||||
job_id: str,
|
||||
prefix: str = "",
|
||||
*,
|
||||
include_stage_status: bool = True,
|
||||
) -> ProjectContextIndex:
|
||||
workspace = self.workspace_root / job_id
|
||||
canonical_input_dir = self.project_root / "input" / job_id
|
||||
input_dir = canonical_input_dir if canonical_input_dir.exists() else workspace / "input"
|
||||
@@ -275,7 +281,8 @@ class ProjectContextIndexer:
|
||||
self._scan_files(index, final_dir, "final")
|
||||
self._scan_stage_dirs(index, workspace)
|
||||
self._load_manifests(index, final_dir)
|
||||
self._load_stage_status(index, job_id, prefix or self._best_prefix(index))
|
||||
if include_stage_status:
|
||||
self._load_stage_status(index, job_id, prefix or self._best_prefix(index))
|
||||
self._scan_logs(index, logs_dir)
|
||||
self._derive_warnings(index)
|
||||
return index
|
||||
@@ -696,11 +703,16 @@ def build_project_context(
|
||||
project_root: Path | str = PROJECT_ROOT,
|
||||
*,
|
||||
allow_legacy_manifest_migration: bool = True,
|
||||
include_stage_status: bool = True,
|
||||
) -> ProjectContextIndex:
|
||||
return ProjectContextIndexer(
|
||||
project_root,
|
||||
allow_legacy_manifest_migration=allow_legacy_manifest_migration,
|
||||
).build(job_id=job_id, prefix=prefix)
|
||||
).build(
|
||||
job_id=job_id,
|
||||
prefix=prefix,
|
||||
include_stage_status=include_stage_status,
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
|
||||
@@ -20,6 +20,7 @@ ANALYSIS_KEYWORDS = {
|
||||
"rnaseq_kegg_enrich": ["kegg富集", "kegg分析", "kegg", "pathway"],
|
||||
"rnaseq_fgsea": ["fgsea", "gsea", "排序富集", "基因集富集"],
|
||||
"rnaseq_candidate_family": ["候选家族", "候选基因家族", "家族排名", "tf注释", "tf family", "tffamily", "pfam注释", "deg注释", "差异基因注释"],
|
||||
"rnaseq_report": ["rnaseq结果报告", "rna-seq结果报告", "转录组结果报告", "rnaseq报告", "rna-seq报告"],
|
||||
"rnaseq_enrich": ["富集分析", "功能富集"],
|
||||
"identify": ["鉴定", "identify", "识别", "找出", "鉴定基因家族"],
|
||||
"identify_refine": ["修订成员", "修订鉴定成员", "修改成员", "删除基因", "删除", "去除基因", "去掉基因", "移除基因", "补充基因", "遗漏基因", "重建蛋白", "人工修订id", "curated members", "refine identify"],
|
||||
@@ -61,6 +62,7 @@ RNASEQ_STAGES = {
|
||||
"rnaseq_kegg_enrich",
|
||||
"rnaseq_fgsea",
|
||||
"rnaseq_candidate_family",
|
||||
"rnaseq_report",
|
||||
"rnaseq_enrich",
|
||||
}
|
||||
|
||||
@@ -179,6 +181,21 @@ def fast_intent_recognition(query: str) -> Optional[Dict[str, Any]]:
|
||||
except Exception:
|
||||
pass
|
||||
has_rnaseq_marker = any(marker in query_lower for marker in RNASEQ_MARKERS)
|
||||
if has_rnaseq_marker and any(
|
||||
keyword in query_lower
|
||||
for keyword in ["生成结果报告", "打开结果报告", "结果报告", "生成报告", "打开报告"]
|
||||
):
|
||||
action = "open" if "打开" in query_lower else "generate"
|
||||
return {
|
||||
"stage": "rnaseq_report",
|
||||
"stages": ["rnaseq_report"],
|
||||
"gene_family": "",
|
||||
"species": "",
|
||||
"prefix": "",
|
||||
"params": {"rnaseqReportAction": action},
|
||||
"confidence": 0.99,
|
||||
"method": "rnaseq_report_deterministic",
|
||||
}
|
||||
header_clean_context = any(kw in query_lower for kw in [
|
||||
"id处理", "处理id", "修订id", "简化id", "清理id", "id简化",
|
||||
"去掉注释", "去除注释", "注释信息", "长注释",
|
||||
|
||||
@@ -29,6 +29,74 @@ WORKSPACE_DIR = WORKSPACE_ROOT
|
||||
|
||||
|
||||
class Orchestrator:
|
||||
# 双语消息映射表:中文 → English(用于用户可见消息)
|
||||
_MSG = {
|
||||
# analysis_node — response 消息
|
||||
"✅ 所有分析阶段已完成!": "✅ All analysis stages completed!",
|
||||
"📋 自动规划剩余分析阶段": "📋 Auto-planned remaining analysis stages",
|
||||
"**已完成**": "**Completed**",
|
||||
"无": "None",
|
||||
"**待执行**": "**Pending**",
|
||||
"个阶段)": "stage(s))",
|
||||
"请确认是否按顺序执行以上分析阶段?": "Confirm to execute the above analysis stages in order?",
|
||||
"我将帮您执行RNA-seq": "I will help you execute RNA-seq",
|
||||
"分析。请确认参数后开始。": "analysis. Please confirm parameters to start.",
|
||||
"RNA-seq分析将通过前端异步执行。": "RNA-seq analysis will be executed asynchronously via frontend.",
|
||||
" 分析。PlanningAgentV2 已根据当前状态生成完整执行计划。": " analysis. PlanningAgentV2 has generated a complete execution plan based on current state.",
|
||||
"我将帮您执行": "I will help you execute",
|
||||
"分析。": "analysis.",
|
||||
"请提供基因家族蛋白序列文件(.fa/.fasta),用于MEME保守基序预测。": "Please provide gene family protein sequence file (.fa/.fasta) for MEME conserved motif prediction.",
|
||||
"请提供config.xlsx及基因组/GFF/CDS/PEP等原始文件,用于完整数据准备。": "Please provide config.xlsx and genome/GFF/CDS/PEP files for complete data preparation.",
|
||||
"请提供基因家族蛋白序列文件,用于Pfam/InterPro结构域注释。": "Please provide gene family protein sequence file for Pfam/InterPro domain annotation.",
|
||||
"请提供基因家族的GFF注释文件和基因组序列。": "Please provide gene family GFF annotation file and genome sequence.",
|
||||
"请提供基因家族成员的启动子序列(上游2000bp)或全基因组序列+GFF。": "Please provide gene family promoter sequences (upstream 2000bp) or whole genome + GFF.",
|
||||
"请提供物种基因组序列(.fasta)和GFF注释文件。": "Please provide species genome sequence (.fasta) and GFF annotation file.",
|
||||
"请提供多个物种的基因组序列和GFF注释文件,以及物种间的同源关系。": "Please provide genome sequences and GFF annotation files for multiple species, along with inter-species orthology.",
|
||||
"请提供染色体长度信息和基因位置信息。": "Please provide chromosome length and gene position information.",
|
||||
"请提供拟南芥同源基因映射文件(ortholog_pairs.txt,格式:拟南芥ID\t苦荞ID),用于STRING数据库互作预测。如未提供,请先运行同源基因识别分析。": "Please provide Arabidopsis ortholog mapping file (ortholog_pairs.txt, format: AtID\tTargetID) for STRING interaction prediction. If unavailable, run ortholog identification first.",
|
||||
"请提供目标物种和参考物种(如拟南芥At)的蛋白序列文件,用于同源基因识别。": "Please provide target and reference species (e.g., Arabidopsis At) protein FASTA for ortholog identification.",
|
||||
"请提供物种间同源基因对(ortholog pairs,格式:基因A\t基因B)或共线性分析结果。": "Please provide inter-species ortholog pairs (format: GeneA\tGeneB) or synteny analysis results.",
|
||||
"请提供RNA-seq表达矩阵(FPKM/TPM/Count)或原始测序数据。": "Please provide RNA-seq expression matrix (FPKM/TPM/Count) or raw sequencing data.",
|
||||
"请提供差异表达基因列表(DEGs)或目标基因集。": "Please provide differentially expressed gene list (DEGs) or target gene set.",
|
||||
"请提供基因家族蛋白序列文件,用于理化性质分析。": "Please provide gene family protein sequence file for physicochemical property analysis.",
|
||||
"请提供基因家族HMM文件和蛋白序列文件,将自动执行标准11阶段分析。": "Please provide gene family HMM file and protein FASTA; standard 11-stage analysis will be executed automatically.",
|
||||
"请提供RNA-seq原始测序数据(FASTQ格式)。": "Please provide RNA-seq raw sequencing data (FASTQ format).",
|
||||
"请提供质控后的clean reads(FASTQ格式)和参考基因组。": "Please provide QC-filtered clean reads (FASTQ format) and reference genome.",
|
||||
"请提供比对后的BAM文件或参考基因组注释GFF。": "Please provide aligned BAM file or reference genome annotation GFF.",
|
||||
"请提供featureCounts计数矩阵和样本分组表,用于VST、PCA、样本相关性和聚类。": "Please provide featureCounts count matrix and sample grouping table for VST, PCA, sample correlation and clustering.",
|
||||
"请提供多时相表达矩阵(如不同发育阶段/处理时间点)。": "Please provide multi-temporal expression matrix (e.g., different developmental stages/treatment time points).",
|
||||
"请提供VST/log2表达矩阵或count矩阵和样本性状表,用于共表达网络构建。": "Please provide VST/log2 expression matrix or count matrix and sample trait table for co-expression network construction.",
|
||||
"请提供差异表达结果和gene2go背景文件,或先执行背景注释。": "Please provide differential expression results and gene2go background file, or run background annotation first.",
|
||||
"请提供差异表达结果和gene2ko背景文件,或先执行背景注释。": "Please provide differential expression results and gene2ko background file, or run background annotation first.",
|
||||
"请提供差异表达排序统计量和gene2ko/KEGG背景,用于fgsea排序富集。": "Please provide differential expression ranking statistics and gene2ko/KEGG background for fgsea ranked enrichment.",
|
||||
"请提供差异表达结果和目标物种蛋白FASTA;若已有InterProScan TSV会优先复用。": "Please provide differential expression results and target species protein FASTA; existing InterProScan TSV will be reused if available.",
|
||||
"请提供表达矩阵(样本数≥6),用于共表达网络构建。": "Please provide expression matrix (samples ≥ 6) for co-expression network construction.",
|
||||
# knowledge_node — system prompt / response 消息
|
||||
"你是BotanicalBaseAgent基因家族分析系统的知识助手。基于提供的文献资料,生成专业、详细的回答。": "You are a knowledge assistant for the BotanicalBaseAgent gene family analysis system. Generate professional, detailed answers based on provided literature.",
|
||||
"📚 基于知识库的回答:": "📚 Knowledge base answer:",
|
||||
"**参考文献**:": "**References**:",
|
||||
"(来源: 本地LLM回答 + RAG辅助检索)": "(Source: local LLM answer + RAG-assisted retrieval)",
|
||||
"⚠️ (本地LLM回答,未使用RAG文献检索)": "⚠️ (local LLM answer, RAG retrieval not used)",
|
||||
"找到": "Found",
|
||||
"条相关文献,正在生成本地LLM回答...": " related references; generating local LLM answer...",
|
||||
"未找到相关文献,生成一般回答": "No related references found; generating general answer",
|
||||
"你是一个专业的基因家族分析意图识别器。": "You are a professional gene family analysis intent recognizer.",
|
||||
"你是一个基因家族分析专家。用户想要进行基因家族分析。": "You are a gene family analysis expert. The user wants to perform gene family analysis.",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def _l(cls, zh_text: str, language: str = "zh") -> str:
|
||||
"""Return language-appropriate text from the bilingual map."""
|
||||
if language == "zh":
|
||||
return zh_text
|
||||
en = cls._MSG.get(zh_text)
|
||||
if en is not None:
|
||||
return en
|
||||
# fallback: try partial match for template strings
|
||||
for zh_key, en_val in cls._MSG.items():
|
||||
if zh_key in zh_text:
|
||||
return zh_text.replace(zh_key, en_val)
|
||||
return zh_text
|
||||
def __init__(
|
||||
self,
|
||||
llm_manager: LLMManager,
|
||||
@@ -313,6 +381,7 @@ class Orchestrator:
|
||||
|
||||
async def _router_node(self, state: GraphState) -> dict:
|
||||
query = state["query"]
|
||||
language = state.get("language", "zh")
|
||||
|
||||
# 使用 RouterAgent 进行意图路由
|
||||
print(f"[RouterNode] Using RouterAgent to classify query: {query[:50]}...")
|
||||
@@ -378,6 +447,7 @@ class Orchestrator:
|
||||
|
||||
async def _knowledge_node(self, state: GraphState) -> dict:
|
||||
query = state["query"]
|
||||
language = state.get("language", "zh")
|
||||
|
||||
try:
|
||||
# 优先调用当前 Dual-KB QA 服务
|
||||
@@ -578,7 +648,7 @@ class Orchestrator:
|
||||
"status": "completed",
|
||||
"mode": "analysis",
|
||||
"final_output": {
|
||||
"response": f"✅ 所有分析阶段已完成!\n\n已完成: {', '.join(completed_stages)}",
|
||||
"response": self._l("✅ 所有分析阶段已完成!", language) + "\n\n" + self._l("**已完成**", language) + ": " + ", ".join(completed_stages),
|
||||
"analysis_plan": {
|
||||
"gene_family": gene_family,
|
||||
"species": species,
|
||||
@@ -595,14 +665,7 @@ class Orchestrator:
|
||||
"status": "completed",
|
||||
"mode": "analysis",
|
||||
"final_output": {
|
||||
"response": f"""📋 自动规划剩余分析阶段
|
||||
|
||||
**已完成**: {', '.join(completed_stages) if completed_stages else '无'}
|
||||
|
||||
**待执行** ({len(remaining_stages)}个阶段):
|
||||
{stages_text}
|
||||
|
||||
请确认是否按顺序执行以上分析阶段?""",
|
||||
"response": f"{self._l('📋 自动规划剩余分析阶段', language)}\n\n{self._l('**已完成**', language)}: {', '.join(completed_stages) if completed_stages else self._l('无', language)}\n\n{self._l('**待执行**', language)} ({len(remaining_stages)}{self._l('个阶段)', language)}:\n{stages_text}\n\n{self._l('请确认是否按顺序执行以上分析阶段?', language)}",
|
||||
"analysis_plan": {
|
||||
"gene_family": gene_family,
|
||||
"species": species,
|
||||
@@ -630,6 +693,7 @@ class Orchestrator:
|
||||
async def _analysis_node(self, state: GraphState) -> dict:
|
||||
query = state["query"]
|
||||
files = state.get("files", [])
|
||||
language = state.get("language", "zh")
|
||||
print(f"[_analysis_node] Entered with query: {query[:50]}...", flush=True)
|
||||
|
||||
try:
|
||||
@@ -821,7 +885,7 @@ class Orchestrator:
|
||||
"status": "completed",
|
||||
"mode": "analysis",
|
||||
"final_output": {
|
||||
"response": f"我将帮您执行RNA-seq {analysis_type}分析。请确认参数后开始。",
|
||||
"response": self._l("我将帮您执行RNA-seq", language) + " " + analysis_type + self._l("分析。请确认参数后开始。", language),
|
||||
"analysis_plan": {
|
||||
"gene_family": gene_family,
|
||||
"species": species,
|
||||
@@ -831,7 +895,7 @@ class Orchestrator:
|
||||
"prefix": f"{species}.{gene_family}",
|
||||
"cpu": 8
|
||||
},
|
||||
"guidance": "RNA-seq分析将通过前端异步执行。"
|
||||
"guidance": self._l("RNA-seq分析将通过前端异步执行。", language)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -878,7 +942,7 @@ class Orchestrator:
|
||||
"status": "completed",
|
||||
"mode": "analysis",
|
||||
"final_output": {
|
||||
"response": f"我将帮您执行RNA-seq {analysis_type} 分析。PlanningAgentV2 已根据当前状态生成完整执行计划。",
|
||||
"response": self._l("我将帮您执行RNA-seq", language) + " " + analysis_type + self._l(" 分析。PlanningAgentV2 已根据当前状态生成完整执行计划。", language),
|
||||
"analysis_plan": plan_result.get("final_output", {}).get("analysis_plan", {})
|
||||
}
|
||||
}
|
||||
@@ -889,7 +953,7 @@ class Orchestrator:
|
||||
"status": "completed",
|
||||
"mode": "analysis",
|
||||
"final_output": {
|
||||
"response": f"我将帮您执行RNA-seq {analysis_type} 分析。请确认参数后开始。",
|
||||
"response": self._l("我将帮您执行RNA-seq", language) + " " + analysis_type + self._l(" 分析。请确认参数后开始。", language),
|
||||
"analysis_plan": {
|
||||
"gene_family": gene_family,
|
||||
"species": species,
|
||||
@@ -900,21 +964,25 @@ class Orchestrator:
|
||||
"prefix": f"{species}.{gene_family}",
|
||||
"cpu": 8
|
||||
},
|
||||
"guidance": f"RNA-seq {analysis_type} 分析将通过前端异步执行。"
|
||||
"guidance": self._l("RNA-seq " + analysis_type + " 分析将通过前端异步执行。", language)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Ka/Ks分析:走通用规划流程(返回analysis_plan让前端执行)
|
||||
# 注意:不再直接调用_execute_kaks,而是通过PlanningAgentV2规划完整流程
|
||||
guidance = guidance_map.get(analysis_type, f"请上传进行{analysis_type}分析所需的文件,或确认使用系统内置数据。")
|
||||
|
||||
guidance = guidance_map.get(analysis_type)
|
||||
if guidance:
|
||||
guidance = self._l(guidance, language)
|
||||
else:
|
||||
guidance = self._l("请上传进行", language) + analysis_type + self._l("分析所需的文件,或确认使用系统内置数据。", language)
|
||||
|
||||
return {
|
||||
"current_agent": "analysis",
|
||||
"status": "completed",
|
||||
"mode": "analysis",
|
||||
"final_output": {
|
||||
"response": f"我将帮您执行{analysis_type}分析。",
|
||||
"response": self._l("我将帮您执行", language) + analysis_type + self._l("分析。", language),
|
||||
"analysis_plan": {
|
||||
"gene_family": gene_family,
|
||||
"species": species,
|
||||
|
||||
+45
-12
@@ -22,6 +22,9 @@ DUAL_KB_QA_BASE_URL = (
|
||||
DUAL_KB_QA_ROUTE = os.environ.get("DUAL_KB_QA_ROUTE", "kb-qa").strip().strip("/") or "kb-qa"
|
||||
DUAL_KB_GRAPH_QA_ROUTE = os.environ.get("DUAL_KB_GRAPH_QA_ROUTE", "v15-graph-qa").strip().strip("/") or "v15-graph-qa"
|
||||
DUAL_KB_QA_VARIANT = os.environ.get("DUAL_KB_QA_VARIANT", "full").strip() or "full"
|
||||
QA_CONNECT_TIMEOUT_SECONDS = 10.0
|
||||
QA_V1_1_73_DEFAULT_TIMEOUT_SECONDS = 900.0
|
||||
QA_V1_1_73_DEFAULT_STREAM_IDLE_TIMEOUT_SECONDS = 45.0
|
||||
|
||||
|
||||
def _env_bool(name: str, default: bool) -> bool:
|
||||
@@ -31,6 +34,35 @@ def _env_bool(name: str, default: bool) -> bool:
|
||||
return value.strip().lower() not in {"0", "false", "no", "off"}
|
||||
|
||||
|
||||
def _positive_timeout(value: object, default: float) -> float:
|
||||
try:
|
||||
parsed = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
return parsed if parsed > 0 else default
|
||||
|
||||
|
||||
def _non_stream_timeout(timeout: float) -> tuple[float, float]:
|
||||
read_timeout = _positive_timeout(
|
||||
os.environ.get("QA_V1_1_73_TIMEOUT", timeout),
|
||||
timeout,
|
||||
)
|
||||
return QA_CONNECT_TIMEOUT_SECONDS, read_timeout
|
||||
|
||||
|
||||
def _stream_timeout(timeout: float | None = None) -> tuple[float, float]:
|
||||
default = (
|
||||
_positive_timeout(timeout, QA_V1_1_73_DEFAULT_STREAM_IDLE_TIMEOUT_SECONDS)
|
||||
if timeout is not None
|
||||
else QA_V1_1_73_DEFAULT_STREAM_IDLE_TIMEOUT_SECONDS
|
||||
)
|
||||
idle_timeout = _positive_timeout(
|
||||
os.environ.get("QA_V1_1_73_STREAM_IDLE_TIMEOUT", default),
|
||||
default,
|
||||
)
|
||||
return QA_CONNECT_TIMEOUT_SECONDS, idle_timeout
|
||||
|
||||
|
||||
def dual_kb_endpoint(path: str = "chat") -> str:
|
||||
return f"{DUAL_KB_QA_BASE_URL}/api/v1/{DUAL_KB_QA_ROUTE}/{path.strip('/')}"
|
||||
|
||||
@@ -221,6 +253,7 @@ def _build_dual_kb_payload(
|
||||
if route == "qa-v1-1-73":
|
||||
payload: dict[str, Any] = {
|
||||
"question": question,
|
||||
"response_detail": "answer",
|
||||
"debug": False,
|
||||
"use_llm": resolved_use_llm,
|
||||
"chain_top_k": 3,
|
||||
@@ -315,7 +348,7 @@ def query_dual_kb_qa(
|
||||
graph_hops: int = 2,
|
||||
graph_limit_nodes: int = 60,
|
||||
graph_limit_edges: int = 100,
|
||||
timeout: float = 240.0,
|
||||
timeout: float = QA_V1_1_73_DEFAULT_TIMEOUT_SECONDS,
|
||||
) -> dict[str, Any]:
|
||||
question = str(question or "").strip()
|
||||
if not question:
|
||||
@@ -338,15 +371,15 @@ def query_dual_kb_qa(
|
||||
effective_use_llm = bool(payload.get("use_llm"))
|
||||
request_path = "chat" if effective_use_llm else "search"
|
||||
|
||||
request_timeout = (
|
||||
max(timeout, float(os.environ.get("QA_V1_1_73_TIMEOUT", "300")))
|
||||
transport_timeout = (
|
||||
_non_stream_timeout(timeout)
|
||||
if route == "qa-v1-1-73"
|
||||
else timeout
|
||||
else float(timeout)
|
||||
)
|
||||
response = requests.post(
|
||||
dual_kb_endpoint(request_path),
|
||||
json=payload,
|
||||
timeout=request_timeout,
|
||||
timeout=transport_timeout,
|
||||
)
|
||||
data = response.json() if response.content else {}
|
||||
initial_detail = data.get("detail") if isinstance(data, dict) else None
|
||||
@@ -359,7 +392,7 @@ def query_dual_kb_qa(
|
||||
response = requests.post(
|
||||
dual_kb_endpoint(request_path),
|
||||
json=payload,
|
||||
timeout=request_timeout,
|
||||
timeout=transport_timeout,
|
||||
)
|
||||
data = response.json() if response.content else {}
|
||||
if response.status_code >= 400:
|
||||
@@ -401,7 +434,7 @@ def stream_dual_kb_qa(
|
||||
graph_hops: int = 2,
|
||||
graph_limit_nodes: int = 60,
|
||||
graph_limit_edges: int = 100,
|
||||
timeout: float = 300.0,
|
||||
timeout: float | None = None,
|
||||
) -> Iterator[str]:
|
||||
"""Proxy truthful QA milestones and normalize the final V1.1.73 payload."""
|
||||
question = str(question or "").strip()
|
||||
@@ -426,17 +459,17 @@ def stream_dual_kb_qa(
|
||||
graph_limit_nodes=graph_limit_nodes,
|
||||
graph_limit_edges=graph_limit_edges,
|
||||
)
|
||||
request_timeout = (
|
||||
max(timeout, float(os.environ.get("QA_V1_1_73_TIMEOUT", "300")))
|
||||
transport_timeout = (
|
||||
_stream_timeout(timeout)
|
||||
if route == "qa-v1-1-73"
|
||||
else timeout
|
||||
else _positive_timeout(timeout, QA_V1_1_73_DEFAULT_STREAM_IDLE_TIMEOUT_SECONDS)
|
||||
)
|
||||
try:
|
||||
response = requests.post(
|
||||
dual_kb_endpoint("chat/stream"),
|
||||
json=payload,
|
||||
stream=True,
|
||||
timeout=(10.0, request_timeout),
|
||||
timeout=transport_timeout,
|
||||
)
|
||||
except requests.Timeout as exc:
|
||||
yield _sse_frame("error", {
|
||||
@@ -475,7 +508,7 @@ def stream_dual_kb_qa(
|
||||
|
||||
event_name = "message"
|
||||
data_lines: list[str] = []
|
||||
for raw_line in response.iter_lines(decode_unicode=True):
|
||||
for raw_line in response.iter_lines(chunk_size=1, decode_unicode=True):
|
||||
line = raw_line.decode("utf-8") if isinstance(raw_line, bytes) else str(raw_line)
|
||||
if line.startswith(":"):
|
||||
yield ": keepalive\n\n"
|
||||
|
||||
@@ -58,10 +58,14 @@ class DeepSeekProvider(BaseLLMProvider):
|
||||
tools = kwargs.get("tools")
|
||||
if tools is not None:
|
||||
request_kwargs["tools"] = tools
|
||||
if tools is not None or kwargs.get("disable_thinking", False):
|
||||
request_kwargs["extra_body"] = {"thinking": {"type": "disabled"}}
|
||||
tool_choice = kwargs.get("tool_choice")
|
||||
if tool_choice is not None:
|
||||
request_kwargs["tool_choice"] = tool_choice
|
||||
response_format = kwargs.get("response_format")
|
||||
if response_format is not None:
|
||||
request_kwargs["response_format"] = response_format
|
||||
|
||||
response = self.client.chat.completions.create(**request_kwargs)
|
||||
message = response.choices[0].message
|
||||
|
||||
@@ -9,7 +9,7 @@ from typing import Any
|
||||
from uuid import NAMESPACE_URL, UUID, uuid4, uuid5
|
||||
|
||||
from gfa_core.mas.agent_state_repository import SQLiteAgentStateRepository
|
||||
from gfa_core.mas.workflow_models import GoalLedger
|
||||
from gfa_core.mas.workflow_models import CurrentRecommendation, GoalLedger
|
||||
|
||||
|
||||
_INTERACTION_KINDS = {
|
||||
@@ -68,7 +68,7 @@ def _history(items: Any) -> tuple[dict[str, str], ...]:
|
||||
content = str(item.get("content") or "")
|
||||
if content:
|
||||
normalized.append({"role": item["role"], "content": content})
|
||||
return tuple(normalized[-10:])
|
||||
return tuple(normalized)
|
||||
|
||||
|
||||
class PersistentConversationStore:
|
||||
@@ -207,7 +207,7 @@ class PersistentConversationStore:
|
||||
),
|
||||
expected_revision=ledger.revision,
|
||||
)
|
||||
stored_messages = self.repository.list_recent_messages(conversation_id, limit=10)
|
||||
stored_messages = self.repository.list_messages(conversation_id)
|
||||
if not stored_messages:
|
||||
for item in _history(client_history):
|
||||
persisted = self.repository.append_message(
|
||||
@@ -220,10 +220,7 @@ class PersistentConversationStore:
|
||||
conversation = self.repository.get_conversation(conversation_id)
|
||||
assert conversation is not None
|
||||
assert conversation.revision == persisted.revision
|
||||
stored_messages = self.repository.list_recent_messages(
|
||||
conversation_id,
|
||||
limit=10,
|
||||
)
|
||||
stored_messages = self.repository.list_messages(conversation_id)
|
||||
current_user_already_saved = bool(
|
||||
stored_messages
|
||||
and stored_messages[-1].role == "user"
|
||||
@@ -264,6 +261,7 @@ class PersistentConversationStore:
|
||||
action: str | None,
|
||||
intent: dict[str, Any] | None = None,
|
||||
proposal: dict[str, str] | None = None,
|
||||
recommendation: Mapping[str, Any] | None = None,
|
||||
) -> GoalLedger:
|
||||
if interaction_kind not in _INTERACTION_KINDS:
|
||||
raise ValueError("interaction_kind is invalid")
|
||||
@@ -362,12 +360,31 @@ class PersistentConversationStore:
|
||||
user_terms = dict(serialized["user_terms"])
|
||||
if turn.job_id:
|
||||
user_terms["active_job_id"] = turn.job_id
|
||||
if action == "show_result":
|
||||
if recommendation is None:
|
||||
user_terms.pop("current_recommendation", None)
|
||||
else:
|
||||
current_recommendation = CurrentRecommendation.model_validate(
|
||||
recommendation
|
||||
)
|
||||
if current_recommendation.job_id != turn.job_id:
|
||||
raise ValueError("recommendation job binding conflict")
|
||||
user_terms["current_recommendation"] = (
|
||||
current_recommendation.model_dump(mode="json", exclude_none=True)
|
||||
)
|
||||
if interaction_kind in {"cancel", "correction", "switch"}:
|
||||
user_terms.pop("current_recommendation", None)
|
||||
if action == "clarify" and isinstance(proposal, dict):
|
||||
normalized_proposal = {
|
||||
key: str(proposal.get(key) or "").strip()
|
||||
for key in ("proposal_id", "request", "subject")
|
||||
}
|
||||
if all(normalized_proposal.values()):
|
||||
original_request = str(
|
||||
proposal.get("original_request") or ""
|
||||
).strip()
|
||||
if original_request:
|
||||
normalized_proposal["original_request"] = original_request
|
||||
workflow_id = str(proposal.get("workflow_id") or "").strip()
|
||||
if workflow_id in {
|
||||
"data_preparation",
|
||||
@@ -376,6 +393,16 @@ class PersistentConversationStore:
|
||||
"rnaseq_family_member",
|
||||
}:
|
||||
normalized_proposal["workflow_id"] = workflow_id
|
||||
action_binding = {
|
||||
key: str(proposal.get(key) or "").strip()
|
||||
for key in (
|
||||
"workflow_action_id",
|
||||
"authorized_stages",
|
||||
"target_stage",
|
||||
)
|
||||
}
|
||||
if all(action_binding.values()):
|
||||
normalized_proposal.update(action_binding)
|
||||
normalized_proposal["job_id"] = turn.job_id
|
||||
user_terms["pending_proposal"] = normalized_proposal
|
||||
if (
|
||||
|
||||
@@ -793,6 +793,26 @@ CAPABILITIES: Dict[str, StageCapability] = {
|
||||
repair_candidates=["gff_id_overlap_report", "motif_database_fallback_to_jaspar", "artifact_reresolve"],
|
||||
rerun_policy="manual",
|
||||
),
|
||||
"rnaseq_report": StageCapability(
|
||||
stage="rnaseq_report",
|
||||
cli="apps/cli/run_rnaseq_report.py",
|
||||
env="test4py311",
|
||||
category="rnaseq",
|
||||
consumes=[
|
||||
ArtifactNeed("successful_rnaseq_manifests", "manifest", alternatives=["final_manifest_scan"]),
|
||||
],
|
||||
provides=[
|
||||
ArtifactProduct("rnaseq_report", "markdown", "{prefix}.rnaseq_report.md"),
|
||||
ArtifactProduct("rnaseq_report_evidence", "json", "{prefix}.rnaseq_report.evidence.json"),
|
||||
ArtifactProduct("rnaseq_report_model", "json", "{prefix}.rnaseq_report.model.json"),
|
||||
ArtifactProduct("rnaseq_report_manifest", "manifest", "{prefix}.rnaseq_report.manifest.json"),
|
||||
],
|
||||
scope_modes=["rnaseq_scope_id"],
|
||||
preflight_checks=["at_least_one_successful_rnaseq_manifest", "prefix_scope_identity_consistent"],
|
||||
postflight_checks=["deterministic_markdown_exists", "evidence_snapshot_exists", "coverage_recorded"],
|
||||
repair_candidates=["manifest_recovery", "regenerate_from_current_snapshot"],
|
||||
rerun_policy="manual_or_after_selected_batch_terminal",
|
||||
),
|
||||
"summary": StageCapability(
|
||||
stage="summary",
|
||||
cli="apps/cli/run_summary.py",
|
||||
@@ -804,11 +824,14 @@ CAPABILITIES: Dict[str, StageCapability] = {
|
||||
],
|
||||
provides=[
|
||||
ArtifactProduct("summary_report", "markdown", "{prefix}.summary_report.md"),
|
||||
ArtifactProduct("summary_evidence_index", "markdown", "{prefix}.summary_evidence_index.md"),
|
||||
ArtifactProduct("summary_evidence", "json", "{prefix}.summary.evidence.json"),
|
||||
ArtifactProduct("deepseek_metadata", "json", "{prefix}.summary.deepseek.json"),
|
||||
ArtifactProduct("summary_manifest", "manifest", "{prefix}.summary.manifest.json"),
|
||||
],
|
||||
scope_modes=["target_only", "target_plus_reference", "all_selected", "rnaseq"],
|
||||
preflight_checks=["at_least_one_success_manifest"],
|
||||
postflight_checks=["summary_contains_stage_evidence"],
|
||||
postflight_checks=["summary_contains_stage_evidence", "deepseek_report_generation_proven"],
|
||||
repair_candidates=["manifest_recovery", "artifact_reresolve"],
|
||||
rerun_policy="auto_after_manifest_recovery",
|
||||
),
|
||||
|
||||
@@ -90,6 +90,7 @@ PIPELINE_STAGES = {
|
||||
"rnaseq_go_enrich",
|
||||
"rnaseq_kegg_enrich",
|
||||
"rnaseq_candidate_family",
|
||||
"rnaseq_report",
|
||||
}
|
||||
|
||||
STAGE_DEPENDENCIES = {
|
||||
@@ -160,7 +161,7 @@ STAGE_DEPENDENCIES = {
|
||||
"interspecies_synteny": ["identify"],
|
||||
"karyotype": ["identify", "synteny"],
|
||||
"ppi": ["identify"],
|
||||
"kaks": ["identify", "ortholog"],
|
||||
"kaks": ["identify"],
|
||||
"combo": ["identify", "phylogeny", "motif", "structure", "domain"],
|
||||
"rnaseq_align": ["rnaseq_qc"],
|
||||
"rnaseq_quant": ["rnaseq_align"],
|
||||
@@ -173,6 +174,9 @@ STAGE_DEPENDENCIES = {
|
||||
"rnaseq_go_enrich": ["rnaseq_de"],
|
||||
"rnaseq_kegg_enrich": ["rnaseq_de"],
|
||||
"rnaseq_candidate_family": ["rnaseq_de", "rnaseq_go_enrich", "rnaseq_kegg_enrich", "rnaseq_fgsea"],
|
||||
# A report is a snapshot of whatever RNA-seq evidence is already successful.
|
||||
# Static dependencies would incorrectly prevent intentional mid-pipeline reports.
|
||||
"rnaseq_report": [],
|
||||
}
|
||||
|
||||
PARALLEL_GROUPS = [
|
||||
@@ -271,6 +275,7 @@ STAGE_RESOURCE_PROFILES = {
|
||||
"rnaseq_kegg_enrich": {"queue": "rnaseq", "resource_class": "rnaseq", "default_cpu": 4, "max_cpu": 4},
|
||||
"rnaseq_fgsea": {"queue": "rnaseq", "resource_class": "rnaseq", "default_cpu": 4, "max_cpu": 4},
|
||||
"rnaseq_candidate_family": {"queue": "rnaseq", "resource_class": "rnaseq", "default_cpu": 4, "max_cpu": 4},
|
||||
"rnaseq_report": {"queue": "light", "resource_class": "light", "default_cpu": 4, "max_cpu": 4},
|
||||
}
|
||||
|
||||
DEFAULT_STAGE_RESOURCE_PROFILE = {
|
||||
@@ -391,6 +396,7 @@ STAGE_ORDER = [
|
||||
"rnaseq_kegg_enrich",
|
||||
"rnaseq_fgsea",
|
||||
"rnaseq_candidate_family",
|
||||
"rnaseq_report",
|
||||
"rnaseq_enrich",
|
||||
"gene_family_report",
|
||||
]
|
||||
@@ -469,6 +475,7 @@ STAGE_INFO = {
|
||||
"rnaseq_enrich": {"name": "RNA-seq富集分析", "estimated_time": 15, "cpu": 4},
|
||||
"rnaseq_fgsea": {"name": "RNA-seq fgsea分析", "estimated_time": 15, "cpu": 4},
|
||||
"rnaseq_candidate_family": {"name": "RNA-seq候选家族筛选", "estimated_time": 20, "cpu": 4},
|
||||
"rnaseq_report": {"name": "RNA-seq结果报告", "estimated_time": 2, "cpu": 4},
|
||||
}
|
||||
|
||||
EXECUTABLE_STAGES = {
|
||||
@@ -544,6 +551,7 @@ EXECUTABLE_STAGES = {
|
||||
"rnaseq_enrich",
|
||||
"rnaseq_fgsea",
|
||||
"rnaseq_candidate_family",
|
||||
"rnaseq_report",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -14,6 +14,7 @@ flag enables decision-making.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from copy import copy
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
@@ -93,6 +94,7 @@ def resolve_stage_requirements(
|
||||
params: dict[str, Any] | None = None,
|
||||
project_root: Path | str = PROJECT_ROOT,
|
||||
shadow_only: bool = True,
|
||||
project_context: ProjectContextIndex | None = None,
|
||||
) -> StageRequirementReport:
|
||||
"""Resolve stage requirements without changing planning or files."""
|
||||
params = dict(params or {})
|
||||
@@ -110,7 +112,16 @@ def resolve_stage_requirements(
|
||||
report.messages.append("Cannot resolve requirements without job_id.")
|
||||
return report
|
||||
|
||||
context = build_project_context(job_id=job_id, prefix=identity.prefix, project_root=project_root)
|
||||
context = project_context
|
||||
if context is None:
|
||||
context = build_project_context(
|
||||
job_id=job_id,
|
||||
prefix=identity.prefix,
|
||||
project_root=project_root,
|
||||
include_stage_status=False,
|
||||
)
|
||||
else:
|
||||
context = copy(context)
|
||||
_apply_identity_to_context(context, identity)
|
||||
|
||||
if identity.stage == "identify":
|
||||
@@ -151,18 +162,28 @@ def resolve_many_stage_requirements(
|
||||
reports: dict[str, dict[str, Any]] = {}
|
||||
params_by_stage = params_by_stage or {}
|
||||
shared_params = shared_params or {}
|
||||
contexts_by_prefix: dict[str, ProjectContextIndex] = {}
|
||||
for stage in stages:
|
||||
stage_params = {**shared_params, **(params_by_stage.get(stage) or {})}
|
||||
if stage not in CORE_REQUIREMENT_STAGES:
|
||||
continue
|
||||
try:
|
||||
effective_prefix = stage_params.get("prefix") or prefix
|
||||
if effective_prefix not in contexts_by_prefix:
|
||||
contexts_by_prefix[effective_prefix] = build_project_context(
|
||||
job_id=job_id,
|
||||
prefix=effective_prefix,
|
||||
project_root=project_root,
|
||||
include_stage_status=False,
|
||||
)
|
||||
report = resolve_stage_requirements(
|
||||
stage=stage,
|
||||
prefix=stage_params.get("prefix") or prefix,
|
||||
prefix=effective_prefix,
|
||||
job_id=job_id,
|
||||
params=stage_params,
|
||||
project_root=project_root,
|
||||
shadow_only=shadow_only,
|
||||
project_context=contexts_by_prefix[effective_prefix],
|
||||
)
|
||||
reports[stage] = report.to_dict()
|
||||
except Exception as exc:
|
||||
|
||||
@@ -33,6 +33,7 @@ RNASEQ_STATUS_STAGES = {
|
||||
"rnaseq_fgsea",
|
||||
"rnaseq_wgcna",
|
||||
"rnaseq_candidate_family",
|
||||
"rnaseq_report",
|
||||
"annotation",
|
||||
"annotation_go",
|
||||
"annotation_kegg",
|
||||
|
||||
@@ -546,6 +546,7 @@ class StageToolExecutor:
|
||||
ppi_reference_selection_mode=(extra_params or {}).get("ppiReferenceSelectionMode") or (extra_params or {}).get("ppi_reference_selection_mode") or "",
|
||||
ppi_reference_user_confirmed=bool_param("ppiReferenceUserConfirmed", "ppi_reference_user_confirmed", default=False),
|
||||
dependency_owner=(extra_params or {}).get("dependencyOwner") or (extra_params or {}).get("dependency_owner") or "",
|
||||
ortholog_mapping_owner=(extra_params or {}).get("orthologMappingOwner") or (extra_params or {}).get("ortholog_mapping_owner") or "",
|
||||
ppi_score=(extra_params or {}).get("ppiScore", 200),
|
||||
genome_prepare_type=(extra_params or {}).get("genomePrepareType", "auto"),
|
||||
kaks_neutral_rate=(extra_params or {}).get("kaksNeutralRate", 7.08e-9),
|
||||
@@ -1516,6 +1517,19 @@ class StageToolRegistry:
|
||||
"timeout": 7200,
|
||||
"conda_env": "rnaseq"
|
||||
},
|
||||
# 24. Formal RNA-seq Result Report
|
||||
{
|
||||
"id": "rnaseq_report",
|
||||
"name": "RNA-seq Result Report",
|
||||
"script": "run_rnaseq_report.py",
|
||||
"category": "rnaseq",
|
||||
"description": "Generate a scoped formal Markdown report from successful RNA-seq manifests",
|
||||
"required_inputs": [],
|
||||
"outputs": ["rnaseq_report", "rnaseq_report_evidence", "rnaseq_report_model"],
|
||||
"parameters": {"cpu": 4},
|
||||
"timeout": 1800,
|
||||
"conda_env": "test4py311"
|
||||
},
|
||||
# Combo Plot - 组合图
|
||||
{
|
||||
"id": "combo",
|
||||
|
||||
@@ -304,8 +304,21 @@ SPECIES_KEYWORDS = {
|
||||
"Cp": ["cp", "papaya", "木瓜"],
|
||||
"Vv": ["vv", "grape", "葡萄"],
|
||||
"Lg": ["lg"],
|
||||
"Gm": ["gm"],
|
||||
"Br": ["br"],
|
||||
"Pt": ["pt"],
|
||||
"Sb": ["sb"],
|
||||
}
|
||||
|
||||
_SPECIES_CODE_PATTERN = (
|
||||
r"(?<![A-Za-z0-9])(?:"
|
||||
+ "|".join(
|
||||
sorted((re.escape(code) for code in SPECIES_KEYWORDS), key=len, reverse=True)
|
||||
)
|
||||
+ r")(?![A-Za-z0-9])"
|
||||
)
|
||||
_ROLE_BOUND_SPECIES_CODE = r"[A-Z][A-Za-z0-9_-]{1,15}"
|
||||
|
||||
SPECIES_TO_LATIN = {
|
||||
"At": "Arabidopsis thaliana",
|
||||
"Os": "Oryza sativa",
|
||||
@@ -348,7 +361,11 @@ def _species_mentions_with_roles(query: str) -> Dict[str, List[str]]:
|
||||
def collect_species(segment: str) -> List[str]:
|
||||
segment_lower = segment.lower()
|
||||
found: List[str] = []
|
||||
for match in re.finditer(r"(?<![A-Za-z0-9])(?:Ft|At|Os|Ls|Zm|Sl|Lc|Dl|Ppy|Py|Cp|Vv|Lg)(?![A-Za-z0-9])", segment, flags=re.IGNORECASE):
|
||||
for match in re.finditer(
|
||||
_SPECIES_CODE_PATTERN,
|
||||
segment,
|
||||
flags=re.IGNORECASE,
|
||||
):
|
||||
code = canonical.get(match.group(0).lower())
|
||||
if code:
|
||||
found.append(code)
|
||||
@@ -371,6 +388,11 @@ def _species_mentions_with_roles(query: str) -> Dict[str, List[str]]:
|
||||
),
|
||||
("comparativeSpecies", r"比较物种|比较|对比|comparative|comparison|outgroup|外群"),
|
||||
]
|
||||
role_bound_marker_patterns = {
|
||||
"targetSpecies": r"目标物种|目标基因组|target\s+(?:species|genome)",
|
||||
"referenceSpecies": r"参考物种|参考基因组|reference\s+(?:species|genome)",
|
||||
"comparativeSpecies": r"比较物种|比较基因组|comparative\s+(?:species|genome)|outgroup",
|
||||
}
|
||||
roles = {"targetSpecies": [], "referenceSpecies": [], "comparativeSpecies": [], "unassigned": []}
|
||||
# A role may follow the species, as in "Ft是目标物种". Resolve
|
||||
# comma-delimited clauses first so a path fragment such as ref_data cannot
|
||||
@@ -380,6 +402,15 @@ def _species_mentions_with_roles(query: str) -> Dict[str, List[str]]:
|
||||
for role, pattern in marker_patterns:
|
||||
if re.search(pattern, clause_lower):
|
||||
roles[role].extend(collect_species(clause))
|
||||
role_bound_marker = role_bound_marker_patterns[role]
|
||||
for role_bound_pattern in (
|
||||
rf"(?i:(?:{role_bound_marker})\s*(?:为|是|作为|is|[::=])?\s*)({_ROLE_BOUND_SPECIES_CODE})",
|
||||
rf"({_ROLE_BOUND_SPECIES_CODE})(?i:\s*(?:是|为|作为|is)?\s*(?:{role_bound_marker}))",
|
||||
):
|
||||
roles[role].extend(
|
||||
match.group(1)
|
||||
for match in re.finditer(role_bound_pattern, clause)
|
||||
)
|
||||
if any(roles[role] for role in ("targetSpecies", "referenceSpecies", "comparativeSpecies")):
|
||||
assigned = set(roles["targetSpecies"] + roles["referenceSpecies"] + roles["comparativeSpecies"])
|
||||
roles["unassigned"] = [sp for sp in collect_species(text) if sp not in assigned]
|
||||
@@ -399,8 +430,7 @@ def _species_mentions_with_roles(query: str) -> Dict[str, List[str]]:
|
||||
return {key: _unique_keep_order(value) for key, value in roles.items()}
|
||||
|
||||
mentions: List[tuple[int, str]] = []
|
||||
code_pattern = r"(?<![A-Za-z0-9])(?:Ft|At|Os|Ls|Zm|Sl|Lc|Dl|Ppy|Py|Cp|Vv|Lg)(?![A-Za-z0-9])"
|
||||
for match in re.finditer(code_pattern, text, flags=re.IGNORECASE):
|
||||
for match in re.finditer(_SPECIES_CODE_PATTERN, text, flags=re.IGNORECASE):
|
||||
code = canonical.get(match.group(0).lower())
|
||||
if code:
|
||||
mentions.append((match.start(), code))
|
||||
|
||||
@@ -2611,6 +2611,50 @@ class WorkflowStateSnapshot(_WorkflowModel):
|
||||
return self
|
||||
|
||||
|
||||
class CurrentRecommendation(_WorkflowModel):
|
||||
recommendation_id: str
|
||||
job_id: str
|
||||
workflow_id: Literal[
|
||||
"data_preparation",
|
||||
"gene_family",
|
||||
"rnaseq_whole_genome",
|
||||
"rnaseq_family_member",
|
||||
]
|
||||
transition_id: str
|
||||
from_state_id: str
|
||||
to_state_id: str
|
||||
target_stage: str
|
||||
authorized_stages: tuple[str, ...] = Field(min_length=1)
|
||||
evidence_digest: str
|
||||
family: str | None = None
|
||||
scope_id: str | None = None
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_recommendation(self) -> "CurrentRecommendation":
|
||||
for field_name in (
|
||||
"recommendation_id",
|
||||
"job_id",
|
||||
"transition_id",
|
||||
"from_state_id",
|
||||
"to_state_id",
|
||||
"target_stage",
|
||||
):
|
||||
_require_nonblank(getattr(self, field_name), field_name)
|
||||
if len(self.authorized_stages) != len(set(self.authorized_stages)):
|
||||
raise ValueError("authorized_stages must contain unique stages")
|
||||
if any(not stage.strip() for stage in self.authorized_stages):
|
||||
raise ValueError("authorized_stages cannot contain blank stages")
|
||||
if self.target_stage not in self.authorized_stages:
|
||||
raise ValueError("target_stage must be authorized")
|
||||
if not _SHA256_RE.fullmatch(self.evidence_digest):
|
||||
raise ValueError("evidence_digest must be a sha256 hex digest")
|
||||
for field_name in ("family", "scope_id"):
|
||||
value = getattr(self, field_name)
|
||||
if value is not None:
|
||||
_require_nonblank(value, field_name)
|
||||
return self
|
||||
|
||||
|
||||
class GoalLedger(_WorkflowModel):
|
||||
goal_ledger_version: Literal["goal_ledger_v1"] = "goal_ledger_v1"
|
||||
conversation_id: str
|
||||
|
||||
@@ -9,6 +9,16 @@ from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
|
||||
ORTHOLOG_DEPENDENCY_OWNERS = {"", "ppi", "kaks"}
|
||||
|
||||
|
||||
def ortholog_output_stem(prefix: str, dependency_owner: str = "") -> str:
|
||||
owner = str(dependency_owner or "").strip().lower()
|
||||
if owner not in ORTHOLOG_DEPENDENCY_OWNERS:
|
||||
raise ValueError(f"unsupported Ortholog dependency owner: {dependency_owner}")
|
||||
return f"{prefix}.{owner}" if owner else prefix
|
||||
|
||||
|
||||
@dataclass
|
||||
class OrthologRunRecord:
|
||||
query_species: str
|
||||
@@ -151,6 +161,7 @@ def materialize_ortholog_final_outputs(
|
||||
*,
|
||||
final_dir: Path,
|
||||
prefix: str,
|
||||
dependency_owner: str = "",
|
||||
run_records: list[OrthologRunRecord],
|
||||
job_id: str,
|
||||
workspace: Path,
|
||||
@@ -170,10 +181,12 @@ def materialize_ortholog_final_outputs(
|
||||
|
||||
final_dir.mkdir(parents=True, exist_ok=True)
|
||||
scope_fields = dict(scope_fields or {})
|
||||
owner = str(dependency_owner or "").strip().lower()
|
||||
output_stem = ortholog_output_stem(prefix, owner)
|
||||
|
||||
pairs_dst = final_dir / f"{prefix}.ortholog_pairs.txt"
|
||||
orthologs_dst = final_dir / f"{prefix}.orthologs.txt"
|
||||
raw_pairs_dst = final_dir / f"{prefix}.ortholog_pairs.raw.txt"
|
||||
pairs_dst = final_dir / f"{output_stem}.ortholog_pairs.txt"
|
||||
orthologs_dst = final_dir / f"{output_stem}.orthologs.txt"
|
||||
raw_pairs_dst = final_dir / f"{output_stem}.ortholog_pairs.raw.txt"
|
||||
|
||||
# run_ortholog.py writes subject<TAB>query<TAB>identity.
|
||||
# Downstream PPI expects query/source -> target/reference.
|
||||
@@ -203,7 +216,7 @@ def materialize_ortholog_final_outputs(
|
||||
pairs_handle.write(out_line)
|
||||
orthologs_handle.write(out_line)
|
||||
|
||||
blast_dst = final_dir / f"{prefix}.ortholog_blastp.txt"
|
||||
blast_dst = final_dir / f"{output_stem}.ortholog_blastp.txt"
|
||||
with blast_dst.open("w", encoding="utf-8") as blast_handle:
|
||||
for record in run_records:
|
||||
blast_src = record.blast_src
|
||||
@@ -266,7 +279,7 @@ def materialize_ortholog_final_outputs(
|
||||
"sha256": _sha256_file(blast_dst),
|
||||
}
|
||||
|
||||
ortholog_manifest = final_dir / f"{prefix}.ortholog.manifest.json"
|
||||
ortholog_manifest = final_dir / f"{output_stem}.ortholog.manifest.json"
|
||||
scope_file = str(scope_fields.pop("scope_file", "") or "").strip()
|
||||
scope_data = dict(scope_fields)
|
||||
|
||||
@@ -283,6 +296,7 @@ def materialize_ortholog_final_outputs(
|
||||
"stage": "ortholog",
|
||||
"status": "success",
|
||||
"prefix": prefix,
|
||||
"dependency_owner": owner,
|
||||
"job_id": job_id,
|
||||
"workspace": str(workspace),
|
||||
"workspace_root": str(workspace),
|
||||
@@ -303,6 +317,7 @@ def materialize_ortholog_final_outputs(
|
||||
],
|
||||
"parameters": {
|
||||
"prefix": prefix,
|
||||
"dependency_owner": owner,
|
||||
"blast_evalue": blast_evalue,
|
||||
"cpu": cpu,
|
||||
"reference_species": reference_species,
|
||||
@@ -340,7 +355,7 @@ def materialize_ortholog_final_outputs(
|
||||
if scope_id:
|
||||
from gfa_core.gene_family_scope import sanitize_token
|
||||
|
||||
scoped_manifest = final_dir / f"{prefix}.{sanitize_token(scope_id)}.ortholog.manifest.json"
|
||||
scoped_manifest = final_dir / f"{output_stem}.{sanitize_token(scope_id)}.ortholog.manifest.json"
|
||||
manifest_data["scope_manifest"] = str(scoped_manifest.resolve())
|
||||
with scoped_manifest.open("w", encoding="utf-8") as handle:
|
||||
json.dump(manifest_data, handle, ensure_ascii=False, indent=2)
|
||||
|
||||
@@ -0,0 +1,313 @@
|
||||
"""Lightweight, provider-free RNA-seq manifest evidence snapshot.
|
||||
|
||||
The live interpretation view and the formal report share this module so scope
|
||||
selection and manifest provenance have one implementation. It intentionally
|
||||
does not import the MAS agent package.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
REPORT_STAGE = "rnaseq_report"
|
||||
SUCCESS_STATUSES = {"success", "partial_success"}
|
||||
GLOBAL_LEGACY_STAGES = {
|
||||
"rnaseq_qc",
|
||||
"rnaseq_align",
|
||||
"rnaseq_quant",
|
||||
"annotation",
|
||||
"annotation_go",
|
||||
"annotation_kegg",
|
||||
}
|
||||
RNASEQ_STAGE_NAMES = {
|
||||
"rnaseq_qc": "RNA-seq质控",
|
||||
"rnaseq_align": "RNA-seq比对",
|
||||
"rnaseq_quant": "RNA-seq定量",
|
||||
"rnaseq_sample_qc": "RNA-seq样本质量评估",
|
||||
"rnaseq_de": "RNA-seq差异表达",
|
||||
"annotation": "背景功能注释",
|
||||
"annotation_go": "GO背景自动注释",
|
||||
"annotation_kegg": "KEGG/KO背景自动注释",
|
||||
"rnaseq_go_enrich": "RNA-seq GO富集分析",
|
||||
"rnaseq_kegg_enrich": "RNA-seq KEGG富集分析",
|
||||
"rnaseq_enrich": "RNA-seq富集分析(兼容入口)",
|
||||
"rnaseq_fgsea": "RNA-seq fgsea排序富集",
|
||||
"rnaseq_wgcna": "RNA-seq WGCNA共表达分析",
|
||||
"rnaseq_candidate_family": "RNA-seq候选家族筛选",
|
||||
"rnaseq_family_score": "RNA-seq候选家族评分",
|
||||
}
|
||||
RNASEQ_STAGE_ORDER = list(RNASEQ_STAGE_NAMES)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RNASeqManifestSelection:
|
||||
rnaseq_scope_id: str
|
||||
discovered_scope_ids: tuple[str, ...]
|
||||
manifests: dict[str, dict[str, Any]]
|
||||
|
||||
|
||||
def read_json(path: Path) -> dict[str, Any]:
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
return data if isinstance(data, dict) else {}
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return {}
|
||||
|
||||
|
||||
def manifest_identity_value(manifest: dict[str, Any], key: str) -> str:
|
||||
context = manifest.get("stage_context") or manifest.get("stageContext") or {}
|
||||
params = manifest.get("parameters") or {}
|
||||
return str(manifest.get(key) or params.get(key) or context.get(key) or "").strip()
|
||||
|
||||
|
||||
def is_rnaseq_evidence_stage(stage: str) -> bool:
|
||||
stage = str(stage or "").strip()
|
||||
return stage != REPORT_STAGE and (
|
||||
stage.startswith("rnaseq_") or stage in {"annotation", "annotation_go", "annotation_kegg"}
|
||||
)
|
||||
|
||||
|
||||
def stage_sort_key(stage: str) -> tuple[int, str]:
|
||||
try:
|
||||
return RNASEQ_STAGE_ORDER.index(stage), stage
|
||||
except ValueError:
|
||||
return len(RNASEQ_STAGE_ORDER), stage
|
||||
|
||||
|
||||
def _manifest_score(path: Path, manifest: dict[str, Any]) -> tuple[int, int, int, float]:
|
||||
status = str(manifest.get("status") or "")
|
||||
status_rank = 2 if status == "success" else 1 if status == "partial_success" else 0
|
||||
metrics = manifest.get("metrics") if isinstance(manifest.get("metrics"), dict) else {}
|
||||
artifacts = manifest.get("artifacts") if isinstance(manifest.get("artifacts"), dict) else {}
|
||||
try:
|
||||
mtime = path.stat().st_mtime
|
||||
except OSError:
|
||||
mtime = 0.0
|
||||
return status_rank, len(metrics), len(artifacts), mtime
|
||||
|
||||
|
||||
def select_rnaseq_manifests(
|
||||
final_dir: Path,
|
||||
prefix: str,
|
||||
*,
|
||||
rnaseq_scope_id: str = "",
|
||||
) -> RNASeqManifestSelection:
|
||||
"""Select one non-mixing manifest per RNA-seq stage for a prefix/scope."""
|
||||
final_dir = Path(final_dir)
|
||||
pattern = f"{prefix}.*.manifest.json" if prefix else "*.manifest.json"
|
||||
candidates: list[tuple[Path, dict[str, Any]]] = []
|
||||
latest_by_scope: dict[str, float] = {}
|
||||
for path in sorted(final_dir.glob(pattern)) if final_dir.exists() else []:
|
||||
manifest = read_json(path)
|
||||
stage = str(manifest.get("stage") or "")
|
||||
if not is_rnaseq_evidence_stage(stage):
|
||||
continue
|
||||
manifest_prefix = str(manifest.get("prefix") or "").strip()
|
||||
if prefix and manifest_prefix and manifest_prefix != prefix:
|
||||
continue
|
||||
candidates.append((path, manifest))
|
||||
scope = manifest_identity_value(manifest, "rnaseq_scope_id")
|
||||
if scope:
|
||||
try:
|
||||
latest_by_scope[scope] = max(latest_by_scope.get(scope, 0.0), path.stat().st_mtime)
|
||||
except OSError:
|
||||
latest_by_scope.setdefault(scope, 0.0)
|
||||
|
||||
discovered = sorted(latest_by_scope, key=lambda item: (-latest_by_scope[item], item))
|
||||
resolved_scope = str(rnaseq_scope_id or "").strip() or (discovered[0] if discovered else "")
|
||||
selected_paths: dict[str, Path] = {}
|
||||
selected: dict[str, dict[str, Any]] = {}
|
||||
for path, manifest in candidates:
|
||||
stage = str(manifest.get("stage") or "")
|
||||
scope = manifest_identity_value(manifest, "rnaseq_scope_id")
|
||||
if resolved_scope and scope and scope != resolved_scope:
|
||||
continue
|
||||
if resolved_scope and not scope and stage not in GLOBAL_LEGACY_STAGES:
|
||||
continue
|
||||
existing = selected.get(stage)
|
||||
if existing is not None and _manifest_score(selected_paths[stage], existing) >= _manifest_score(path, manifest):
|
||||
continue
|
||||
manifest = dict(manifest)
|
||||
manifest["_manifest_path"] = str(path)
|
||||
selected[stage] = manifest
|
||||
selected_paths[stage] = path
|
||||
return RNASeqManifestSelection(resolved_scope, tuple(discovered), selected)
|
||||
|
||||
|
||||
def _compact_value(value: Any, limit: int = 240) -> str:
|
||||
if isinstance(value, (dict, list)):
|
||||
text = json.dumps(value, ensure_ascii=False, sort_keys=True)
|
||||
else:
|
||||
text = str(value)
|
||||
return text if len(text) <= limit else text[: limit - 1] + "…"
|
||||
|
||||
|
||||
def resolve_artifact_path(
|
||||
final_dir: Path,
|
||||
item: dict[str, Any],
|
||||
*,
|
||||
require_exists: bool = False,
|
||||
) -> Path | None:
|
||||
"""Resolve artifacts after a workspace has been moved or mounted elsewhere."""
|
||||
final_dir = Path(final_dir)
|
||||
workspace = final_dir.parent
|
||||
candidates: list[Path] = []
|
||||
for raw in (item.get("full_path"), item.get("path")):
|
||||
if not raw:
|
||||
continue
|
||||
path = Path(str(raw))
|
||||
candidates.append(path if path.is_absolute() else workspace / path)
|
||||
if path.is_absolute():
|
||||
candidates.append(final_dir / path.name)
|
||||
|
||||
unique: list[Path] = []
|
||||
seen: set[str] = set()
|
||||
for path in candidates:
|
||||
identity = str(path)
|
||||
if identity not in seen:
|
||||
seen.add(identity)
|
||||
unique.append(path)
|
||||
for path in unique:
|
||||
if path.exists():
|
||||
return path
|
||||
if require_exists:
|
||||
return None
|
||||
return unique[0] if unique else None
|
||||
|
||||
|
||||
def build_rnaseq_evidence_snapshot(
|
||||
final_dir: Path,
|
||||
prefix: str,
|
||||
*,
|
||||
rnaseq_scope_id: str = "",
|
||||
include_stages: Iterable[str] | None = None,
|
||||
selection: RNASeqManifestSelection | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Build a compact evidence object without invoking an LLM or agent cache."""
|
||||
selection = selection or select_rnaseq_manifests(
|
||||
final_dir, prefix, rnaseq_scope_id=rnaseq_scope_id
|
||||
)
|
||||
allowed = {str(stage) for stage in include_stages or [] if str(stage).strip()}
|
||||
manifests = {
|
||||
stage: manifest
|
||||
for stage, manifest in selection.manifests.items()
|
||||
if not allowed or stage in allowed
|
||||
}
|
||||
stages = sorted(manifests, key=stage_sort_key)
|
||||
stage_flow: list[dict[str, Any]] = []
|
||||
findings: list[dict[str, Any]] = []
|
||||
reviews: list[dict[str, Any]] = []
|
||||
cards: list[dict[str, Any]] = []
|
||||
risks: list[dict[str, Any]] = []
|
||||
card_index = 0
|
||||
for stage in stages:
|
||||
manifest = manifests[stage]
|
||||
status = str(manifest.get("status") or "unknown")
|
||||
metrics = manifest.get("metrics") if isinstance(manifest.get("metrics"), dict) else {}
|
||||
parameters = manifest.get("parameters") if isinstance(manifest.get("parameters"), dict) else {}
|
||||
artifacts = manifest.get("artifacts") if isinstance(manifest.get("artifacts"), dict) else {}
|
||||
metric_items = list(sorted(metrics.items()))[:12]
|
||||
metric_summary = ", ".join(f"{key}={_compact_value(value, 80)}" for key, value in metric_items[:5])
|
||||
key_result = metric_summary or f"登记产物 {len(artifacts)} 个"
|
||||
stage_flow.append(
|
||||
{
|
||||
"id": stage,
|
||||
"name": RNASEQ_STAGE_NAMES.get(stage, stage),
|
||||
"status": status,
|
||||
"key_result": key_result,
|
||||
}
|
||||
)
|
||||
findings.append(
|
||||
{
|
||||
"stage": stage,
|
||||
"title": RNASEQ_STAGE_NAMES.get(stage, stage),
|
||||
"description": f"manifest 状态为 {status};{key_result}。",
|
||||
}
|
||||
)
|
||||
limitations: list[str] = []
|
||||
if status == "partial_success":
|
||||
limitations.append("该阶段为 partial_success,结果可用但不完整。")
|
||||
elif status != "success":
|
||||
limitations.append("该阶段未成功,不纳入正式结论。")
|
||||
missing_artifacts: list[str] = []
|
||||
for name, item in sorted(artifacts.items()):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
path = resolve_artifact_path(final_dir, item)
|
||||
exists = bool(path and path.exists())
|
||||
card_index += 1
|
||||
cards.append(
|
||||
{
|
||||
"id": f"RNA-E{card_index:04d}",
|
||||
"stage": stage,
|
||||
"claim": f"产物 {name}",
|
||||
"evidence": f"path={path or ''}, exists={exists}",
|
||||
"source_type": "raw_file" if exists else "manifest",
|
||||
}
|
||||
)
|
||||
if not exists:
|
||||
missing_artifacts.append(str(name))
|
||||
for key, value in metric_items:
|
||||
card_index += 1
|
||||
cards.append(
|
||||
{
|
||||
"id": f"RNA-E{card_index:04d}",
|
||||
"stage": stage,
|
||||
"claim": f"指标 {key}",
|
||||
"evidence": _compact_value(value),
|
||||
"source_type": "manifest",
|
||||
}
|
||||
)
|
||||
safe_parameter_items = [
|
||||
(key, value)
|
||||
for key, value in sorted(parameters.items())
|
||||
if not any(secret in str(key).lower() for secret in ("api_key", "token", "secret", "password"))
|
||||
][:10]
|
||||
for key, value in safe_parameter_items:
|
||||
card_index += 1
|
||||
cards.append(
|
||||
{
|
||||
"id": f"RNA-E{card_index:04d}",
|
||||
"stage": stage,
|
||||
"claim": f"参数 {key}",
|
||||
"evidence": _compact_value(value),
|
||||
"source_type": "manifest",
|
||||
}
|
||||
)
|
||||
if missing_artifacts:
|
||||
shown = missing_artifacts[:12]
|
||||
remainder = len(missing_artifacts) - len(shown)
|
||||
suffix = f"(另有 {remainder} 个,详见 evidence JSON)" if remainder else ""
|
||||
message = f"manifest 登记但文件不存在的产物:{', '.join(shown)}{suffix}"
|
||||
limitations.append(message)
|
||||
risks.append({"stage": stage, "severity": "high", "message": message})
|
||||
if status not in SUCCESS_STATUSES:
|
||||
risks.append({"stage": stage, "severity": "high", "message": f"阶段状态为 {status},不得作为成功证据。"})
|
||||
reviews.append({"stage": stage, "stage_name": RNASEQ_STAGE_NAMES.get(stage, stage), "limitations": limitations})
|
||||
|
||||
successful = [stage for stage in stages if str(manifests[stage].get("status") or "") in SUCCESS_STATUSES]
|
||||
return {
|
||||
"schema": "rnaseq_manifest_evidence_v1",
|
||||
"metadata": {
|
||||
"prefix": prefix,
|
||||
"rnaseq_scope_id": selection.rnaseq_scope_id,
|
||||
"discovered_scope_ids": list(selection.discovered_scope_ids),
|
||||
"included_stages": stages,
|
||||
"successful_stages": successful,
|
||||
"interpretation_boundary": "仅陈述当前 prefix 与 RNA-seq scope 下 manifest/产物支持的事实;富集、相关性和候选排序不得写成调控因果。",
|
||||
},
|
||||
"stage_flow": {"visualization": " -> ".join(stages), "stages": stage_flow},
|
||||
"key_findings": findings,
|
||||
"stage_reviews": reviews,
|
||||
"evidence_cards": cards,
|
||||
"risk_flags": risks,
|
||||
"major_claims": [],
|
||||
"biological_story": (
|
||||
f"当前报告汇总了 {len(successful)} 个成功或部分成功的 RNA-seq 阶段。"
|
||||
"跨阶段生物学解释必须以各阶段数值、图表和风险标记为边界。"
|
||||
),
|
||||
"next_steps": ["补齐缺失或失败阶段后重新生成报告,以刷新证据快照。"],
|
||||
}
|
||||
@@ -18,7 +18,7 @@ import mimetypes
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
from urllib.parse import unquote
|
||||
@@ -79,8 +79,21 @@ class RNASeqReportResult:
|
||||
reused: bool = False
|
||||
|
||||
|
||||
BEIJING_TIMEZONE = timezone(timedelta(hours=8))
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now().astimezone().isoformat(timespec="seconds")
|
||||
return datetime.now(BEIJING_TIMEZONE).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def _display_generated_at(value: str) -> str:
|
||||
try:
|
||||
parsed = datetime.fromisoformat(value)
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=BEIJING_TIMEZONE)
|
||||
return parsed.astimezone(BEIJING_TIMEZONE).strftime("%Y-%m-%d %H:%M:%S")
|
||||
except (TypeError, ValueError):
|
||||
return str(value or "").replace("T", " ")
|
||||
|
||||
|
||||
def _read_json(path: Path) -> dict[str, Any]:
|
||||
@@ -1224,7 +1237,7 @@ def _render_deterministic_markdown(
|
||||
lines = [
|
||||
f"# {prefix} RNA-seq 结果解读报告",
|
||||
"",
|
||||
f"生成时间:{generated_at} ",
|
||||
f"生成时间(北京时间):{_display_generated_at(generated_at)} ",
|
||||
f"任务编号:`{workspace.name}`",
|
||||
"",
|
||||
"**任务结果路径:** ",
|
||||
|
||||
@@ -27,6 +27,11 @@
|
||||
- 若 reference CDS 在 config/scope 中缺失,可从已知 reference 数据目录解析,例如 At 的 TAIR10 CDS。
|
||||
- KaKs 是 reference dependency owner:若缺少 ortholog mapping,可以要求调度器先自动插入 Ortholog;但若 reference CDS/PEP 不完整,必须先提示上传/下载完整参考注释包,不能只跑 Ortholog 后再失败。
|
||||
- 若 PPI 与 KaKs 同批执行,reference resolver 必须按阶段分别检查;KaKs 使用 `requested_stages=["kaks"]` 检查 reference PEP/CDS 与 Ortholog,而其缺失状态不得覆盖 PPI 的独立就绪状态。
|
||||
- 普通计划不显示隐式 Ortholog,KaKs 也不声明对 Ortholog 或 PPI 的静态硬依赖。执行层仅用软顺序保证 KaKs 参数确认发生在 PPI 真实终态之后。
|
||||
- KaKs 确认必须显式选择参考来源:复用本次 PPI 技术参考,或使用项目配置参考。选择只写入 KaKs 参数,不得修改 PPI owner 或其它阶段的参考 scope。
|
||||
- 复用 PPI 参考时,只有 identity 完全匹配的 `final/{prefix}.ppi.orthologs.txt` 可用,并以 `orthologMappingOwner=ppi` 提交。
|
||||
- 选择项目参考时,缺少精确映射则运行 `dependency_owner=kaks` 的内部 Ortholog,输出 `final/{prefix}.kaks.orthologs.txt`,随后以 `orthologMappingOwner=kaks` 提交。
|
||||
- 两种 owner 都不得回落到另一 owner 或 standalone 映射。缺少所选参考的 PEP/CDS/映射只暂停 KaKs,不影响已完成 PPI 或其它阶段。
|
||||
|
||||
### Legacy 模式
|
||||
|
||||
|
||||
@@ -59,6 +59,19 @@
|
||||
| `{prefix}.ortholog.manifest.json` | json | 标准 manifest |
|
||||
| `{prefix}.{scope_id}.ortholog.manifest.json` | json | scope 模式下的 scoped manifest 副本 |
|
||||
|
||||
Owner namespace:
|
||||
|
||||
| producer 身份 | 映射/manifest stem | 用途 |
|
||||
|---|---|---|
|
||||
| standalone (`dependency_owner=""`) | `{prefix}.ortholog*` | 用户显式请求的 Ortholog,保持原契约 |
|
||||
| PPI (`dependency_owner="ppi"`) | `{prefix}.ppi.ortholog*` | PPI STRING 技术参考专属内部映射 |
|
||||
| KaKs (`dependency_owner="kaks"`) | `{prefix}.kaks.ortholog*` | KaKs 显式项目参考专属内部映射 |
|
||||
|
||||
- owner namespace 同时作用于 orthologs、pairs、raw pairs、BLAST、identity sidecar、标准 manifest 与 scoped manifest。
|
||||
- manifest 的 `stage` 仍为 `ortholog`;`parameters.dependency_owner` 记录 producer 身份。
|
||||
- standalone、PPI、KaKs 三套产物可为同一 prefix 并存;重跑一个 owner 不得覆盖或改变另一个 owner。
|
||||
- 下游按 owner token 解析候选路径,不接受前端传入任意文件路径,也不得跨 owner 回落。
|
||||
|
||||
## Manifest 最小字段
|
||||
|
||||
```json
|
||||
@@ -104,6 +117,7 @@
|
||||
- 单 target + `target_only` scope 时,`analysis_route` 通常为 `single_species`;含 reference 的 comparative scope 通常为 `multi_species`。
|
||||
- postflight strict identity 诊断依赖上述字段;materialize 落盘时必须补齐,不能只写 `scope_id`。
|
||||
- PPI-owned producer 的 `reference_taxon_id` 来自已确认 STRING TaxID,`reference_storage_key` 决定本地 PEP 路径及 mapping 的 Target_Species。
|
||||
- PPI-owned producer 的生物学 scope 必须与 PPI consumer 相同,固定为目标家族 `target_only`;技术参考物种只写入 owner/provenance 参数。
|
||||
- sidecar 与当前请求必须在 `reference_taxon_id`、`reference_storage_key`、reference PEP SHA256、target/query PEP identity、scope/member identity、mapping path/SHA256 和 `mapping_identity` 上完全一致,才允许跳过 BLAST 并复用中间 pairs。
|
||||
- sidecar 缺失、无法解析、字段缺失或任一 identity 冲突都视为不可复用;应按原 Ortholog 命令重新生成,不得根据“文件存在”推断身份。
|
||||
- PPI preflight 不得只读取 final mapping 的 Target_Species 后判定 ready;还必须同时验证 final mapping sidecar、对应 Ortholog manifest,以及当前 reference PEP、target/query PEP、scope/member 和 mapping identity。任一输入变化或身份材料缺失均返回 `needs_ortholog` / stale identity conflict。
|
||||
|
||||
@@ -181,6 +181,17 @@ python skills/ppi/run.py \
|
||||
## 下游阶段
|
||||
- summary: 可汇总PPI结果
|
||||
|
||||
## 内部 Ortholog 身份与调度
|
||||
|
||||
- 普通基因家族计划只展示 PPI;PPI 所需的 Ortholog 是执行层内部 producer,不作为可见计划阶段或独立确认弹窗。
|
||||
- PPI 的生物学分析身份固定为目标家族成员 `target_only`;项目参考物种不得进入 PPI 的 `scope_id`、`analysis_route` 或 `member_set_id`。
|
||||
- STRING TaxID、技术参考 storage key 和同源映射参考只属于 PPI 技术溯源,不得被当作生物学 `referenceSpecies` 重建 PPI scope。
|
||||
- 非目标物种直连 STRING 路线使用 `dependency_owner=ppi`,映射输出为 `final/{prefix}.ppi.orthologs.txt`(及同 owner 的 pairs/raw/BLAST/manifest/sidecar)。
|
||||
- PPI 只能读取 PPI owner 映射;不得因 standalone 或 KaKs owner 文件存在而静默复用。
|
||||
- PPI 的 STRING 技术参考只属于 PPI。它不得覆盖项目参考物种或 KaKs 的显式参考选择。
|
||||
- PPI 成功、失败或用户明确跳过都会释放后续 KaKs 确认;正在进行的 PPI 参考包下载不是终态。
|
||||
- PPI-owned Ortholog 完成后必须用同一 target-only 身份重新执行预检;只有返回 `ready` 才能提交 PPI。
|
||||
|
||||
## 网络统计指标
|
||||
|
||||
| 指标 | 描述 |
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# RNA-seq Result Report Contract v1
|
||||
|
||||
## Stage
|
||||
|
||||
- Stage id: `rnaseq_report`
|
||||
- Script: `apps/cli/run_rnaseq_report.py`
|
||||
- Scope: one exact `prefix + rnaseq_scope_id` evidence snapshot
|
||||
- Purpose: materialize a formal, reproducible RNA-seq Markdown result report without depending on frontend session state.
|
||||
|
||||
## Inputs
|
||||
|
||||
- At least one RNA-seq/annotation manifest with status `success` or `partial_success` under `final/`.
|
||||
- All selected evidence must match the requested prefix and RNA-seq scope. Legacy unscoped QC/alignment/quantification/annotation manifests may be treated as experiment-global evidence.
|
||||
- `expected_stages` is optional and records the stages selected for the current batch. It controls coverage only; it must not become a static execution dependency.
|
||||
- An existing configured `LLMManager` provider is optional. No provider or model name is required by this stage.
|
||||
|
||||
If no successful RNA-seq evidence exists, the stage must return `暂无可生成的 RNA-seq 结果` and must not create an empty report.
|
||||
|
||||
## Required Outputs
|
||||
|
||||
- `final/{prefix}.rnaseq_report.md`
|
||||
- `final/{prefix}.rnaseq_report.evidence.json`
|
||||
- `final/{prefix}.rnaseq_report.model.json`
|
||||
- `final/{prefix}.rnaseq_report.manifest.json`
|
||||
|
||||
The manifest stage status is `success` whenever these outputs are generated. Content completeness is recorded separately as `metrics.coverage_status = complete|partial`.
|
||||
|
||||
## Generation Semantics
|
||||
|
||||
- `generate`: always rebuild from the current manifest snapshot.
|
||||
- `open`: reuse a fresh report; regenerate first when the source/request fingerprint changed.
|
||||
- The deterministic Markdown body is always complete enough to read without an API.
|
||||
- Optional model output is a labeled language-enhancement section only. Model failure must retain the deterministic report and be recorded in `.model.json`.
|
||||
- `rnaseq_report` manifests are excluded from their own evidence scan.
|
||||
|
||||
## Boundary With Deep Interpretation
|
||||
|
||||
Both features may call the pure `gfa_core.rnaseq_evidence` snapshot builder. The formal report must not call `/api/agent/interpret`, read its session cache, or replace the frontend deep-interpretation view.
|
||||
|
||||
## Validation
|
||||
|
||||
- Every manifest artifact path exists and its digest is recorded.
|
||||
- Evidence contains only the selected prefix/scope.
|
||||
- `coverage_status=complete` only when every expected stage has status `success`.
|
||||
- Missing, failed, or `partial_success` expected stages yield `coverage_status=partial` while the report manifest remains `success`.
|
||||
@@ -3,7 +3,7 @@
|
||||
## 阶段信息
|
||||
|
||||
- **阶段名称**: summary
|
||||
- **阶段描述**: 汇总整个 job 的 manifest、artifact、核心指标和一致性检查;作为结果入口和质控层,不承担深度生物学解读。
|
||||
- **阶段描述**: 先汇总整个 job 的 manifest、artifact、核心指标和一致性检查,再通过 DeepSeek API 将本地证据写成通俗、可追溯的结果解读报告。
|
||||
- **脚本**: `apps/cli/run_summary.py`
|
||||
|
||||
## 输入契约
|
||||
@@ -30,15 +30,18 @@ Summary 应囊括所有已运行阶段,包括但不限于:
|
||||
| 文件 | 类型 | 描述 |
|
||||
|------|------|------|
|
||||
| `{prefix}.summary_report.md` | markdown | 人可读汇总报告 |
|
||||
| `{prefix}.summary_evidence_index.md` | markdown | 确定性结果索引与完整性检查;不经过 LLM |
|
||||
| `{prefix}.summary.evidence.json` | json | 提交给 DeepSeek 的本地事实包 |
|
||||
| `{prefix}.summary.deepseek.json` | json | 模型、用量、图表覆盖率和报告校验记录 |
|
||||
| `{prefix}.summary.files.tsv` | tsv | artifact 文件索引 |
|
||||
| `{prefix}.summary.manifest.json` | json | 机器可读总汇总 |
|
||||
| `{prefix}.{scope_id}.summary_report.md` | markdown | 指定 scope 的报告 |
|
||||
| `{prefix}.{scope_id}.summary.files.tsv` | tsv | 指定 scope 的文件索引 |
|
||||
| `{prefix}.{scope_id}.summary.manifest.json` | json | 指定 scope 的机器汇总 |
|
||||
|
||||
## Summary 与深度解读的边界
|
||||
## Summary 的确定性层与 DeepSeek 层
|
||||
|
||||
Summary 做:
|
||||
确定性层做:
|
||||
|
||||
- 阶段状态汇总
|
||||
- 输出文件索引
|
||||
@@ -46,11 +49,18 @@ Summary 做:
|
||||
- scope/family/species 一致性检查
|
||||
- 缺失依赖或 artifact 检查
|
||||
|
||||
Summary 不做:
|
||||
DeepSeek 层做:
|
||||
|
||||
- 论文式结果解释
|
||||
- 候选基因排序的主观判断
|
||||
- 跨数据库机制推断
|
||||
- 文献背景补全
|
||||
- 读取确定性层生成的证据包,不直接扫描和猜测未知结果
|
||||
- 按分析阶段逐项解释结果和每一个逻辑图
|
||||
- 用短句和通俗中文说明“看到了什么、可能说明什么、不能说明什么”
|
||||
- 汇总候选成员证据、限制和下一步验证建议
|
||||
|
||||
深度解读应读取 `summary.manifest.json` 和关键结果文件,作为单独阶段执行。
|
||||
DeepSeek 层禁止:
|
||||
|
||||
- 补造本地证据中没有的数字、成员、分组、功能或文献
|
||||
- 把预测、相关或候选线索写成已经证明的功能
|
||||
- 在高风险或缺失阶段给出肯定结论
|
||||
- 把全基因组背景数量误写成家族成员数
|
||||
|
||||
默认运行必须成功调用 DeepSeek API 并通过报告完整性检查;`--no-deepseek` 只用于离线测试或管理员排障,生成物必须标记为确定性回退报告。
|
||||
|
||||
@@ -33,10 +33,6 @@
|
||||
|
||||
各阶段依次写明分析对象、源结果中的直接观察、谨慎解释、限制,并仅在有助复核时列出少量相对路径。
|
||||
|
||||
### 2.2 语言边界
|
||||
|
||||
预测关系统一表述为线索;没有逐成员表或独立证据时,不指定候选基因,不声称功能改变或调控关系已经证实。
|
||||
|
||||
<!-- GFA_SECTION common.completeness -->
|
||||
## 三、结果总览与完整性检查
|
||||
|
||||
|
||||
+58
-19
@@ -2,15 +2,19 @@
|
||||
# PlantAI-dev 9000 服务管理脚本
|
||||
# 用法: bash manage_plantai.sh {start|stop|restart|status} [--port PORT]
|
||||
# 默认端口: 9000
|
||||
#
|
||||
# 项目: /home/xzq/5090/teamWorkspace/zh/PlantAI-dev
|
||||
# 端口: 9000 (web), Redis DB 14 (queue prefix: plantai_dev_9002)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
if [ -f "$SCRIPT_DIR/.env" ]; then
|
||||
set -a
|
||||
. "$SCRIPT_DIR/.env"
|
||||
set +a
|
||||
fi
|
||||
|
||||
# 解析命令行参数:第一个参数是命令,后面可选 --port / -p
|
||||
CMD=""
|
||||
WEB_PORT="9000"
|
||||
WEB_PORT="${PLANTAI_WEB_PORT:-9000}"
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
start|stop|restart|status)
|
||||
@@ -33,12 +37,15 @@ done
|
||||
CMD="${CMD:-status}"
|
||||
|
||||
# Conda 环境:启动脚本内部也依赖这个环境
|
||||
CONDA_ENV="${CONDA_ENV:-/home/xzq/miniconda3/envs/test4py311}"
|
||||
CONDA_BASE="${BBAV5_CONDA_BASE:-/home/xia-lab/miniconda3}"
|
||||
DEFAULT_ENV="${BBAV5_DEFAULT_ENV:-test4py311}"
|
||||
CONDA_ENV="${CONDA_ENV:-$CONDA_BASE/envs/$DEFAULT_ENV}"
|
||||
export PATH="$CONDA_ENV/bin:$PATH"
|
||||
|
||||
# 项目配置(与 .env 和启动脚本保持一致)
|
||||
PROJECT_DIR="/home/xzq/5090/teamWorkspace/zh/PlantAI-dev"
|
||||
QUEUE_PREFIX="plantai_dev_9002"
|
||||
PROJECT_DIR="$SCRIPT_DIR"
|
||||
QUEUE_PREFIX="${BBAV5_QUEUE_PREFIX:-plantai_4090_9000}"
|
||||
REDIS_URL="${BBAV5_REDIS_URL:-redis://localhost:6379/10}"
|
||||
WEB_LOG="/tmp/plantai_${WEB_PORT}.log"
|
||||
CELERY_LOG="/tmp/celery_${WEB_PORT}.log"
|
||||
|
||||
@@ -46,28 +53,49 @@ red() { echo -e "\033[31m$*\033[0m"; }
|
||||
green(){ echo -e "\033[32m$*\033[0m"; }
|
||||
cyan() { echo -e "\033[36m$*\033[0m"; }
|
||||
|
||||
web_pid_for_port() {
|
||||
ss -tlnp 2>/dev/null \
|
||||
| grep ":$WEB_PORT " \
|
||||
| grep -oP 'pid=\K\d+' \
|
||||
| head -n 1 \
|
||||
|| true
|
||||
}
|
||||
|
||||
web_process_cwd() {
|
||||
readlink -f "/proc/$1/cwd" 2>/dev/null || true
|
||||
}
|
||||
|
||||
web_pid_belongs_to_project() {
|
||||
[ "$(web_process_cwd "$1")" = "$PROJECT_DIR" ]
|
||||
}
|
||||
|
||||
status() {
|
||||
echo ""
|
||||
cyan "=== PlantAI-dev ${WEB_PORT} 服务状态 ==="
|
||||
echo ""
|
||||
|
||||
# Web 服务
|
||||
WEB_PID=$(ss -tlnp 2>/dev/null | grep ":$WEB_PORT " | grep -oP 'pid=\K\d+' || true)
|
||||
WEB_PID="$(web_pid_for_port)"
|
||||
if [ -n "${WEB_PID:-}" ]; then
|
||||
green "[✓] Web 服务运行中 (端口 $WEB_PORT, PID $WEB_PID)"
|
||||
WEB_CWD="$(web_process_cwd "$WEB_PID")"
|
||||
if web_pid_belongs_to_project "$WEB_PID"; then
|
||||
green "[✓] Web 服务运行中 (端口 $WEB_PORT, PID $WEB_PID)"
|
||||
else
|
||||
red "[!] 端口 $WEB_PORT 被其他项目占用 (PID $WEB_PID, cwd ${WEB_CWD:-未知})"
|
||||
fi
|
||||
else
|
||||
red "[✗] Web 服务未启动 (端口 $WEB_PORT)"
|
||||
fi
|
||||
|
||||
# Redis
|
||||
if redis-cli ping > /dev/null 2>&1; then
|
||||
green "[✓] Redis 运行中 (localhost:6379)"
|
||||
if redis-cli -u "$REDIS_URL" ping > /dev/null 2>&1; then
|
||||
green "[✓] Redis 运行中 ($REDIS_URL)"
|
||||
else
|
||||
red "[✗] Redis 未启动"
|
||||
fi
|
||||
|
||||
# Celery workers
|
||||
CELERY_COUNT=$(pgrep -fc "$QUEUE_PREFIX" || echo 0)
|
||||
CELERY_COUNT="$(pgrep -fc "$QUEUE_PREFIX" || true)"
|
||||
if [ "$CELERY_COUNT" -gt 0 ]; then
|
||||
green "[✓] Celery workers 运行中 (${CELERY_COUNT} 进程, 队列前缀: $QUEUE_PREFIX)"
|
||||
echo " Worker 列表:"
|
||||
@@ -86,13 +114,18 @@ stop() {
|
||||
echo ""
|
||||
|
||||
# 停止 Web 服务
|
||||
WEB_PID=$(ss -tlnp 2>/dev/null | grep ":$WEB_PORT " | grep -oP 'pid=\K\d+' || true)
|
||||
WEB_PID="$(web_pid_for_port)"
|
||||
if [ -n "${WEB_PID:-}" ]; then
|
||||
WEB_CWD="$(web_process_cwd "$WEB_PID")"
|
||||
if ! web_pid_belongs_to_project "$WEB_PID"; then
|
||||
red "[!] 拒绝停止:端口 $WEB_PORT 属于其他项目 (PID $WEB_PID, cwd ${WEB_CWD:-未知})"
|
||||
return 1
|
||||
fi
|
||||
echo "正在停止 Web 服务 (PID $WEB_PID)..."
|
||||
kill "$WEB_PID" 2>/dev/null || true
|
||||
sleep 1
|
||||
# 如果还没死,强制 kill
|
||||
if ss -tlnp 2>/dev/null | grep -q ":$WEB_PORT"; then
|
||||
# 仅强制停止仍属于本项目的原进程,避免 PID 复用造成误杀。
|
||||
if kill -0 "$WEB_PID" 2>/dev/null && web_pid_belongs_to_project "$WEB_PID"; then
|
||||
kill -9 "$WEB_PID" 2>/dev/null || true
|
||||
fi
|
||||
green "[✓] Web 服务已停止"
|
||||
@@ -125,12 +158,18 @@ start() {
|
||||
echo ""
|
||||
|
||||
# 检查是否已在运行
|
||||
if ss -tlnp 2>/dev/null | grep -q ":$WEB_PORT"; then
|
||||
red "[!] 端口 $WEB_PORT 已被占用,请先停止服务"
|
||||
WEB_PID="$(web_pid_for_port)"
|
||||
if [ -n "${WEB_PID:-}" ]; then
|
||||
WEB_CWD="$(web_process_cwd "$WEB_PID")"
|
||||
if web_pid_belongs_to_project "$WEB_PID"; then
|
||||
red "[!] 本项目 Web 已在端口 $WEB_PORT 运行 (PID $WEB_PID)"
|
||||
else
|
||||
red "[!] 端口 $WEB_PORT 被其他项目占用 (PID $WEB_PID, cwd ${WEB_CWD:-未知})"
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! redis-cli ping > /dev/null 2>&1; then
|
||||
if ! redis-cli -u "$REDIS_URL" ping > /dev/null 2>&1; then
|
||||
red "[!] Redis 未运行,请先启动 Redis"
|
||||
exit 1
|
||||
fi
|
||||
@@ -141,7 +180,7 @@ start() {
|
||||
echo "正在启动 Celery workers..."
|
||||
nohup bash start_celery_9000.sh > "$CELERY_LOG" 2>&1 &
|
||||
sleep 3
|
||||
CELERY_COUNT=$(pgrep -fc "$QUEUE_PREFIX" || echo 0)
|
||||
CELERY_COUNT="$(pgrep -fc "$QUEUE_PREFIX" || true)"
|
||||
if [ "$CELERY_COUNT" -gt 0 ]; then
|
||||
green " [✓] Celery workers 已启动 (${CELERY_COUNT} 进程)"
|
||||
else
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
# 基因家族结果解读框架(鉴定到 PEP)
|
||||
|
||||
本文件是报告结构和语言规范,不是结果事实。所有数字、成员、分组和结论必须来自任务证据包。
|
||||
|
||||
## 1. 首页和一页式摘要
|
||||
|
||||
- 写明任务编号、家族、目标物种、参考物种、分析范围和报告时间。
|
||||
- 用 5—8 条短句回答:鉴定到多少成员、分类是否清楚、保守特征是什么、分布和复制有什么证据、调控预测有什么线索、蛋白有什么特点、最大风险是什么。
|
||||
- 明确“计算结果用于筛选线索,不能替代实验验证”。
|
||||
|
||||
## 2. 分析流程与完成情况
|
||||
|
||||
- 按真实执行顺序列出阶段、状态、关键产物和缺失项。
|
||||
- 区分成功、部分成功、失败和未运行。
|
||||
- 缺失阶段不能由相邻阶段补写。
|
||||
|
||||
## 3. 统一图解读方法
|
||||
|
||||
每个逻辑图必须有独立小节,依次回答:
|
||||
|
||||
1. 图用来回答什么问题。
|
||||
2. 本任务证据能确认什么。
|
||||
3. 用普通语言解释它可能意味着什么。
|
||||
4. 哪些细节仍需要打开原图或原始表复核。
|
||||
5. 这张图不能证明什么。
|
||||
|
||||
不要只介绍横轴、纵轴和颜色。首次出现专业词时立即解释,例如“motif(蛋白中较短、经常重复出现的保守片段)”。
|
||||
|
||||
## 4. 基因家族鉴定
|
||||
|
||||
- 输入蛋白总数、HMM 命中、BLAST 命中、候选数、最终成员数。
|
||||
- 对照成员 ID 文件和 PEP 序列数是否一致。
|
||||
- 说明成员筛选依据和被排除对象的原因。
|
||||
- 不能把全基因组蛋白数写成家族成员数。
|
||||
- 鉴定成功只说明找到了符合规则的成员,不能直接证明每个成员功能相同。
|
||||
|
||||
## 5. 系统发育和参考分类
|
||||
|
||||
- 解读普通树、环形树、多序列比对图、参考分类树和分类置信度表。
|
||||
- 说明树上靠近表示序列更相似,不等于功能完全相同。
|
||||
- 有分支支持率时说明分支稳定性;没有时明确写缺口。
|
||||
- 参考物种用于辅助命名和分类,不能替代目标物种实验。
|
||||
- subfamily confirm 要写清自动分类、人工确认和最终分类之间的关系。
|
||||
|
||||
## 6. Motif
|
||||
|
||||
- 解读 SeqLogo、motif 数量统计、排列图、树与 motif 组合图。
|
||||
- 说明同组成员是否共享相似片段,是否有成员出现缺失、增加或顺序变化。
|
||||
- motif 只提供结构和潜在功能线索,不能单独确定生物学功能。
|
||||
- 没有成员级位点表时,不得编造成员间具体差异。
|
||||
|
||||
## 7. Domain
|
||||
|
||||
- 解读主要功能区域统计、树与 domain 结构图、多区域或重复区域。
|
||||
- 重点核对显著命中数、覆盖成员数和主要类型。
|
||||
- manifest 与原始结果冲突时,以风险提示为准,不写成确定结论。
|
||||
- domain 完整性支持蛋白具备某类功能基础,但不等于功能已经验证。
|
||||
|
||||
## 8. 基因结构
|
||||
|
||||
- 解读外显子/内含子结构图、树与结构组合图。
|
||||
- 比较同组成员的结构是否大体相似,标记超长或异常成员。
|
||||
- 结构差异可能提示分化,也可能来自注释问题,必须保留两种可能。
|
||||
|
||||
## 9. 综合组合图
|
||||
|
||||
- 对树、motif、domain、基因结构同图展示进行联合解释。
|
||||
- 判断分类与结构证据是否一致。
|
||||
- 指出证据冲突成员和需要复核的成员。
|
||||
- 组合图整合的是多类计算证据,不是功能实验结果。
|
||||
|
||||
## 10. 启动子元件
|
||||
|
||||
- 解读元件总量、类别分布、基因×元件热图、位置分布、富集、共现和复杂度图。
|
||||
- TATA、CAAT 等核心元件高频出现不能单独写成胁迫或激素响应。
|
||||
- ABRE、MYB/MYC、LTR 等只能写成潜在调控线索。
|
||||
- 没有表达数据时,不能写成基因已经响应某种处理。
|
||||
|
||||
## 11. TFBS
|
||||
|
||||
- 解读高频 motif 柱状图、基因×motif 热图、全部 motif 热图、位置分布、位置密度、启动子结构、TF—基因网络和 motif 来源图。
|
||||
- 说明预测到的是可能结合的位置,不是实际结合实验。
|
||||
- 网络中的连线表示预测关系,不代表直接调控已经发生。
|
||||
- 每张图都应写明数据是否齐全以及成员级解释边界。
|
||||
|
||||
## 12. 种内共线性
|
||||
|
||||
- 解读 Circos 图、种内共线性图和复制类型统计。
|
||||
- 只有检测到涉及家族成员的基因对时,才能讨论家族复制。
|
||||
- 若家族成员基因对为 0,必须明确写“本任务未获得支持片段复制驱动扩张的证据”。
|
||||
- 全基因组共线性数量不能当作家族共线性数量。
|
||||
|
||||
## 13. 染色体分布
|
||||
|
||||
- 解读染色体定位图、聚集分布和染色体偏好。
|
||||
- 必须先确认实际定位到多少家族成员。
|
||||
- 若定位数为 0 或存在高风险,明确说明该图当前不能用于染色体分布结论,并建议检查基因 ID 与注释版本。
|
||||
|
||||
## 14. 种间共线性和 Ortholog
|
||||
|
||||
- 解读多物种关系树、环形树、多物种共线性图、标记版图和两两物种图。
|
||||
- 说明目标物种与参考物种之间是否找到可信对应关系。
|
||||
- ortholog 表示跨物种较可能对应的基因,不等于功能完全相同。
|
||||
- 未获得 Ka/Ks 时,不得推断选择压力。
|
||||
|
||||
## 15. PEP 蛋白特征
|
||||
|
||||
- 解读蛋白长度、分子量、等电点、亲疏水性、跨膜条带图、跨膜汇总图和亚细胞定位图。
|
||||
- 标记长度异常、跨膜区域多、定位特殊或序列提取失败的成员。
|
||||
- 预测结果用来筛选特殊成员,不能替代蛋白定位或功能实验。
|
||||
- 说明 PEP 序列数是否与最终成员数一致。
|
||||
|
||||
## 16. 多证据综合和候选成员
|
||||
|
||||
至少使用以下证据建立候选成员表:
|
||||
|
||||
- 鉴定证据是否完整。
|
||||
- 分类是否稳定。
|
||||
- motif/domain/结构是否与同组成员一致或具有特殊性。
|
||||
- 启动子和 TFBS 是否提供合理线索。
|
||||
- 种内或种间保守关系是否有本地证据。
|
||||
- 蛋白特征是否完整、是否存在异常。
|
||||
- 是否有表达或实验数据支持;没有时明确写“本任务未提供”。
|
||||
|
||||
候选成员分为高、中、低优先级。评分只用于整理证据,不能解释为功能概率。
|
||||
|
||||
## 17. 主要限制
|
||||
|
||||
- 数据库和注释版本限制。
|
||||
- 参考物种推断限制。
|
||||
- 树分支支持、motif/domain 统计或成员级表缺失。
|
||||
- 染色体定位、共线性或 ID 映射失败。
|
||||
- 启动子、TFBS、定位和跨膜均属于预测。
|
||||
- 缺少表达、遗传和功能实验时,不能形成因果结论。
|
||||
|
||||
## 18. 下一步验证建议
|
||||
|
||||
按优先级给出可执行建议,例如:
|
||||
|
||||
1. 修复高风险的数据或 ID 映射问题。
|
||||
2. 核对候选成员序列和基因注释。
|
||||
3. 用表达数据检查候选成员在相关组织或处理下是否变化。
|
||||
4. 进行亚细胞定位、蛋白互作或启动子验证。
|
||||
5. 对最高优先级成员开展过表达、敲除或互补实验。
|
||||
|
||||
## 19. 报告语言红线
|
||||
|
||||
- “证明”改为“支持”或“提供线索”。
|
||||
- “调控”改为“可能参与调控”,除非有直接实验。
|
||||
- “导致”改为“与……同时出现”或“可能有关”。
|
||||
- “该家族成员全部具有某功能”改为“部分成员具有相关结构或预测线索”。
|
||||
- 无结果不等于没有生物学现象,只能写“本任务在当前数据和方法下未检测到”。
|
||||
|
||||
## 20. 报告结尾
|
||||
|
||||
- 给出综合证据矩阵。
|
||||
- 列出 3—10 个优先候选成员;证据不足时可以不强行列候选。
|
||||
- 给出“读者最需要记住的五句话”。
|
||||
- 最后一段再次说明:候选结论需要后续实验验证。
|
||||
@@ -0,0 +1,43 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
QA_SERVICE_ROOT = PROJECT_ROOT / "services" / "qa_v1_1_73"
|
||||
if str(QA_SERVICE_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(QA_SERVICE_ROOT))
|
||||
|
||||
from src.rag.qa_v1_1_73 import __version__
|
||||
from src.rag.qa_v1_1_73.api import router
|
||||
from src.rag.qa_v1_1_73.backend_guard import validate_backends
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="Plant Biology QA V1.1.73",
|
||||
version=__version__,
|
||||
description=(
|
||||
"Production full-graph QA with assertion-anchored Qdrant recall, "
|
||||
"bounded canonical Neo4j expansion, BGE reranking, and citation validation."
|
||||
),
|
||||
)
|
||||
app.include_router(router)
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
def validate_production_backends() -> None:
|
||||
app.state.kb_backend_identity = validate_backends()
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {
|
||||
"status": "ok",
|
||||
"service": "qa_v1_1_73",
|
||||
"version": __version__,
|
||||
"source": "plantai_git_repository",
|
||||
"backend": app.state.kb_backend_identity,
|
||||
}
|
||||
Executable
+84
@@ -0,0 +1,84 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
QA_SERVICE_ROOT="$PROJECT_ROOT/services/qa_v1_1_73"
|
||||
APP_MODULE="scripts.qa_v1_1_73_server:app"
|
||||
RUNTIME_ENV="${QA_V1_RUNTIME_ENV:-/home/xzq/5090/PlanAI-cooperate/RAG/runtime/qa_v1_1_73.env}"
|
||||
PYTHON="${QA_V1_PYTHON:-/home/xzq/miniconda3/envs/gfpy312/bin/python}"
|
||||
PORT="${QA_V1_1_73_PORT:-7920}"
|
||||
RUNTIME_DIR="${QA_V1_RUNTIME_DIR:-$PROJECT_ROOT/.runtime/qa_v1_1_73}"
|
||||
LOG="$RUNTIME_DIR/qa_$PORT.log"
|
||||
PIDFILE="$RUNTIME_DIR/qa_$PORT.pid"
|
||||
|
||||
mkdir -p "$RUNTIME_DIR"
|
||||
|
||||
if [[ ! -x "$PYTHON" ]]; then
|
||||
echo "ERROR: Python interpreter is missing: $PYTHON" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -r "$RUNTIME_ENV" ]]; then
|
||||
echo "ERROR: Runtime environment is missing: $RUNTIME_ENV" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -f "$QA_SERVICE_ROOT/src/rag/qa_v1_1_73/api.py" ]]; then
|
||||
echo "ERROR: Versioned QA source is missing from $QA_SERVICE_ROOT" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ss -ltn | awk '{print $4}' | grep -qE ":$PORT$"; then
|
||||
echo "ERROR: Port $PORT is already in use." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
set -a
|
||||
source "$RUNTIME_ENV"
|
||||
set +a
|
||||
|
||||
export GENE_FAMILY_ROOT="$QA_SERVICE_ROOT"
|
||||
export PYTHONPATH="$QA_SERVICE_ROOT:$PROJECT_ROOT"
|
||||
export NEO4J_URI="bolt://127.0.0.1:18788"
|
||||
export NEO4J_USER="${NEO4J_USER:-neo4j}"
|
||||
export QA_V1_GRAPH_SCHEMA="canonical"
|
||||
export V15_QDRANT_URL="http://127.0.0.1:26333"
|
||||
export V15_QDRANT_COLLECTION="plant_srt_direct_v1_bge_m3_20260702"
|
||||
export QA_V1_EMBED_URL="http://127.0.0.1:11434/api/embed"
|
||||
export QA_V1_EMBED_MODEL="bge-m3:567m"
|
||||
export QA_V1_BGE_RERANKER_MODEL="/home/xzq/5090/PlantAI-rag/bge-reranker-base"
|
||||
export QA_V1_BGE_DEVICE="cuda"
|
||||
export QA_PAPER_METADATA_REGISTRY="/home/xzq/5090/PlantAI-dev4090-7/gene-family-mas-v2_qa_v1_1_73_current/data/knowledge_base/paper_metadata_registry_v1_20260717/paper_metadata_registry_full.jsonl"
|
||||
export QA_V1_ENABLE_PROVISIONAL_DRAFT="0"
|
||||
export QA_V1_MAX_ROUNDS="1"
|
||||
export QA_V1_ENABLE_AGENTIC_CONTROLLER="1"
|
||||
export QA_V1_AGENTIC_MAX_REPLANS="1"
|
||||
export QA_V1_ALLOW_AGENTIC_REVIEW_FALLBACK="1"
|
||||
export QA_V1_ENABLE_AGENTIC_HOP_GATE="1"
|
||||
export QA_V1_AGENTIC_HOP_GATE_MAX_HOPS="3"
|
||||
export QA_ENABLE_QUESTION_DECOMPOSITION="1"
|
||||
export QA_V1_SUBQUESTION_WORKERS="3"
|
||||
export QA_V1_VECTOR_QUERY_WORKERS="2"
|
||||
export QA_V1_BOUNDED_MAX_FRONTIER="24"
|
||||
export QA_V1_BOUNDED_PER_NODE="12"
|
||||
export QA_V1_BOUNDED_MAX_HOPS="3"
|
||||
export QA_V1_BOUNDED_NEO4J_TIMEOUT="8"
|
||||
export QA_V1_BOUNDED_QDRANT_TIMEOUT="30"
|
||||
export QA_V1_BOUNDED_EMBED_TIMEOUT="30"
|
||||
export LD_LIBRARY_PATH="/home/xzq/miniconda3/envs/gfpy312/lib:${LD_LIBRARY_PATH-}"
|
||||
|
||||
nohup "$PYTHON" -m uvicorn "$APP_MODULE" \
|
||||
--app-dir "$PROJECT_ROOT" \
|
||||
--host 127.0.0.1 \
|
||||
--port "$PORT" > "$LOG" 2>&1 < /dev/null &
|
||||
echo $! > "$PIDFILE"
|
||||
|
||||
for _ in $(seq 1 90); do
|
||||
if curl --max-time 2 -fsS "http://127.0.0.1:$PORT/health" >/dev/null 2>&1; then
|
||||
echo "QA service started on http://127.0.0.1:$PORT"
|
||||
echo "PID file: $PIDFILE"
|
||||
echo "Log: $LOG"
|
||||
exit 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
echo "ERROR: QA service failed to become healthy. See $LOG" >&2
|
||||
exit 1
|
||||
Executable
+52
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SIDECAR_PORT="${QA_V1_SIDECAR_PORT:-7945}"
|
||||
EVIDENCE_DIR="$PROJECT_ROOT/.runtime/qa_v1_1_73/formal_sidecar_$SIDECAR_PORT"
|
||||
QUESTION="In which species was tissue-specific expression of CAMTA3 (SR1) examined?"
|
||||
|
||||
mkdir -p "$EVIDENCE_DIR"
|
||||
|
||||
QA_V1_1_73_PORT="$SIDECAR_PORT" \
|
||||
QA_V1_RUNTIME_DIR="$EVIDENCE_DIR" \
|
||||
bash "$PROJECT_ROOT/scripts/start_qa_v1_1_73_7920.sh" \
|
||||
> "$EVIDENCE_DIR/launch.txt" 2>&1
|
||||
|
||||
curl -fsS --max-time 15 \
|
||||
"http://127.0.0.1:$SIDECAR_PORT/health" \
|
||||
> "$EVIDENCE_DIR/health.json"
|
||||
curl -fsS --max-time 15 \
|
||||
"http://127.0.0.1:$SIDECAR_PORT/api/v1/qa-v1-1-73/info" \
|
||||
> "$EVIDENCE_DIR/info.json"
|
||||
curl -sS --max-time 150 \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"question\":\"$QUESTION\",\"debug\":false,\"use_llm\":true,\"chain_top_k\":3,\"chain_hops\":2,\"chain_max_edges\":12,\"chain_max_nodes\":10,\"native_chain_enabled\":true,\"native_chain_max_hops\":3,\"native_chain_top_k\":5}" \
|
||||
-o "$EVIDENCE_DIR/response.json" \
|
||||
-w "status=%{http_code} total=%{time_total}s bytes=%{size_download}\\n" \
|
||||
"http://127.0.0.1:$SIDECAR_PORT/api/v1/qa-v1-1-73/chat" \
|
||||
> "$EVIDENCE_DIR/timing.txt"
|
||||
|
||||
"${QA_V1_PYTHON:-/home/xzq/miniconda3/envs/gfpy312/bin/python}" - "$EVIDENCE_DIR" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
evidence_dir = Path(sys.argv[1])
|
||||
health = json.loads((evidence_dir / "health.json").read_text(encoding="utf-8"))
|
||||
info = json.loads((evidence_dir / "info.json").read_text(encoding="utf-8"))
|
||||
response = json.loads((evidence_dir / "response.json").read_text(encoding="utf-8"))
|
||||
timing = (evidence_dir / "timing.txt").read_text(encoding="utf-8")
|
||||
|
||||
assert health["status"] == "ok"
|
||||
assert health["source"] == "plantai_git_repository"
|
||||
assert "sandbox" not in json.dumps(health).lower()
|
||||
assert "sandbox" not in json.dumps(info).lower()
|
||||
assert info["config"]["graph_schema"] == "canonical"
|
||||
assert "status=200" in timing
|
||||
assert response.get("success", True) is True
|
||||
assert "Solanum lycopersicum" in response.get("answer", "")
|
||||
assert response.get("native_graph_chains")
|
||||
|
||||
(evidence_dir / "PASS").write_text("sidecar validation passed\n", encoding="utf-8")
|
||||
PY
|
||||
@@ -0,0 +1,159 @@
|
||||
Arabidopsis thaliana:
|
||||
type: species
|
||||
aliases:
|
||||
- Arabidopsis thaliana
|
||||
- A. thaliana
|
||||
- Arabidopsis
|
||||
- 拟南芥
|
||||
forbidden_expansions:
|
||||
- Brassicaceae
|
||||
- cruciferous plants
|
||||
- 十字花科
|
||||
|
||||
Oryza sativa:
|
||||
type: species
|
||||
aliases:
|
||||
- Oryza sativa
|
||||
- O. sativa
|
||||
- rice
|
||||
- 水稻
|
||||
|
||||
Glycine max:
|
||||
type: species
|
||||
aliases:
|
||||
- Glycine max
|
||||
- G. max
|
||||
- soybean
|
||||
- soybean plant
|
||||
- 大豆
|
||||
|
||||
Zea mays:
|
||||
type: species
|
||||
aliases:
|
||||
- Zea mays
|
||||
- Z. mays
|
||||
- maize
|
||||
- corn
|
||||
- 玉米
|
||||
|
||||
Litchi chinensis:
|
||||
type: species
|
||||
aliases:
|
||||
- Litchi chinensis
|
||||
- L. chinensis
|
||||
- litchi
|
||||
- lychee
|
||||
- 荔枝
|
||||
|
||||
Triticum aestivum:
|
||||
type: species
|
||||
retrieval_label: wheat
|
||||
aliases:
|
||||
- Triticum aestivum
|
||||
- T. aestivum
|
||||
- wheat
|
||||
- bread wheat
|
||||
- common wheat
|
||||
- 小麦
|
||||
|
||||
Solanum lycopersicum:
|
||||
type: species
|
||||
aliases:
|
||||
- Solanum lycopersicum
|
||||
- S. lycopersicum
|
||||
- tomato
|
||||
- 番茄
|
||||
|
||||
Solanum melongena:
|
||||
type: species
|
||||
aliases:
|
||||
- Solanum melongena
|
||||
- S. melongena
|
||||
- eggplant
|
||||
- brinjal
|
||||
- 茄子
|
||||
|
||||
Solanum tuberosum:
|
||||
type: species
|
||||
aliases:
|
||||
- Solanum tuberosum
|
||||
- S. tuberosum
|
||||
- potato
|
||||
- 马铃薯
|
||||
- 土豆
|
||||
|
||||
Ipomoea batatas:
|
||||
type: species
|
||||
aliases:
|
||||
- Ipomoea batatas
|
||||
- I. batatas
|
||||
- sweet potato
|
||||
- sweetpotato
|
||||
- 甘薯
|
||||
- 红薯
|
||||
forbidden_expansions:
|
||||
- Solanum tuberosum
|
||||
- S. tuberosum
|
||||
- potato
|
||||
- 马铃薯
|
||||
- 土豆
|
||||
|
||||
GmP450 genes:
|
||||
type: gene_family
|
||||
aliases:
|
||||
- GmP450 genes
|
||||
- GmP450 gene family
|
||||
- GmP450
|
||||
- soybean P450 genes
|
||||
- soybean cytochrome P450 genes
|
||||
|
||||
SmCYP450 genes:
|
||||
type: gene_family
|
||||
aliases:
|
||||
- SmCYP450 genes
|
||||
- SmCYP450 gene family
|
||||
- SmCYP450
|
||||
- eggplant CYP450 genes
|
||||
- eggplant cytochrome P450 genes
|
||||
|
||||
SPL gene family:
|
||||
type: gene_family
|
||||
retrieval_label: SPL genes
|
||||
aliases:
|
||||
- SPL gene family
|
||||
- SPL 基因家族
|
||||
- SPL genes
|
||||
- SPLs
|
||||
- SPL
|
||||
- SBP genes
|
||||
- SBP-box genes
|
||||
- Squamosa promoter-binding protein-like genes
|
||||
- SQUAMOSA promoter binding protein-like genes
|
||||
|
||||
flowering:
|
||||
type: biological_process
|
||||
retrieval_terms:
|
||||
- ovary
|
||||
- anthesis
|
||||
- expression
|
||||
aliases:
|
||||
- flowering
|
||||
- flowering time
|
||||
- floral transition
|
||||
- 开花
|
||||
|
||||
GST genes:
|
||||
type: gene_family
|
||||
aliases:
|
||||
- GST genes
|
||||
- GST gene family
|
||||
- GSTs
|
||||
- GST
|
||||
- glutathione S-transferase genes
|
||||
- glutathione S-transferase gene family
|
||||
- glutathione S-transferases
|
||||
- glutathione S-transferase
|
||||
- 谷胱甘肽S-转移酶
|
||||
- 谷胱甘肽 S-转移酶
|
||||
- 谷胱甘肽S-转移酶基因
|
||||
- 谷胱甘肽S-转移酶基因家族
|
||||
@@ -0,0 +1,200 @@
|
||||
expression:
|
||||
location:
|
||||
- expressed_in
|
||||
- highly_expressed_in
|
||||
- predominantly_expressed_in
|
||||
- accumulated_at_higher_level_in
|
||||
- detected_in
|
||||
- enriched_in
|
||||
up:
|
||||
- upregulated_in
|
||||
- induced_under
|
||||
- induced_in
|
||||
- increased_in
|
||||
- increased_under
|
||||
- enhanced_expression_in
|
||||
- accumulated_at_higher_level_in
|
||||
down:
|
||||
- downregulated_in
|
||||
- decreased_in
|
||||
- decreased_under
|
||||
- reduced_in
|
||||
- repressed_in
|
||||
condition:
|
||||
- differentially_expressed_under
|
||||
- expression_changed_under
|
||||
- expression_changed_in
|
||||
- responds_to
|
||||
- responsive_to
|
||||
- not_induced_under
|
||||
|
||||
catalog_count:
|
||||
- has_member_count
|
||||
- identified_in
|
||||
- systematically_identified_in
|
||||
- genome_wide_identified_in
|
||||
- were_identified_in
|
||||
- was_identified_in
|
||||
- identified
|
||||
- were identified in
|
||||
- was identified in
|
||||
- identified in
|
||||
- contains
|
||||
- contain
|
||||
- encodes
|
||||
- include
|
||||
- includes
|
||||
- had
|
||||
- has_number_of_members
|
||||
- comprehensively_analyzed_in
|
||||
- analyzed_in
|
||||
- surveyed_in
|
||||
|
||||
classification:
|
||||
- belongs_to
|
||||
- member_of
|
||||
- classified_into
|
||||
- divided_into
|
||||
- separated_into
|
||||
- grouped_into
|
||||
- clustered_with
|
||||
|
||||
localization:
|
||||
- localizes_to
|
||||
- located_in
|
||||
- predicted_to_localize_to
|
||||
- subcellular_localized_to
|
||||
|
||||
structure:
|
||||
- contains_domain
|
||||
- contains_motif
|
||||
- contains_cis_element
|
||||
- has_intron_count
|
||||
- has_exon_count
|
||||
- has_conserved_domain
|
||||
- lacks_binding_site_for
|
||||
|
||||
function_effect:
|
||||
- promotes
|
||||
- inhibits
|
||||
- enhances
|
||||
- reduces
|
||||
- affects
|
||||
- regulates
|
||||
- involved_in
|
||||
- participates_in
|
||||
- contributes_to
|
||||
- associated_with
|
||||
|
||||
perturbation_effect:
|
||||
- overexpression_increases
|
||||
- overexpression_enhances
|
||||
- overexpression_promotes
|
||||
- overexpression_causes
|
||||
- knockdown_reduces
|
||||
- knockdown_enhances
|
||||
- silencing_reduces
|
||||
- mutation_affects
|
||||
|
||||
method:
|
||||
- used_tool
|
||||
- analyzed_by
|
||||
- predicted_by
|
||||
- aligned_with
|
||||
- constructed_with
|
||||
- identified_by
|
||||
|
||||
dataset:
|
||||
- has_accession
|
||||
- available_at
|
||||
- has_data_type
|
||||
- from_database
|
||||
- from_article
|
||||
- used_for
|
||||
|
||||
prior_work:
|
||||
- reported_in
|
||||
- previously_reported_in
|
||||
- cited_as
|
||||
- known_to
|
||||
|
||||
stress_response:
|
||||
- responds_to
|
||||
- responsive_to
|
||||
- induced_by
|
||||
- induced_under
|
||||
- induced_in
|
||||
- upregulated_under
|
||||
- upregulated_in
|
||||
- involved_in_stress_response
|
||||
- associated_with_stress_tolerance
|
||||
- confers_tolerance_to
|
||||
- enhances_tolerance_to
|
||||
- improves_tolerance_to
|
||||
- stress tolerance
|
||||
- abiotic stress
|
||||
- biotic stress
|
||||
|
||||
detoxification:
|
||||
- involved_in_detoxification
|
||||
- detoxifies
|
||||
- detoxification
|
||||
- conjugates_with_glutathione
|
||||
- catalyzes_glutathione_conjugation
|
||||
- scavenges
|
||||
- removes
|
||||
- clears
|
||||
- reduces_ros
|
||||
- maintains_redox_homeostasis
|
||||
- glutathione conjugation
|
||||
- xenobiotic detoxification
|
||||
|
||||
oxidative_stress:
|
||||
- scavenges_ros
|
||||
- scavenges_h2o2
|
||||
- reduces_h2o2
|
||||
- regulates_redox_homeostasis
|
||||
- enhances_oxidative_stress_tolerance
|
||||
- involved_in_antioxidant_response
|
||||
- reactive oxygen species
|
||||
- oxidative stress
|
||||
- redox homeostasis
|
||||
|
||||
functional_validation:
|
||||
- overexpression_enhances
|
||||
- overexpression_increases
|
||||
- overexpression_confers
|
||||
- overexpression_improves
|
||||
- overexpression_promotes
|
||||
- over-expression
|
||||
- overexpression
|
||||
- co_expression_enhances
|
||||
- co-expression
|
||||
- transgenic
|
||||
- mutation_affects
|
||||
- knockdown_reduces
|
||||
- silencing_reduces
|
||||
|
||||
family_expansion:
|
||||
- expanded_by
|
||||
- tandemly_duplicated
|
||||
- segmentally_duplicated
|
||||
- retained_after_wgd
|
||||
- underwent_positive_selection
|
||||
- functionally_diverged
|
||||
- supergene family
|
||||
- gene family expansion
|
||||
- tandem duplication
|
||||
- segmental duplication
|
||||
- positive selection
|
||||
|
||||
signaling_regulation:
|
||||
- upregulated_by
|
||||
- activated_by
|
||||
- regulated_by
|
||||
- downstream_of
|
||||
- target_of
|
||||
- ABA signaling
|
||||
- signal transduction
|
||||
- kinase
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Versioned source package for the PlantAI full-graph QA service."""
|
||||
@@ -0,0 +1 @@
|
||||
"""RAG modules vendored for the PlantAI full-graph QA service."""
|
||||
@@ -0,0 +1,2 @@
|
||||
"""Dual-KB taxonomy-aware retrieval helpers."""
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
class EntityAliasNormalizer:
|
||||
def __init__(self, path: str | Path | None = None):
|
||||
self.path = Path(path) if path else None
|
||||
self.entries: Dict[str, Dict[str, Any]] = {}
|
||||
self.alias_to_canonical: Dict[str, str] = {}
|
||||
if self.path and self.path.exists():
|
||||
self._load(self.path)
|
||||
|
||||
def _load(self, path: Path) -> None:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
|
||||
for canonical, payload in data.items():
|
||||
if not isinstance(payload, dict):
|
||||
payload = {"aliases": payload if isinstance(payload, list) else []}
|
||||
aliases = [str(x) for x in payload.get("aliases", []) if str(x).strip()]
|
||||
aliases.append(str(canonical))
|
||||
entry = {
|
||||
"canonical": str(canonical),
|
||||
"type": str(payload.get("type", "")),
|
||||
"aliases": sorted(set(aliases), key=len, reverse=True),
|
||||
"forbidden_expansions": [str(x) for x in payload.get("forbidden_expansions", [])],
|
||||
}
|
||||
self.entries[str(canonical)] = entry
|
||||
for alias in entry["aliases"]:
|
||||
self.alias_to_canonical[self._norm(alias)] = str(canonical)
|
||||
|
||||
@staticmethod
|
||||
def _norm(text: str) -> str:
|
||||
return re.sub(r"\s+", " ", str(text).strip().lower())
|
||||
|
||||
def normalize_mentions(self, question: str, mentions: List[str]) -> Dict[str, Any]:
|
||||
normalized: List[Dict[str, Any]] = []
|
||||
seen = set()
|
||||
hay = self._norm(question)
|
||||
for mention in mentions:
|
||||
m_norm = self._norm(mention)
|
||||
canonical = self.alias_to_canonical.get(m_norm, mention)
|
||||
key = self._norm(canonical)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
entry = self.entries.get(canonical, {})
|
||||
normalized.append({
|
||||
"mention": mention,
|
||||
"canonical": canonical,
|
||||
"type": entry.get("type", ""),
|
||||
"matched_alias": m_norm in self.alias_to_canonical,
|
||||
"forbidden_expansions": entry.get("forbidden_expansions", []),
|
||||
})
|
||||
# Also detect aliases directly in the question.
|
||||
for alias_norm, canonical in self.alias_to_canonical.items():
|
||||
if alias_norm in hay and self._norm(canonical) not in seen:
|
||||
seen.add(self._norm(canonical))
|
||||
entry = self.entries.get(canonical, {})
|
||||
normalized.append({
|
||||
"mention": alias_norm,
|
||||
"canonical": canonical,
|
||||
"type": entry.get("type", ""),
|
||||
"matched_alias": True,
|
||||
"forbidden_expansions": entry.get("forbidden_expansions", []),
|
||||
})
|
||||
return {"entities": normalized, "config_path": str(self.path or "")}
|
||||
|
||||
def forbidden_hits(self, question: str, row_text: str, normalized_entities: List[Dict[str, Any]]) -> List[str]:
|
||||
q = self._norm(question)
|
||||
text = self._norm(row_text)
|
||||
hits: List[str] = []
|
||||
for ent in normalized_entities:
|
||||
# Only enforce forbidden expansions when the canonical/alias appears in the question.
|
||||
canonical = self._norm(ent.get("canonical", ""))
|
||||
mention = self._norm(ent.get("mention", ""))
|
||||
if canonical not in q and mention not in q:
|
||||
continue
|
||||
for bad in ent.get("forbidden_expansions", []) or []:
|
||||
bad_norm = self._norm(bad)
|
||||
if bad_norm and bad_norm in text:
|
||||
hits.append(bad)
|
||||
return hits
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
"""Production QA V1.1.73 assertion-anchored graph expansion."""
|
||||
|
||||
__version__ = "1.1.73-assertion-anchored"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,361 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Tuple
|
||||
|
||||
from neo4j import GraphDatabase
|
||||
|
||||
from src.rag.dual_kb_taxonomy_rag.entity_alias import EntityAliasNormalizer
|
||||
|
||||
from .schemas import EvidenceItem
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return str(value or "").strip()
|
||||
|
||||
|
||||
def _norm(value: Any) -> str:
|
||||
return " ".join(_text(value).lower().replace("_", " ").split())
|
||||
|
||||
|
||||
def _type(value: Any) -> str:
|
||||
return _norm(value).replace(" ", "_")
|
||||
|
||||
|
||||
def _pmcid(value: Any) -> str:
|
||||
match = re.search(r"\bPMC\d+", _text(value), re.IGNORECASE)
|
||||
return match.group(0).upper() if match else ""
|
||||
|
||||
|
||||
class AgenticGraphRetriever:
|
||||
def __init__(
|
||||
self,
|
||||
driver: Any | None = None,
|
||||
*,
|
||||
graph_schema: str | None = None,
|
||||
alias_path: str | Path | None = None,
|
||||
):
|
||||
self._owns_driver = driver is None
|
||||
self.graph_schema = (
|
||||
graph_schema or os.environ.get("QA_V1_GRAPH_SCHEMA", "canonical")
|
||||
).strip().lower()
|
||||
resolved_alias_path = alias_path or (
|
||||
Path(__file__).resolve().parents[3] / "configs/rag/entity_alias.yaml"
|
||||
)
|
||||
self.aliaser = EntityAliasNormalizer(resolved_alias_path)
|
||||
self.driver = driver or GraphDatabase.driver(
|
||||
os.environ.get("NEO4J_URI", "bolt://127.0.0.1:18788"),
|
||||
auth=(
|
||||
os.environ.get("NEO4J_USER", "neo4j"),
|
||||
os.environ.get("NEO4J_PASSWORD", ""),
|
||||
),
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
if self._owns_driver and self.driver is not None:
|
||||
self.driver.close()
|
||||
|
||||
def retrieve(
|
||||
self, plan: Dict[str, Any]
|
||||
) -> Tuple[List[EvidenceItem], List[Dict[str, Any]], Dict[str, Any]]:
|
||||
anchors = [item for item in plan.get("anchors") or [] if isinstance(item, dict)]
|
||||
anchor_ids = [
|
||||
_text(item.get("canonical_id"))
|
||||
for item in anchors
|
||||
if _text(item.get("canonical_id")).startswith("cent_")
|
||||
]
|
||||
anchor_names = []
|
||||
for item in anchors:
|
||||
for value in (item.get("canonical_name"), item.get("mention")):
|
||||
normalized = _norm(value)
|
||||
if normalized and normalized not in anchor_names:
|
||||
anchor_names.append(normalized)
|
||||
canonical_name = _text(item.get("canonical_name"))
|
||||
alias_entry = self.aliaser.entries.get(canonical_name) or {}
|
||||
for value in alias_entry.get("aliases") or []:
|
||||
normalized = _norm(value)
|
||||
if normalized and normalized not in anchor_names:
|
||||
anchor_names.append(normalized)
|
||||
|
||||
relation_plan = plan.get("relation_plan") if isinstance(plan.get("relation_plan"), dict) else {}
|
||||
graph_plan = plan.get("graph_plan") if isinstance(plan.get("graph_plan"), dict) else {}
|
||||
allow_incoming = bool(relation_plan.get("allow_incoming", True))
|
||||
allow_outgoing = bool(relation_plan.get("allow_outgoing", True))
|
||||
terminal_types = {
|
||||
_type(value)
|
||||
for value in (
|
||||
graph_plan.get("required_terminal_types")
|
||||
or [
|
||||
entity_type
|
||||
for target in plan.get("answer_targets") or []
|
||||
for entity_type in target.get("allowed_entity_types") or []
|
||||
]
|
||||
)
|
||||
if _type(value)
|
||||
}
|
||||
terminal_type_wildcard = "unknown" in terminal_types
|
||||
# Planner uses "unknown" to mean that no terminal type is constrained.
|
||||
terminal_types.discard("unknown")
|
||||
relation_categories = {
|
||||
_norm(value)
|
||||
for value in relation_plan.get("allowed_relation_categories") or []
|
||||
if _norm(value)
|
||||
}
|
||||
audit: Dict[str, Any] = {
|
||||
"enabled": bool(graph_plan.get("enabled", True)),
|
||||
"anchor_ids": anchor_ids,
|
||||
"anchor_names": anchor_names,
|
||||
"allow_incoming": allow_incoming,
|
||||
"allow_outgoing": allow_outgoing,
|
||||
"required_terminal_types": sorted(terminal_types),
|
||||
"terminal_type_wildcard": terminal_type_wildcard,
|
||||
"max_hops": max(1, min(int(graph_plan.get("max_hops", 1)), 3)),
|
||||
"query_mode": (
|
||||
"legacy_anchor_neighborhood"
|
||||
if self.graph_schema == "legacy"
|
||||
else "canonical_anchor_neighborhood"
|
||||
),
|
||||
"preserve_assertion_direction": True,
|
||||
"filtered_reasons": {},
|
||||
}
|
||||
if not audit["enabled"]:
|
||||
audit["status"] = "disabled_by_plan"
|
||||
return [], [], audit
|
||||
if not anchor_ids and not anchor_names:
|
||||
audit["status"] = "no_verified_anchor"
|
||||
return [], [], audit
|
||||
if not allow_incoming and not allow_outgoing:
|
||||
audit["status"] = "no_direction_enabled"
|
||||
return [], [], audit
|
||||
|
||||
rows = self.driver.execute_query(
|
||||
self._statement(),
|
||||
parameters_={
|
||||
"anchor_ids": anchor_ids,
|
||||
"anchor_names": anchor_names,
|
||||
"limit": 100 if self.graph_schema == "legacy" else 300,
|
||||
"layers": [
|
||||
"main_kg",
|
||||
"background_relations",
|
||||
"method_relations",
|
||||
"methodology_relations",
|
||||
"dataset_metadata",
|
||||
],
|
||||
},
|
||||
result_transformer_=lambda result: [dict(record) for record in result],
|
||||
)
|
||||
rows = list(rows or [])
|
||||
evidence: List[EvidenceItem] = []
|
||||
chains: List[Dict[str, Any]] = []
|
||||
seen_assertions = set()
|
||||
|
||||
def filtered(reason: str) -> None:
|
||||
counts = audit["filtered_reasons"]
|
||||
counts[reason] = int(counts.get(reason) or 0) + 1
|
||||
|
||||
for row in rows:
|
||||
assertion_id = _text(row.get("assertion_id"))
|
||||
if not assertion_id:
|
||||
filtered("missing_assertion_id")
|
||||
continue
|
||||
if assertion_id in seen_assertions:
|
||||
filtered("duplicate_assertion")
|
||||
continue
|
||||
direction = _norm(row.get("direction"))
|
||||
if (direction == "incoming" and not allow_incoming) or (
|
||||
direction == "outgoing" and not allow_outgoing
|
||||
):
|
||||
filtered("direction_disabled")
|
||||
continue
|
||||
if direction not in {"incoming", "outgoing"}:
|
||||
filtered("unknown_direction")
|
||||
continue
|
||||
|
||||
answer_entity = row.get("source") if direction == "incoming" else row.get("target")
|
||||
answer_type = row.get("source_type") if direction == "incoming" else row.get("target_type")
|
||||
if terminal_types and _type(answer_type) not in terminal_types:
|
||||
filtered("terminal_type_mismatch")
|
||||
continue
|
||||
category = _norm(row.get("relation_category"))
|
||||
if relation_categories and category not in relation_categories:
|
||||
filtered("relation_category_mismatch")
|
||||
continue
|
||||
|
||||
seen_assertions.add(assertion_id)
|
||||
anchor_name = _text(row.get("anchor_name"))
|
||||
paper_id = _text(row.get("paper_id"))
|
||||
pmcid = _text(row.get("pmcid")) or _pmcid(paper_id)
|
||||
raw = {
|
||||
"assertion_id": assertion_id,
|
||||
"source_type": _text(row.get("source_type")),
|
||||
"target_type": _text(row.get("target_type")),
|
||||
"source_canonical_id": _text(row.get("source_canonical_id")),
|
||||
"target_canonical_id": _text(row.get("target_canonical_id")),
|
||||
"relation_category": _text(row.get("relation_category")),
|
||||
"navigation_direction": direction,
|
||||
"anchor_id": _text(row.get("anchor_id")),
|
||||
"anchor_name": anchor_name,
|
||||
"answer_entity": _text(answer_entity),
|
||||
"answer_entity_type": _type(answer_type),
|
||||
"preserved_assertion_direction": True,
|
||||
}
|
||||
item = EvidenceItem(
|
||||
evidence_id="",
|
||||
kb="V16/V15",
|
||||
source=_text(row.get("source")),
|
||||
relation=_text(row.get("relation")),
|
||||
target=_text(row.get("target")),
|
||||
evidence=_text(row.get("evidence")),
|
||||
paper_id=paper_id,
|
||||
title=_text(row.get("title")),
|
||||
pmid=_text(row.get("pmid")),
|
||||
pmcid=pmcid,
|
||||
doi=_text(row.get("doi")),
|
||||
species=_text(row.get("species")),
|
||||
output_layer="canonical_graph",
|
||||
relation_id=assertion_id,
|
||||
retrieval_methods=["agentic_anchor_graph", direction],
|
||||
final_score=float(row.get("best_score") or 0.0),
|
||||
raw=raw,
|
||||
)
|
||||
evidence.append(item)
|
||||
hop = {
|
||||
"hop_index": 1,
|
||||
"from_id": _text(row.get("source_canonical_id")),
|
||||
"from_name": _text(row.get("source")),
|
||||
"to_id": _text(row.get("target_canonical_id")),
|
||||
"to_name": _text(row.get("target")),
|
||||
"source": _text(row.get("source")),
|
||||
"source_type": _text(row.get("source_type")),
|
||||
"source_canonical_id": _text(row.get("source_canonical_id")),
|
||||
"relation": _text(row.get("relation")),
|
||||
"relation_category": _text(row.get("relation_category")),
|
||||
"target": _text(row.get("target")),
|
||||
"target_type": _text(row.get("target_type")),
|
||||
"target_canonical_id": _text(row.get("target_canonical_id")),
|
||||
"assertion_id": assertion_id,
|
||||
"paper_id": item.paper_id,
|
||||
"title": item.title,
|
||||
"pmid": item.pmid,
|
||||
"pmcid": item.pmcid,
|
||||
"doi": item.doi,
|
||||
"species": item.species,
|
||||
"evidence": item.evidence,
|
||||
"support_count": int(row.get("support_count") or 0),
|
||||
"paper_count": int(row.get("paper_count") or 0),
|
||||
"best_score": float(row.get("best_score") or 0.0),
|
||||
"navigation_direction": direction,
|
||||
"preserved_assertion_direction": True,
|
||||
}
|
||||
chains.append(
|
||||
{
|
||||
"chain_id": f"AG{len(chains) + 1}",
|
||||
"chain_source": "agentic_anchor_graph",
|
||||
"hop_count": 1,
|
||||
"path_score": float(row.get("best_score") or 0.0),
|
||||
"start": anchor_name,
|
||||
"end": _text(answer_entity),
|
||||
"anchor_id": _text(row.get("anchor_id")),
|
||||
"anchor_name": anchor_name,
|
||||
"answer_entity": _text(answer_entity),
|
||||
"answer_entity_type": _type(answer_type),
|
||||
"navigation_direction": direction,
|
||||
"preserve_assertion_direction": True,
|
||||
"paper_id": item.paper_id,
|
||||
"pmid": item.pmid,
|
||||
"pmcid": item.pmcid,
|
||||
"doi": item.doi,
|
||||
"hops": [hop],
|
||||
"score_breakdown": {
|
||||
"anchor_entity_match": 1.0,
|
||||
"answer_type_match": 1.0,
|
||||
"direction_match": 1.0,
|
||||
"final_score": float(row.get("best_score") or 0.0),
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
audit["raw_assertion_count"] = len(rows)
|
||||
audit["returned_assertion_count"] = len(evidence)
|
||||
audit["returned_chain_count"] = len(chains)
|
||||
audit["status"] = "ok" if evidence else "no_matching_assertions"
|
||||
return evidence, chains, audit
|
||||
|
||||
def _statement(self) -> str:
|
||||
if self.graph_schema == "legacy":
|
||||
return """
|
||||
MATCH (anchor:Entity)
|
||||
WHERE toLower(anchor.name) IN $anchor_names
|
||||
OR toLower(coalesce(anchor.name_norm, anchor.name)) IN $anchor_names
|
||||
WITH anchor LIMIT 24
|
||||
MATCH (anchor)-[r:RELATION]-(neighbor:Entity)
|
||||
WHERE r.output_layer IN $layers
|
||||
WITH anchor, r, startNode(r) AS source_node, endNode(r) AS target_node,
|
||||
CASE
|
||||
WHEN endNode(r) = anchor THEN 'incoming'
|
||||
ELSE 'outgoing'
|
||||
END AS direction
|
||||
RETURN coalesce(anchor.entity_id, anchor.id, '') AS anchor_id,
|
||||
anchor.name AS anchor_name,
|
||||
direction,
|
||||
r.relation_id AS assertion_id,
|
||||
source_node.name AS source,
|
||||
source_node.entity_type AS source_type,
|
||||
coalesce(source_node.entity_id, source_node.id, '') AS source_canonical_id,
|
||||
coalesce(r.relation_norm, r.relation) AS relation,
|
||||
'' AS relation_category,
|
||||
target_node.name AS target,
|
||||
target_node.entity_type AS target_type,
|
||||
coalesce(target_node.entity_id, target_node.id, '') AS target_canonical_id,
|
||||
r.evidence AS evidence,
|
||||
coalesce(r.canonical_paper_id, r.paper_id, '') AS paper_id,
|
||||
'' AS title,
|
||||
coalesce(r.pubmed_id, '') AS pmid,
|
||||
'' AS pmcid,
|
||||
'' AS doi,
|
||||
coalesce(r.species, '') AS species,
|
||||
1 AS support_count,
|
||||
1 AS paper_count,
|
||||
coalesce(r.confidence, 0.0) AS best_score
|
||||
ORDER BY CASE WHEN coalesce(r.evidence, '') = '' THEN 1 ELSE 0 END,
|
||||
coalesce(r.confidence, 0.0) DESC
|
||||
LIMIT $limit
|
||||
"""
|
||||
return """
|
||||
MATCH (a:SRT2CanonicalAssertion)
|
||||
WHERE a.source_canonical_id IN $anchor_ids
|
||||
OR a.target_canonical_id IN $anchor_ids
|
||||
OR toLower(a.source) IN $anchor_names
|
||||
OR toLower(a.target) IN $anchor_names
|
||||
WITH a,
|
||||
CASE
|
||||
WHEN a.target_canonical_id IN $anchor_ids OR toLower(a.target) IN $anchor_names
|
||||
THEN 'incoming'
|
||||
ELSE 'outgoing'
|
||||
END AS direction,
|
||||
CASE
|
||||
WHEN a.target_canonical_id IN $anchor_ids OR toLower(a.target) IN $anchor_names
|
||||
THEN a.target_canonical_id
|
||||
ELSE a.source_canonical_id
|
||||
END AS anchor_id,
|
||||
CASE
|
||||
WHEN a.target_canonical_id IN $anchor_ids OR toLower(a.target) IN $anchor_names
|
||||
THEN a.target
|
||||
ELSE a.source
|
||||
END AS anchor_name
|
||||
RETURN anchor_id, anchor_name, direction,
|
||||
a.assertion_id AS assertion_id,
|
||||
a.source AS source, a.source_type AS source_type,
|
||||
a.source_canonical_id AS source_canonical_id,
|
||||
a.relation_norm AS relation, a.relation_category AS relation_category,
|
||||
a.target AS target, a.target_type AS target_type,
|
||||
a.target_canonical_id AS target_canonical_id,
|
||||
a.best_evidence AS evidence, a.best_paper_id AS paper_id,
|
||||
a.title AS title, a.pmid AS pmid, a.pmcid AS pmcid, a.doi AS doi,
|
||||
'' AS species, a.support_count AS support_count,
|
||||
a.paper_count AS paper_count, a.best_score AS best_score
|
||||
ORDER BY a.paper_count DESC, a.support_count DESC, a.best_score DESC
|
||||
LIMIT $limit
|
||||
"""
|
||||
@@ -0,0 +1,251 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from typing import Any, Dict, Iterable, List, Sequence
|
||||
|
||||
from .llm_client import LLMClient
|
||||
from .schemas import RetrievalHypothesis
|
||||
|
||||
|
||||
class AgenticHopGateError(RuntimeError):
|
||||
"""Raised when the Agent cannot produce a valid hop decision."""
|
||||
|
||||
|
||||
class AgenticHopGate:
|
||||
"""Let an Agent decide whether verified graph paths need another hop.
|
||||
|
||||
The gate never queries Neo4j and never invents a path. It only reviews a
|
||||
compact representation of paths already verified by the graph retriever.
|
||||
Python retains operational safeguards such as the maximum hop count and
|
||||
cycle prevention, but does not infer the semantic hop count.
|
||||
"""
|
||||
|
||||
SYSTEM = """You are the hop-control Agent for a scientific GraphRAG system.
|
||||
|
||||
Decide whether the CURRENT VERIFIED GRAPH PATHS already support a complete
|
||||
answer to the ORIGINAL QUESTION. Do not predict a hop count before retrieval.
|
||||
Do not answer the question. Do not use outside biological knowledge.
|
||||
|
||||
Choose exactly one action:
|
||||
- stop_answerable: the current paths cover every requested answer slot.
|
||||
- continue: the current paths are useful but at least one answer slot remains
|
||||
unresolved; select only the path IDs whose terminal entities should be
|
||||
expanded by one additional graph hop.
|
||||
- stop_insufficient: the current paths do not support a complete answer and
|
||||
none of their terminal entities provides a defensible next frontier.
|
||||
|
||||
Treat entity spelling or harmless surface-form differences as aliases when the
|
||||
provided path itself makes the identity clear. Preserve relation direction,
|
||||
species, treatment, genotype, tissue, time, negation, and evidence scope.
|
||||
Similarity by itself is not proof. Return JSON only."""
|
||||
|
||||
def __init__(self, llm: LLMClient, *, max_paths: int = 12) -> None:
|
||||
self.llm = llm
|
||||
self.max_paths = max(4, min(int(max_paths or 18), 24))
|
||||
|
||||
def decide(
|
||||
self,
|
||||
question: str,
|
||||
hypothesis: RetrievalHypothesis,
|
||||
paths: Sequence[Any],
|
||||
*,
|
||||
current_hop: int,
|
||||
safety_max_hops: int,
|
||||
task_context: Dict[str, Any] | None = None,
|
||||
) -> Dict[str, Any]:
|
||||
started = time.perf_counter()
|
||||
compact_paths = self._compact_paths(paths)
|
||||
if not compact_paths:
|
||||
raise AgenticHopGateError("hop gate received no verified paths")
|
||||
|
||||
payload = {
|
||||
"original_question": str(question or "").strip(),
|
||||
"question_type": str(
|
||||
(task_context or {}).get("question_type")
|
||||
or hypothesis.question_type
|
||||
or ""
|
||||
),
|
||||
"requested_answer_slots": list(hypothesis.answer_slots or []),
|
||||
"task_answer_slot": str((task_context or {}).get("answer_slot") or ""),
|
||||
"task_constraints": [
|
||||
str(value)
|
||||
for value in ((task_context or {}).get("constraints") or [])
|
||||
if str(value).strip()
|
||||
][:12],
|
||||
"current_hop": int(current_hop),
|
||||
"safety_max_hops": int(safety_max_hops),
|
||||
"can_expand_further": int(current_hop) < int(safety_max_hops),
|
||||
"verified_paths": compact_paths,
|
||||
"required_output_schema": {
|
||||
"action": "stop_answerable|continue|stop_insufficient",
|
||||
"answer_sufficient": False,
|
||||
"covered_answer_slots": ["slot already supported"],
|
||||
"unresolved_answer_slots": ["slot still missing"],
|
||||
"expand_path_ids": ["H1P01"],
|
||||
"next_relation_intents": ["relation concept to seek"],
|
||||
"reason": "brief evidence-grounded decision",
|
||||
},
|
||||
}
|
||||
|
||||
attempts: List[Dict[str, Any]] = []
|
||||
value: Dict[str, Any] | None = None
|
||||
error = ""
|
||||
for attempt in range(2):
|
||||
request_payload = dict(payload)
|
||||
if attempt:
|
||||
request_payload["format_repair"] = (
|
||||
"The previous response was malformed or internally inconsistent. "
|
||||
"Return one complete JSON object matching required_output_schema."
|
||||
)
|
||||
request_payload["previous_error"] = error
|
||||
raw = self.llm.complete_json(
|
||||
self.SYSTEM,
|
||||
json.dumps(request_payload, ensure_ascii=False),
|
||||
max_tokens=5000,
|
||||
)
|
||||
try:
|
||||
value = self._validate(raw, compact_paths, current_hop, safety_max_hops)
|
||||
attempts.append({"attempt": attempt + 1, "valid": True})
|
||||
break
|
||||
except Exception as exc:
|
||||
error = f"{type(exc).__name__}: {exc}"
|
||||
attempts.append(
|
||||
{
|
||||
"attempt": attempt + 1,
|
||||
"valid": False,
|
||||
"error": error[:500],
|
||||
}
|
||||
)
|
||||
value = None
|
||||
|
||||
if value is None:
|
||||
raise AgenticHopGateError(
|
||||
f"Agent hop decision invalid after {len(attempts)} attempts: {error}"
|
||||
)
|
||||
|
||||
id_to_index = {
|
||||
str(item["path_id"]): int(item["path_index"])
|
||||
for item in compact_paths
|
||||
}
|
||||
value["expand_path_indexes"] = [
|
||||
id_to_index[path_id] for path_id in value["expand_path_ids"]
|
||||
]
|
||||
value["current_hop"] = int(current_hop)
|
||||
value["safety_max_hops"] = int(safety_max_hops)
|
||||
value["candidate_path_count"] = len(compact_paths)
|
||||
value["json_attempts"] = attempts
|
||||
value["elapsed_seconds"] = time.perf_counter() - started
|
||||
value["decision_source"] = "llm_agent_after_verified_hop"
|
||||
return value
|
||||
|
||||
@staticmethod
|
||||
def _validate(
|
||||
value: Any,
|
||||
compact_paths: Sequence[Dict[str, Any]],
|
||||
current_hop: int,
|
||||
safety_max_hops: int,
|
||||
) -> Dict[str, Any]:
|
||||
if not isinstance(value, dict) or value.get("_json_parse_error"):
|
||||
raw_head = (
|
||||
str(value.get("_raw_text_head") or "")[:300]
|
||||
if isinstance(value, dict)
|
||||
else ""
|
||||
)
|
||||
raise ValueError(
|
||||
"response is not a JSON object"
|
||||
+ (f"; raw_head={raw_head!r}" if raw_head else "; raw_head is empty")
|
||||
)
|
||||
action = str(value.get("action") or "").strip().lower()
|
||||
if action not in {"stop_answerable", "continue", "stop_insufficient"}:
|
||||
raise ValueError("action is missing or unsupported")
|
||||
sufficient = value.get("answer_sufficient")
|
||||
if not isinstance(sufficient, bool):
|
||||
raise ValueError("answer_sufficient must be a JSON boolean")
|
||||
if action == "stop_answerable" and not sufficient:
|
||||
raise ValueError("stop_answerable requires answer_sufficient=true")
|
||||
if action in {"continue", "stop_insufficient"} and sufficient:
|
||||
raise ValueError(f"{action} requires answer_sufficient=false")
|
||||
|
||||
valid_ids = {str(item.get("path_id") or "") for item in compact_paths}
|
||||
expand_ids = AgenticHopGate._dedupe_strings(value.get("expand_path_ids") or [])
|
||||
unknown_ids = [path_id for path_id in expand_ids if path_id not in valid_ids]
|
||||
if unknown_ids:
|
||||
raise ValueError(f"expand_path_ids contains unknown IDs: {unknown_ids}")
|
||||
if action == "continue":
|
||||
if int(current_hop) >= int(safety_max_hops):
|
||||
raise ValueError("continue is forbidden at the safety hop limit")
|
||||
if not expand_ids:
|
||||
raise ValueError("continue requires at least one verified expand_path_id")
|
||||
elif expand_ids:
|
||||
raise ValueError(f"{action} must not include expand_path_ids")
|
||||
|
||||
reason = str(value.get("reason") or "").strip()
|
||||
if not reason:
|
||||
raise ValueError("reason is required")
|
||||
return {
|
||||
"action": action,
|
||||
"answer_sufficient": sufficient,
|
||||
"covered_answer_slots": AgenticHopGate._dedupe_strings(
|
||||
value.get("covered_answer_slots") or []
|
||||
)[:12],
|
||||
"unresolved_answer_slots": AgenticHopGate._dedupe_strings(
|
||||
value.get("unresolved_answer_slots") or []
|
||||
)[:12],
|
||||
"expand_path_ids": expand_ids,
|
||||
"next_relation_intents": AgenticHopGate._dedupe_strings(
|
||||
value.get("next_relation_intents") or []
|
||||
)[:12],
|
||||
"reason": reason[:1000],
|
||||
}
|
||||
|
||||
def _compact_paths(self, paths: Sequence[Any]) -> List[Dict[str, Any]]:
|
||||
ranked = sorted(
|
||||
enumerate(paths),
|
||||
key=lambda pair: float(getattr(pair[1], "path_score", 0.0) or 0.0),
|
||||
reverse=True,
|
||||
)[: self.max_paths]
|
||||
output: List[Dict[str, Any]] = []
|
||||
for display_index, (path_index, path) in enumerate(ranked, start=1):
|
||||
hops = []
|
||||
for hop in list(getattr(path, "hops", []) or []):
|
||||
evidence = " ".join(str(getattr(hop, "evidence", "") or "").split())
|
||||
hops.append(
|
||||
{
|
||||
"source": str(getattr(hop, "source", "") or ""),
|
||||
"relation": str(getattr(hop, "relation", "") or ""),
|
||||
"target": str(getattr(hop, "target", "") or ""),
|
||||
"source_type": str(getattr(hop, "source_type", "") or ""),
|
||||
"target_type": str(getattr(hop, "target_type", "") or ""),
|
||||
"paper_id": str(getattr(hop, "paper_id", "") or ""),
|
||||
"evidence": evidence[:260],
|
||||
}
|
||||
)
|
||||
output.append(
|
||||
{
|
||||
"path_id": f"H{len(hops)}P{display_index:02d}",
|
||||
"path_index": path_index,
|
||||
"path_score": round(
|
||||
float(getattr(path, "path_score", 0.0) or 0.0), 6
|
||||
),
|
||||
"terminal_entity": (
|
||||
str(getattr(path.hops[-1], "target", "") or "")
|
||||
if getattr(path, "hops", None)
|
||||
else ""
|
||||
),
|
||||
"hops": hops,
|
||||
}
|
||||
)
|
||||
return output
|
||||
|
||||
@staticmethod
|
||||
def _dedupe_strings(values: Iterable[Any]) -> List[str]:
|
||||
output: List[str] = []
|
||||
seen: set[str] = set()
|
||||
for value in values:
|
||||
text = str(value or "").strip()
|
||||
key = text.casefold()
|
||||
if text and key not in seen:
|
||||
seen.add(key)
|
||||
output.append(text)
|
||||
return output
|
||||
@@ -0,0 +1,273 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from .evidence_pack import format_evidence_pack
|
||||
from .llm_client import LLMClient
|
||||
from .schemas import EvidenceItem, RetrievalHypothesis
|
||||
|
||||
|
||||
class AnswerComposer:
|
||||
SYSTEM = """You are a helpful plant-biology QA assistant. Your task is to answer the user's question using only the provided data report.
|
||||
|
||||
Use a natural explanatory style. Do not answer with a table unless the user explicitly asks for a table.
|
||||
|
||||
Write the answer as substantive bullet points or short paragraphs. Each point should state one supported finding, explain its biological meaning in relation to the user's question, and cite the supporting records.
|
||||
|
||||
Do not merely list SRT triples. Synthesize the provided SRTs and evidence into readable biological statements, but do not add unsupported claims.
|
||||
|
||||
Use all supplied evidence that directly answers the question or materially completes an explanation. Combine redundant records into one finding with shared citations rather than repeating them. Omit evidence that is wholly unrelated; do not mention or cite unrelated records merely to maximize citation coverage.
|
||||
Before writing, silently perform this final-answer consistency check using only
|
||||
the supplied Data Report, GraphChains, and Agent-approved subquestion groups:
|
||||
1. List each directly supported answer slot and ensure that it is stated once
|
||||
in the answer.
|
||||
2. Never call a requested fact unsupported, unavailable, absent, or
|
||||
insufficient when a supplied record directly states that fact.
|
||||
3. For a supported multi-hop GraphChain, follow the continuous chain through
|
||||
its final requested terminal. Do not stop at a bridge entity when the next
|
||||
supplied hop answers the question.
|
||||
4. When only part of a question is supported, answer every supported part
|
||||
first, then limit any uncertainty statement strictly to the unsupported
|
||||
part.
|
||||
|
||||
These are evidence-use checks, not a license to infer missing facts. Do not
|
||||
invent a bridge, relation, or terminal that is not present in the supplied
|
||||
report.
|
||||
|
||||
|
||||
For broad family-function, survey, or open-ended questions, begin with a concise overview and then organize the answer by the distinct functions supported by the evidence. For each function, explain the richest supported process chain available: the relevant context or trigger when present, the function, any intermediate biological process, and the resulting biological outcome. This process-level explanation is valuable even when no specific gene member or molecular target is available.
|
||||
|
||||
For a broad gene-family survey, consolidate the evidence into roughly five to eight major functional or mechanistic themes when the records support that many. Prefer one to three of the most relevant citations per theme and merge redundant family-level records. A complete Chinese answer should normally remain within about 900 to 1500 Chinese characters; an English answer should normally remain within about 500 to 900 words. These are synthesis targets, not reasons to omit a distinct supported mechanism or a directly requested biological process.
|
||||
|
||||
Specific genes, targets, binding sites, and experimental methods are not required unless the question asks for them. When supplied evidence contains useful examples, include them selectively to deepen the explanation without changing a family-level question into an unnecessary gene-by-gene catalogue.
|
||||
|
||||
Match causal language to the evidence. Do not turn a process-level graph chain, association, prediction, or co-occurrence into a direct molecular mechanism. State direct activation, repression, binding, or causation only when the supplied records support it.
|
||||
|
||||
Calibrate answer strength to the evidence level. Expression patterns, association, co-expression, enrichment, and computational prediction support involvement or a candidate role; they do not by themselves support "major", "core", "primary", or causal regulation. Use such strong terms only when the original evidence explicitly establishes them or reports functional perturbation evidence such as a knockout, knockdown, overexpression, complementation, or a directly measured phenotype.
|
||||
|
||||
When the question asks which genes are major but the supplied records contain only expression, association, developmental-context, or heterologous evidence, begin by stating that the evidence does not establish a major regulator. Then list the supported candidates by evidence type. Never call expression evidence direct functional evidence. A heterologous experiment may support a cross-species candidate function, but it is not native-species functional or causal proof.
|
||||
|
||||
Keep developmental scope exact. Evidence for spike development or spikelet development does not automatically establish direct flowering-time regulation. Describe the supported organ or developmental process, and mention flowering-time control only when the evidence directly measures or functionally perturbs flowering time.
|
||||
|
||||
If several records support the same idea, group them into one point and cite the most relevant records. Do not repeat the same idea multiple times.
|
||||
|
||||
If the evidence is partial, say exactly what is supported and what is not supported. If the provided data do not support an answer, say that the provided data are insufficient.
|
||||
|
||||
Preserve species, gene names, host species, tissue, developmental stage, treatment, time point, genotype, negation, quantity, phenotype, and effect direction. Do not merge same-named genes across species.
|
||||
|
||||
Citation rule:
|
||||
Every factual claim based on the report must use only the exact Preferred citation strings supplied with its supporting records. Never invent, alter, infer, or substitute an identifier. Do not expose internal source IDs, graph hop IDs, or dataset-style audit citations in ordinary answer prose.
|
||||
|
||||
Omit factual claims whose Preferred citation is unavailable.
|
||||
|
||||
Do not write paper titles in the answer. Titles are reference-card metadata, not answer prose.
|
||||
|
||||
When the same evidence context supports it, write a concrete statement containing species + gene or family member + supported function or process. Also include supported condition, direction, quantity, mechanism, or phenotype when present. These details are conditional: do not require or invent a specific gene or species when the evidence supports only a family-level statement.
|
||||
|
||||
Claim species are evidence-level facts. Paper species are study context only. Do not promote paper species to claim species. When only Paper species is supplied, phrase it as a study of that species rather than asserting that the SRT finding is species-specific.
|
||||
|
||||
Group redundant findings, but retain every distinct Preferred citation that supports the grouped statement. Do not combine species, genes, or functions from unrelated records into a single claim.
|
||||
|
||||
Do not use unsupported general knowledge. By default, prefer not to use general knowledge.
|
||||
"""
|
||||
|
||||
def __init__(self, llm: LLMClient):
|
||||
self.llm = llm
|
||||
|
||||
def compose(
|
||||
self,
|
||||
question: str,
|
||||
hypothesis: RetrievalHypothesis,
|
||||
evidence: List[EvidenceItem],
|
||||
assessment: Dict[str, Any],
|
||||
claim_verification: List[Dict[str, Any]],
|
||||
graph_chain_context: str = "",
|
||||
correction_note: str = "",
|
||||
agent_handoff: Dict[str, Any] | None = None,
|
||||
subquestion_evidence_groups: List[Dict[str, Any]] | None = None,
|
||||
) -> str:
|
||||
context_data = self._build_context_data(evidence, graph_chain_context)
|
||||
allowed_evidence_markers = {
|
||||
marker
|
||||
for item in evidence
|
||||
for marker in (
|
||||
str(item.evidence_id or "").strip(),
|
||||
str(item.relation_id or "").strip(),
|
||||
str((item.raw or {}).get("assertion_id") or "").strip(),
|
||||
)
|
||||
if marker
|
||||
}
|
||||
allowed_chain_ids = set(
|
||||
re.findall(r"^Chain summary\s+(\S+)", graph_chain_context, flags=re.MULTILINE)
|
||||
)
|
||||
subquestion_block = self._build_subquestion_context(
|
||||
subquestion_evidence_groups or [],
|
||||
allowed_evidence_markers=allowed_evidence_markers,
|
||||
allowed_chain_ids=allowed_chain_ids,
|
||||
)
|
||||
correction_block = (
|
||||
f"--- Citation correction required ---\n\n{correction_note}\n\n"
|
||||
if correction_note
|
||||
else ""
|
||||
)
|
||||
handoff_block = (
|
||||
"--- Agent-approved final handoff ---\n\n"
|
||||
+ json.dumps(agent_handoff, ensure_ascii=False, indent=2)
|
||||
+ "\n\nUse only the IDs and constraints in this handoff. Do not mention rejected candidates.\n\n"
|
||||
if agent_handoff
|
||||
else ""
|
||||
)
|
||||
return self.llm.complete(
|
||||
self.SYSTEM,
|
||||
f"""--- Data Report ---
|
||||
|
||||
{context_data}
|
||||
|
||||
--- Retrieval Notes ---
|
||||
|
||||
Question type:
|
||||
{hypothesis.question_type}
|
||||
|
||||
Possible answer hints:
|
||||
{hypothesis.answer_slots}
|
||||
|
||||
Evidence coverage assessment:
|
||||
{assessment}
|
||||
|
||||
Claim verification:
|
||||
{claim_verification}
|
||||
|
||||
These retrieval notes are only hints for selecting relevant records. Do not structure the answer around them mechanically.
|
||||
|
||||
{handoff_block}{subquestion_block}--- Multi-hop Path Rule ---
|
||||
|
||||
When GraphChains are supplied, prefer a continuous path that covers the entities and relations stated in the question in their given order.
|
||||
Answer with the terminal object reached by the final relation asked in the question.
|
||||
Entities already stated in the question are known bridge entities: do not stop at a known bridge when a supported next hop answers the question.
|
||||
Do not combine hops unless the target canonical entity of one hop is the source canonical entity of the next hop.
|
||||
|
||||
--- Answer Style ---
|
||||
|
||||
Answer in the same language as the user's question unless the user requests another language.
|
||||
|
||||
Preserve gene symbols and Latin species names exactly while translating explanatory prose into the user's language.
|
||||
|
||||
Use natural explanatory bullet points or short paragraphs.
|
||||
|
||||
Do not use a table unless the user explicitly asks for one.
|
||||
|
||||
Begin with the direct answer to the user's question. For broad function or survey questions, give a concise overview before the detailed evidence-supported functions.
|
||||
|
||||
Then explain as many distinct, relevant findings as the supplied evidence supports; do not impose a fixed 1-4 point limit. Each point should interpret the evidence in relation to the question, not merely restate a triple.
|
||||
|
||||
Use all records that directly support the answer or materially complete a biological process explanation. Merge duplicate support into shared citations and omit wholly unrelated records. Use only the exact Preferred citation strings printed in the Data Report.
|
||||
|
||||
For each major function, connect the function to intermediate biological processes and outcomes whenever the evidence permits. Specific gene-level details are optional unless requested. Never infer a direct molecular mechanism from a process-level or associative chain alone.
|
||||
|
||||
If the records contain related but not directly answering evidence, mention it only if it helps clarify the answer, and clearly state that it is indirect.
|
||||
|
||||
If the provided data are insufficient, say so directly.
|
||||
|
||||
{correction_block}
|
||||
Now answer the following question:
|
||||
|
||||
{question}
|
||||
""",
|
||||
temperature=0.1,
|
||||
max_tokens=12000,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _build_subquestion_context(
|
||||
groups: List[Dict[str, Any]],
|
||||
*,
|
||||
allowed_evidence_markers: set[str],
|
||||
allowed_chain_ids: set[str],
|
||||
) -> str:
|
||||
if not groups:
|
||||
return ""
|
||||
sections = [
|
||||
"--- Agent-approved subquestion evidence groups ---",
|
||||
"",
|
||||
(
|
||||
"Synthesize these groups around the original question; do not mechanically "
|
||||
"repeat the subquestions. Use only evidence and chains shown in these groups."
|
||||
),
|
||||
(
|
||||
"For mechanism questions, organize supported explanations as: "
|
||||
"trigger/context → molecular action → protected target or process → biological outcome. "
|
||||
"Do not treat expression under stress or a bare tolerance statement as the mechanism. "
|
||||
"Do not add an intermediate step such as binding, degradation, signaling, or refolding "
|
||||
"unless that exact step is explicitly supported by an accepted record in the group."
|
||||
),
|
||||
]
|
||||
for group in groups:
|
||||
sections.extend(
|
||||
[
|
||||
"",
|
||||
f"Group {group.get('subquestion_id') or '?'}",
|
||||
f"Subquestion: {group.get('question') or ''}",
|
||||
f"Role: {group.get('role') or 'supporting'}",
|
||||
f"Answer slot: {group.get('answer_slot') or ''}",
|
||||
f"Status: {group.get('status') or 'EMPTY'}",
|
||||
]
|
||||
)
|
||||
evidence = [
|
||||
item
|
||||
for item in group.get("evidence") or []
|
||||
if isinstance(item, EvidenceItem)
|
||||
]
|
||||
evidence_references: List[str] = []
|
||||
for item in evidence:
|
||||
markers = {
|
||||
str(item.evidence_id or "").strip(),
|
||||
str(item.relation_id or "").strip(),
|
||||
str((item.raw or {}).get("assertion_id") or "").strip(),
|
||||
}
|
||||
markers.discard("")
|
||||
if not markers.intersection(allowed_evidence_markers):
|
||||
continue
|
||||
reference = (
|
||||
str(item.relation_id or "").strip()
|
||||
or str((item.raw or {}).get("assertion_id") or "").strip()
|
||||
or str(item.evidence_id or "").strip()
|
||||
)
|
||||
evidence_references.append(
|
||||
f"- {reference}: {item.source} -- {item.relation} --> {item.target}"
|
||||
)
|
||||
if evidence_references:
|
||||
sections.append(
|
||||
"Accepted evidence references (full records appear once in the Data Report):\n"
|
||||
+ "\n".join(evidence_references)
|
||||
)
|
||||
else:
|
||||
sections.append("Accepted evidence references: none")
|
||||
chain_ids = [
|
||||
str(chain.get("chain_id") or "").strip()
|
||||
for chain in group.get("chains") or []
|
||||
if isinstance(chain, dict)
|
||||
and str(chain.get("chain_id") or "").strip() in allowed_chain_ids
|
||||
]
|
||||
if chain_ids:
|
||||
sections.append(
|
||||
"Accepted chain references (full chains appear once in GraphChains): "
|
||||
+ ", ".join(dict.fromkeys(chain_ids))
|
||||
)
|
||||
else:
|
||||
sections.append("Accepted chain references: none")
|
||||
return "\n".join(sections).strip() + "\n\n"
|
||||
|
||||
@staticmethod
|
||||
def _build_context_data(evidence: List[EvidenceItem], graph_chain_context: str) -> str:
|
||||
sections: List[str] = []
|
||||
if evidence:
|
||||
sections.append("Dataset: Sources\n" + format_evidence_pack(evidence))
|
||||
else:
|
||||
sections.append("Dataset: Sources\nNo separate top-k SRT evidence was supplied for this answer.")
|
||||
if graph_chain_context:
|
||||
sections.append("Dataset: GraphChains\n" + graph_chain_context)
|
||||
else:
|
||||
sections.append("Dataset: GraphChains\nNo graph knowledge chain was available.")
|
||||
return "\n\n".join(sections).strip()
|
||||
@@ -0,0 +1,313 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import queue
|
||||
import threading
|
||||
from functools import lru_cache
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from . import __version__
|
||||
from .config import QAConfig
|
||||
from .pipeline import QAAnswerError, QAV1Pipeline
|
||||
|
||||
|
||||
class QARequest(BaseModel):
|
||||
question: str = Field(min_length=1)
|
||||
debug: bool = False
|
||||
use_provisional_draft: Optional[bool] = None
|
||||
max_rounds: Optional[int] = Field(default=None, ge=1, le=2)
|
||||
ablation: Optional[Dict[str, Any]] = None
|
||||
use_llm: Optional[bool] = None
|
||||
chain_top_k: Optional[int] = Field(default=3, ge=0, le=20)
|
||||
chain_hops: Optional[int] = Field(default=2, ge=1, le=2)
|
||||
chain_max_edges: Optional[int] = Field(default=12, ge=0, le=120)
|
||||
chain_max_nodes: Optional[int] = Field(default=10, ge=0, le=80)
|
||||
chain_mode: Optional[str] = Field(default="display")
|
||||
chain_min_score: Optional[float] = Field(default=0.45, ge=0.0, le=1.0)
|
||||
native_chain_enabled: bool = True
|
||||
native_chain_max_hops: int = Field(default=3, ge=1, le=3)
|
||||
native_chain_top_k: int = Field(default=5, ge=0, le=20)
|
||||
native_chain_min_hops: int = Field(default=1, ge=1, le=3)
|
||||
native_chain_beam_width: int = Field(default=60, ge=1, le=300)
|
||||
native_chain_per_node_expand: int = Field(default=12, ge=1, le=50)
|
||||
endpoint_path_enabled: bool = False
|
||||
endpoint_path_max_hops: int = Field(default=3, ge=1, le=3)
|
||||
endpoint_path_top_k: int = Field(default=5, ge=0, le=10)
|
||||
endpoint_path_max_endpoint_candidates: int = Field(default=3, ge=1, le=5)
|
||||
endpoint_path_per_node_limit: int = Field(default=16, ge=2, le=50)
|
||||
endpoint_path_max_expansions: int = Field(default=24, ge=10, le=1000)
|
||||
endpoint_path_max_frontier_states: int = Field(default=120, ge=20, le=2000)
|
||||
endpoint_path_time_budget_seconds: float = Field(default=5.0, ge=1.0, le=30.0)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_pipeline() -> QAV1Pipeline:
|
||||
return QAV1Pipeline()
|
||||
|
||||
|
||||
router = APIRouter(prefix="/api/v1/qa-v1-1-73", tags=["QA V1.1.73"] )
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
def health() -> Dict[str, Any]:
|
||||
return {"status": "ok", "service": "qa_v1_1_73", "version": __version__}
|
||||
|
||||
|
||||
@router.get("/info")
|
||||
def info() -> Dict[str, Any]:
|
||||
config = QAConfig.from_env()
|
||||
return {
|
||||
"name": "QA V1.1.73 Assertion-Anchored Full Graph",
|
||||
"version": __version__,
|
||||
"pipeline": [
|
||||
"question",
|
||||
"provisional answer",
|
||||
"structured planner",
|
||||
"direct Qdrant recall over V15/V16 canonical assertions",
|
||||
"canonical seed extraction",
|
||||
"Neo4j exact seed retrieval",
|
||||
"Neo4j bounded graph expansion",
|
||||
"BGE rerank",
|
||||
"graph-balanced top30 evidence chain, with at least 15 graph-derived items when available",
|
||||
"BGE top10 Qdrant source seed selection for cascaded native Neo4j two-hop chain retrieval",
|
||||
"BGE top10 complete assertions locked as hop 1 and expanded from each target in Neo4j",
|
||||
"merge assertion-anchored chains with the unchanged formal native graph lane",
|
||||
"compose a validated final LLM answer for chat; retrieval-only search returns evidence without an answer",
|
||||
],
|
||||
"safety_boundary": (
|
||||
"The provisional answer is never a final evidence source. "
|
||||
"The original-question retrieval lane is always retained."
|
||||
),
|
||||
"config": {
|
||||
"alias_path": str(config.alias_path),
|
||||
"taxonomy_path": str(config.taxonomy_path),
|
||||
"reranker_model": str(config.reranker_model),
|
||||
"max_rounds": config.max_rounds,
|
||||
"max_evidence": config.max_evidence,
|
||||
"provisional_draft": config.enable_provisional_draft,
|
||||
"second_round": config.enable_second_round,
|
||||
"bge_rerank": config.enable_bge_rerank,
|
||||
"graph_schema": config.graph_schema,
|
||||
"knowledge_chain_defaults": {
|
||||
"chain_top_k": 3,
|
||||
"chain_hops": 2,
|
||||
"chain_max_edges": 12,
|
||||
"chain_max_nodes": 10,
|
||||
"chain_mode": "display",
|
||||
"used_for_answer_generation": True,
|
||||
"graph_min_evidence_in_top30": 15,
|
||||
"native_chain_max_hops": 3,
|
||||
"native_chain_top_k": 5,
|
||||
"native_chain_min_hops": 1,
|
||||
"endpoint_path_enabled": False,
|
||||
"endpoint_path_max_hops": 3,
|
||||
"endpoint_path_top_k": 5,
|
||||
},
|
||||
"request_ablation_supported": [
|
||||
"use_provisional_draft",
|
||||
"max_rounds",
|
||||
"ablation.retrieval.use_qdrant",
|
||||
"ablation.retrieval.use_neo4j",
|
||||
"ablation.retrieval.use_graph_expansion",
|
||||
"ablation.retrieval.use_v15",
|
||||
"ablation.use_bge_rerank",
|
||||
"ablation.native_seed_policy",
|
||||
"ablation.native_seed_top_k",
|
||||
"ablation.native_target_seed_policy",
|
||||
"endpoint_path_enabled",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.post("/chat")
|
||||
async def chat(request: QARequest) -> Dict[str, Any]:
|
||||
if request.use_llm is False:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail={"error_code": "llm_required", "answer": "", "message": "Final-answer LLM is required"},
|
||||
)
|
||||
pipeline = get_pipeline()
|
||||
if not pipeline.llm.available:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail={"error_code": "llm_unavailable", "answer": "", "message": "Final-answer LLM is not configured"},
|
||||
)
|
||||
try:
|
||||
ablation = dict(request.ablation or {})
|
||||
ablation["use_final_llm"] = True
|
||||
return await asyncio.to_thread(
|
||||
pipeline.answer,
|
||||
request.question,
|
||||
debug=request.debug,
|
||||
use_provisional_draft=request.use_provisional_draft,
|
||||
max_rounds=request.max_rounds,
|
||||
ablation=ablation,
|
||||
chain_top_k=request.chain_top_k,
|
||||
chain_hops=request.chain_hops,
|
||||
chain_max_edges=request.chain_max_edges,
|
||||
chain_max_nodes=request.chain_max_nodes,
|
||||
chain_mode=request.chain_mode,
|
||||
chain_min_score=request.chain_min_score,
|
||||
native_chain_enabled=request.native_chain_enabled,
|
||||
native_chain_max_hops=request.native_chain_max_hops,
|
||||
native_chain_top_k=request.native_chain_top_k,
|
||||
native_chain_min_hops=request.native_chain_min_hops,
|
||||
native_chain_beam_width=request.native_chain_beam_width,
|
||||
native_chain_per_node_expand=request.native_chain_per_node_expand,
|
||||
endpoint_path_enabled=request.endpoint_path_enabled,
|
||||
endpoint_path_max_hops=request.endpoint_path_max_hops,
|
||||
endpoint_path_top_k=request.endpoint_path_top_k,
|
||||
endpoint_path_max_endpoint_candidates=request.endpoint_path_max_endpoint_candidates,
|
||||
endpoint_path_per_node_limit=request.endpoint_path_per_node_limit,
|
||||
endpoint_path_max_expansions=request.endpoint_path_max_expansions,
|
||||
endpoint_path_max_frontier_states=request.endpoint_path_max_frontier_states,
|
||||
endpoint_path_time_budget_seconds=request.endpoint_path_time_budget_seconds,
|
||||
)
|
||||
except QAAnswerError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.status_code,
|
||||
detail={"error_code": exc.error_code, "answer": "", "message": str(exc)},
|
||||
) from exc
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail={"error_code": "qa_failure", "answer": "", "message": str(exc)},
|
||||
) from exc
|
||||
|
||||
|
||||
@router.post("/chat/stream")
|
||||
async def chat_stream(request: QARequest) -> StreamingResponse:
|
||||
"""Stream truthful pipeline milestones, then the unchanged chat payload."""
|
||||
if request.use_llm is False:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail={"error_code": "llm_required", "answer": "", "message": "Final-answer LLM is required"},
|
||||
)
|
||||
pipeline = get_pipeline()
|
||||
if not pipeline.llm.available:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail={"error_code": "llm_unavailable", "answer": "", "message": "Final-answer LLM is not configured"},
|
||||
)
|
||||
|
||||
events: queue.Queue[tuple[str, Dict[str, Any]] | None] = queue.Queue()
|
||||
|
||||
def emit(payload: Dict[str, Any]) -> None:
|
||||
events.put(("progress", payload))
|
||||
|
||||
def run() -> None:
|
||||
try:
|
||||
ablation = dict(request.ablation or {})
|
||||
ablation["use_final_llm"] = True
|
||||
result = pipeline.answer(
|
||||
request.question,
|
||||
debug=request.debug,
|
||||
use_provisional_draft=request.use_provisional_draft,
|
||||
max_rounds=request.max_rounds,
|
||||
ablation=ablation,
|
||||
chain_top_k=request.chain_top_k,
|
||||
chain_hops=request.chain_hops,
|
||||
chain_max_edges=request.chain_max_edges,
|
||||
chain_max_nodes=request.chain_max_nodes,
|
||||
chain_mode=request.chain_mode,
|
||||
chain_min_score=request.chain_min_score,
|
||||
native_chain_enabled=request.native_chain_enabled,
|
||||
native_chain_max_hops=request.native_chain_max_hops,
|
||||
native_chain_top_k=request.native_chain_top_k,
|
||||
native_chain_min_hops=request.native_chain_min_hops,
|
||||
native_chain_beam_width=request.native_chain_beam_width,
|
||||
native_chain_per_node_expand=request.native_chain_per_node_expand,
|
||||
endpoint_path_enabled=request.endpoint_path_enabled,
|
||||
endpoint_path_max_hops=request.endpoint_path_max_hops,
|
||||
endpoint_path_top_k=request.endpoint_path_top_k,
|
||||
endpoint_path_max_endpoint_candidates=request.endpoint_path_max_endpoint_candidates,
|
||||
endpoint_path_per_node_limit=request.endpoint_path_per_node_limit,
|
||||
endpoint_path_max_expansions=request.endpoint_path_max_expansions,
|
||||
endpoint_path_max_frontier_states=request.endpoint_path_max_frontier_states,
|
||||
endpoint_path_time_budget_seconds=request.endpoint_path_time_budget_seconds,
|
||||
progress_callback=emit,
|
||||
)
|
||||
events.put(("complete", result))
|
||||
except QAAnswerError as exc:
|
||||
events.put(("error", {
|
||||
"success": False,
|
||||
"status": exc.status_code,
|
||||
"error_code": exc.error_code,
|
||||
"message": str(exc),
|
||||
"answer": "",
|
||||
}))
|
||||
except Exception as exc:
|
||||
events.put(("error", {
|
||||
"success": False,
|
||||
"status": 500,
|
||||
"error_code": "qa_failure",
|
||||
"message": str(exc),
|
||||
"answer": "",
|
||||
}))
|
||||
finally:
|
||||
events.put(None)
|
||||
|
||||
threading.Thread(target=run, name="qa-v1-1-73-stream", daemon=True).start()
|
||||
|
||||
def stream():
|
||||
while True:
|
||||
try:
|
||||
item = events.get(timeout=15)
|
||||
except queue.Empty:
|
||||
yield ": keepalive\n\n"
|
||||
continue
|
||||
if item is None:
|
||||
break
|
||||
event_name, payload = item
|
||||
yield (
|
||||
f"event: {event_name}\n"
|
||||
f"data: {json.dumps(payload, ensure_ascii=False)}\n\n"
|
||||
)
|
||||
|
||||
return StreamingResponse(
|
||||
stream(),
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/search")
|
||||
async def search(request: QARequest) -> Dict[str, Any]:
|
||||
response = await asyncio.to_thread(
|
||||
get_pipeline().answer,
|
||||
request.question,
|
||||
debug=True,
|
||||
use_provisional_draft=request.use_provisional_draft,
|
||||
max_rounds=request.max_rounds,
|
||||
ablation=dict(request.ablation or {}),
|
||||
chain_top_k=request.chain_top_k,
|
||||
chain_hops=request.chain_hops,
|
||||
chain_max_edges=request.chain_max_edges,
|
||||
chain_max_nodes=request.chain_max_nodes,
|
||||
chain_mode=request.chain_mode,
|
||||
chain_min_score=request.chain_min_score,
|
||||
native_chain_enabled=request.native_chain_enabled,
|
||||
native_chain_max_hops=request.native_chain_max_hops,
|
||||
native_chain_top_k=request.native_chain_top_k,
|
||||
native_chain_min_hops=request.native_chain_min_hops,
|
||||
native_chain_beam_width=request.native_chain_beam_width,
|
||||
native_chain_per_node_expand=request.native_chain_per_node_expand,
|
||||
endpoint_path_enabled=request.endpoint_path_enabled,
|
||||
endpoint_path_max_hops=request.endpoint_path_max_hops,
|
||||
endpoint_path_top_k=request.endpoint_path_top_k,
|
||||
endpoint_path_max_endpoint_candidates=request.endpoint_path_max_endpoint_candidates,
|
||||
endpoint_path_per_node_limit=request.endpoint_path_per_node_limit,
|
||||
endpoint_path_max_expansions=request.endpoint_path_max_expansions,
|
||||
endpoint_path_max_frontier_states=request.endpoint_path_max_frontier_states,
|
||||
endpoint_path_time_budget_seconds=request.endpoint_path_time_budget_seconds,
|
||||
retrieval_only=True,
|
||||
)
|
||||
if isinstance(response, dict):
|
||||
response["answer"] = ""
|
||||
return response
|
||||
@@ -0,0 +1,544 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from typing import Any, Dict, Iterable, List, Tuple
|
||||
|
||||
from neo4j import Query
|
||||
|
||||
from .bounded_canonical_retriever import BoundedCanonicalGraphPathRetriever
|
||||
from .graph_path_chain import BLOCKED_CATEGORIES, GraphHop, GraphPath, _norm, _text
|
||||
from .schemas import RetrievalHypothesis
|
||||
|
||||
|
||||
UNKNOWN_ANSWER_TYPES = {
|
||||
"",
|
||||
"any",
|
||||
"none",
|
||||
"not specified",
|
||||
"unspecified",
|
||||
"unknown",
|
||||
}
|
||||
|
||||
|
||||
class AssertionAnchoredGraphRetriever(BoundedCanonicalGraphPathRetriever):
|
||||
"""Lock reranked vector assertions as hop 1, then expand their targets.
|
||||
|
||||
Qdrant and Neo4j were built from the same canonical assertion corpus. The
|
||||
vector assertion ID is therefore a reliable cross-store join key. The
|
||||
public query entrance remains natural-language/vector retrieval; canonical
|
||||
entity IDs are used only after Neo4j has verified the selected assertion.
|
||||
"""
|
||||
|
||||
CHAIN_SOURCE = "neo4j_vector_assertion_anchored_path"
|
||||
|
||||
def retrieve_from_assertions(
|
||||
self,
|
||||
question: str,
|
||||
hypothesis: RetrievalHypothesis,
|
||||
assertion_records: Iterable[Dict[str, Any]],
|
||||
*,
|
||||
relation_categories: Iterable[str] | None = None,
|
||||
answer_type_groups: Iterable[str] | None = None,
|
||||
max_hops: int = 3,
|
||||
per_node_expand: int = 12,
|
||||
beam_width: int = 80,
|
||||
top_k_chains: int = 10,
|
||||
hop_gate: Any | None = None,
|
||||
hop_gate_context: Dict[str, Any] | None = None,
|
||||
) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
|
||||
started = time.perf_counter()
|
||||
max_hops = max(1, min(int(max_hops or 3), 3))
|
||||
per_node_expand = max(1, min(int(per_node_expand or 12), 30))
|
||||
beam_width = max(10, min(int(beam_width or 80), 200))
|
||||
top_k_chains = max(0, min(int(top_k_chains or 10), 20))
|
||||
records = self._clean_assertion_records(assertion_records)[:10]
|
||||
answer_types = [
|
||||
_norm(value)
|
||||
for value in (answer_type_groups or [])
|
||||
if _norm(value) not in UNKNOWN_ANSWER_TYPES
|
||||
]
|
||||
audit: Dict[str, Any] = {
|
||||
"enabled": True,
|
||||
"chain_source": self.CHAIN_SOURCE,
|
||||
"policy": "qdrant_bge_top10_assertion_locked_as_hop1_then_target_expansion",
|
||||
"requested_assertion_count": len(records),
|
||||
"requested_assertion_ids": [row["assertion_id"] for row in records],
|
||||
"max_hops": max_hops,
|
||||
"per_node_expand": per_node_expand,
|
||||
"beam_width": beam_width,
|
||||
"top_k_chains": top_k_chains,
|
||||
"answer_type_groups": answer_types,
|
||||
"hop_stats": [],
|
||||
"agentic_hop_gate": {
|
||||
"enabled": hop_gate is not None,
|
||||
"decisions": [],
|
||||
},
|
||||
"filtered_reasons": {},
|
||||
"fallback_required": False,
|
||||
}
|
||||
if top_k_chains <= 0:
|
||||
audit["status"] = "disabled_by_top_k"
|
||||
return [], audit
|
||||
if not records:
|
||||
audit.update(
|
||||
{
|
||||
"status": "no_vector_assertions",
|
||||
"fallback_required": True,
|
||||
"elapsed_seconds": time.perf_counter() - started,
|
||||
}
|
||||
)
|
||||
return [], audit
|
||||
|
||||
self._bounded_reset_metrics()
|
||||
self._bounded_prepare_question(question)
|
||||
first_rows = self._load_verified_first_hops(records)
|
||||
audit["verified_first_hop_count"] = len(first_rows)
|
||||
audit["verified_first_hop_assertion_ids"] = [
|
||||
_text(row.get("assertion_id")) for row in first_rows
|
||||
]
|
||||
if not first_rows:
|
||||
audit.update(
|
||||
{
|
||||
"status": "no_assertion_mapped_to_neo4j",
|
||||
"fallback_required": True,
|
||||
"bounded_graph_backend": dict(self._bounded_metrics),
|
||||
"elapsed_seconds": time.perf_counter() - started,
|
||||
}
|
||||
)
|
||||
return [], audit
|
||||
|
||||
question_norm = _norm(question)
|
||||
allowed = self._allowed_categories(hypothesis, relation_categories)
|
||||
allowed.update(self._categories_from_question(question_norm))
|
||||
relation_clues = self._relation_clues(question)
|
||||
priority_relation_clues = self._priority_relation_clues(question_norm)
|
||||
seed_norms = {
|
||||
value
|
||||
for row in first_rows
|
||||
for value in (_norm(row.get("source")), _norm(row.get("target")))
|
||||
if value
|
||||
}
|
||||
entity_clues = self._visible_entity_clues(question_norm, seed_norms)
|
||||
|
||||
first_paths: List[GraphPath] = []
|
||||
for row in first_rows:
|
||||
edge_score, edge_breakdown = self._score_edge(
|
||||
row, question_norm, seed_norms, allowed
|
||||
)
|
||||
vector_score = float(row.get("seed_rerank_score") or 0.0)
|
||||
edge_score = min(1.0, 0.55 * edge_score + 0.45 * vector_score)
|
||||
edge_breakdown.update(
|
||||
{
|
||||
"vector_assertion_first_hop_verified": 1.0,
|
||||
"vector_rerank_score": vector_score,
|
||||
}
|
||||
)
|
||||
hop = self._row_to_hop(row, hop_index=1, edge_score=edge_score, edge_breakdown=edge_breakdown)
|
||||
path = GraphPath(
|
||||
hops=[hop],
|
||||
node_ids=[hop.source_canonical_id, hop.target_canonical_id],
|
||||
)
|
||||
path.path_score, path.score_breakdown = self._score_anchored_path(
|
||||
path,
|
||||
question_norm,
|
||||
seed_norms,
|
||||
allowed,
|
||||
relation_clues,
|
||||
answer_types,
|
||||
entity_clues,
|
||||
vector_score,
|
||||
)
|
||||
first_paths.append(path)
|
||||
|
||||
all_paths: List[GraphPath] = list(first_paths)
|
||||
beam = self._select_beam_paths(
|
||||
list(first_paths), beam_width, entity_clues=entity_clues
|
||||
)
|
||||
filtered: Dict[str, int] = {}
|
||||
if hop_gate is not None and beam:
|
||||
decision = hop_gate.decide(
|
||||
question,
|
||||
hypothesis,
|
||||
beam,
|
||||
current_hop=1,
|
||||
safety_max_hops=max_hops,
|
||||
task_context=hop_gate_context or {},
|
||||
)
|
||||
audit["agentic_hop_gate"]["decisions"].append(
|
||||
self._public_hop_gate_decision(decision)
|
||||
)
|
||||
if decision["action"] == "continue":
|
||||
beam = self._paths_at_indexes(
|
||||
beam, decision.get("expand_path_indexes") or []
|
||||
)
|
||||
else:
|
||||
beam = []
|
||||
|
||||
for hop_index in range(2, max_hops + 1):
|
||||
frontier_ids = self._dedupe([path.end_id for path in beam if path.end_id])
|
||||
if not frontier_ids:
|
||||
break
|
||||
rows = self._fetch_neighbors(
|
||||
frontier_ids,
|
||||
allowed,
|
||||
relation_clues=relation_clues,
|
||||
priority_relation_clues=priority_relation_clues,
|
||||
entity_clues=entity_clues,
|
||||
answer_type_groups=answer_types,
|
||||
per_node_expand=per_node_expand,
|
||||
)
|
||||
paths_by_end: Dict[str, List[GraphPath]] = {}
|
||||
for path in beam:
|
||||
paths_by_end.setdefault(path.end_id, []).append(path)
|
||||
next_paths: List[GraphPath] = []
|
||||
for row in rows:
|
||||
from_id = _text(row.get("from_id"))
|
||||
to_id = _text(row.get("to_id"))
|
||||
if not from_id or not to_id:
|
||||
self._inc(filtered, "missing_node")
|
||||
continue
|
||||
if _text(row.get("relation_category")) in BLOCKED_CATEGORIES:
|
||||
self._inc(filtered, "blocked_relation_category")
|
||||
continue
|
||||
if (
|
||||
_text(row.get("source_canonical_id")) != from_id
|
||||
or _text(row.get("target_canonical_id")) != to_id
|
||||
):
|
||||
self._inc(filtered, "non_directed_srt_continuity")
|
||||
continue
|
||||
for base in paths_by_end.get(from_id, []):
|
||||
assertion_id = _text(row.get("assertion_id"))
|
||||
if assertion_id and assertion_id in base.assertion_ids():
|
||||
self._inc(filtered, "repeat_assertion")
|
||||
continue
|
||||
if to_id in base.node_ids:
|
||||
self._inc(filtered, "repeat_node")
|
||||
continue
|
||||
edge_score, edge_breakdown = self._score_edge(
|
||||
row, question_norm, seed_norms, allowed
|
||||
)
|
||||
relation_tokens = {
|
||||
token
|
||||
for token in _norm(row.get("relation")).split()
|
||||
if len(token) >= 4
|
||||
}
|
||||
question_tokens = set(question_norm.split())
|
||||
relation_question_overlap = (
|
||||
len(relation_tokens & question_tokens)
|
||||
/ max(1, len(relation_tokens))
|
||||
)
|
||||
edge_score = min(
|
||||
1.0, edge_score + 0.18 * relation_question_overlap
|
||||
)
|
||||
edge_breakdown["relation_question_token_overlap"] = (
|
||||
relation_question_overlap
|
||||
)
|
||||
if edge_score < 0.12:
|
||||
self._inc(filtered, "low_edge_score")
|
||||
continue
|
||||
hop = self._row_to_hop(
|
||||
row,
|
||||
hop_index=hop_index,
|
||||
edge_score=edge_score,
|
||||
edge_breakdown=edge_breakdown,
|
||||
)
|
||||
path = GraphPath(
|
||||
hops=base.hops + [hop],
|
||||
node_ids=base.node_ids + [to_id],
|
||||
)
|
||||
vector_score = float(
|
||||
base.score_breakdown.get("vector_first_hop_score") or 0.0
|
||||
)
|
||||
path.path_score, path.score_breakdown = self._score_anchored_path(
|
||||
path,
|
||||
question_norm,
|
||||
seed_norms,
|
||||
allowed,
|
||||
relation_clues,
|
||||
answer_types,
|
||||
entity_clues,
|
||||
vector_score,
|
||||
)
|
||||
next_paths.append(path)
|
||||
next_paths = self._dedupe_paths(next_paths)
|
||||
next_paths.sort(key=lambda item: item.path_score, reverse=True)
|
||||
all_paths.extend(next_paths)
|
||||
candidate_beam = self._select_beam_paths(
|
||||
next_paths,
|
||||
beam_width,
|
||||
entity_clues=entity_clues,
|
||||
)
|
||||
if hop_gate is not None and candidate_beam:
|
||||
decision = hop_gate.decide(
|
||||
question,
|
||||
hypothesis,
|
||||
candidate_beam,
|
||||
current_hop=hop_index,
|
||||
safety_max_hops=max_hops,
|
||||
task_context=hop_gate_context or {},
|
||||
)
|
||||
audit["agentic_hop_gate"]["decisions"].append(
|
||||
self._public_hop_gate_decision(decision)
|
||||
)
|
||||
if decision["action"] == "continue":
|
||||
beam = self._paths_at_indexes(
|
||||
candidate_beam,
|
||||
decision.get("expand_path_indexes") or [],
|
||||
)
|
||||
else:
|
||||
beam = []
|
||||
else:
|
||||
beam = candidate_beam
|
||||
audit["hop_stats"].append(
|
||||
{
|
||||
"hop": hop_index,
|
||||
"frontier_count": len(frontier_ids),
|
||||
"raw_edge_count": len(rows),
|
||||
"candidate_path_count": len(next_paths),
|
||||
"kept_beam_count": len(beam),
|
||||
}
|
||||
)
|
||||
if not beam:
|
||||
break
|
||||
|
||||
all_paths = self._dedupe_paths(all_paths)
|
||||
selected = self._select_diverse_paths(all_paths, top_k_chains)
|
||||
chains = []
|
||||
for index, path in enumerate(selected, start=1):
|
||||
chain = path.to_chain_dict(index)
|
||||
chain["chain_source"] = self.CHAIN_SOURCE
|
||||
chain["first_hop_assertion_id"] = path.hops[0].assertion_id if path.hops else ""
|
||||
chain["first_hop_locked"] = True
|
||||
chains.append(chain)
|
||||
|
||||
audit.update(
|
||||
{
|
||||
"status": "ok" if chains else "no_expanded_chain",
|
||||
"fallback_required": not bool(chains),
|
||||
"candidate_path_count": len(all_paths),
|
||||
"returned_chain_count": len(chains),
|
||||
"selected_hop_counts": {
|
||||
str(hop): sum(1 for path in selected if len(path.hops) == hop)
|
||||
for hop in range(1, max_hops + 1)
|
||||
},
|
||||
"filtered_reasons": filtered,
|
||||
"bounded_graph_backend": dict(self._bounded_metrics),
|
||||
"elapsed_seconds": time.perf_counter() - started,
|
||||
}
|
||||
)
|
||||
return chains, audit
|
||||
|
||||
@staticmethod
|
||||
def _paths_at_indexes(paths: List[GraphPath], indexes: Iterable[Any]) -> List[GraphPath]:
|
||||
selected: List[GraphPath] = []
|
||||
seen: set[int] = set()
|
||||
for value in indexes:
|
||||
try:
|
||||
index = int(value)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if index in seen or index < 0 or index >= len(paths):
|
||||
continue
|
||||
seen.add(index)
|
||||
selected.append(paths[index])
|
||||
return selected
|
||||
|
||||
@staticmethod
|
||||
def _public_hop_gate_decision(decision: Dict[str, Any]) -> Dict[str, Any]:
|
||||
return {
|
||||
key: value
|
||||
for key, value in decision.items()
|
||||
if key != "expand_path_indexes"
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _clean_assertion_records(
|
||||
assertion_records: Iterable[Dict[str, Any]],
|
||||
) -> List[Dict[str, Any]]:
|
||||
output: List[Dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
for record in assertion_records or []:
|
||||
if not isinstance(record, dict):
|
||||
continue
|
||||
assertion_id = _text(record.get("assertion_id"))
|
||||
if not assertion_id.startswith("cass_") or assertion_id in seen:
|
||||
continue
|
||||
seen.add(assertion_id)
|
||||
output.append(dict(record))
|
||||
return output
|
||||
|
||||
def _load_verified_first_hops(
|
||||
self, records: List[Dict[str, Any]]
|
||||
) -> List[Dict[str, Any]]:
|
||||
statement = Query(
|
||||
"""
|
||||
UNWIND $records AS seed
|
||||
MATCH (a:SRT2CanonicalAssertion {assertion_id: seed.assertion_id})
|
||||
MATCH (s:SRT2CanonicalEntity {canonical_id: a.source_canonical_id})
|
||||
MATCH (t:SRT2CanonicalEntity {canonical_id: a.target_canonical_id})
|
||||
MATCH (s)-[r:SRT2_CANONICAL_ASSERTS]-(t)
|
||||
WHERE r.assertion_id = a.assertion_id
|
||||
RETURN seed.rank_after_bge AS seed_rank,
|
||||
seed.final_score AS seed_final_score,
|
||||
seed.rerank_score AS seed_rerank_score,
|
||||
seed.vector_score AS seed_vector_score,
|
||||
a.assertion_id AS assertion_id,
|
||||
s.canonical_id AS from_id,
|
||||
s.name AS from_name,
|
||||
t.canonical_id AS to_id,
|
||||
t.name AS to_name,
|
||||
a.source AS source,
|
||||
a.source_type AS source_type,
|
||||
a.source_canonical_id AS source_canonical_id,
|
||||
a.relation_norm AS relation,
|
||||
a.relation_category AS relation_category,
|
||||
a.target AS target,
|
||||
a.target_type AS target_type,
|
||||
a.target_canonical_id AS target_canonical_id,
|
||||
a.best_evidence AS evidence,
|
||||
a.best_paper_id AS paper_id,
|
||||
a.title AS title,
|
||||
a.pmid AS pmid,
|
||||
a.pmcid AS pmcid,
|
||||
a.doi AS doi,
|
||||
'' AS species,
|
||||
coalesce(a.support_count, 0) AS support_count,
|
||||
coalesce(a.paper_count, 0) AS paper_count,
|
||||
coalesce(a.best_score, 0.0) AS best_score,
|
||||
1 AS category_match,
|
||||
0 AS relation_clue_score
|
||||
ORDER BY seed.rank_after_bge ASC
|
||||
""",
|
||||
timeout=self._bounded_neo4j_timeout,
|
||||
)
|
||||
with self.driver.session() as session:
|
||||
return [dict(record) for record in session.run(statement, records=records)]
|
||||
|
||||
def _score_anchored_path(
|
||||
self,
|
||||
path: GraphPath,
|
||||
question_norm: str,
|
||||
seed_norms: set[str],
|
||||
allowed: set[str],
|
||||
relation_clues: List[str],
|
||||
answer_types: List[str],
|
||||
entity_clues: List[str],
|
||||
vector_score: float,
|
||||
) -> Tuple[float, Dict[str, Any]]:
|
||||
base_score, breakdown = self._score_path(
|
||||
path,
|
||||
question_norm,
|
||||
seed_norms,
|
||||
allowed,
|
||||
relation_clues,
|
||||
answer_types,
|
||||
entity_clues,
|
||||
)
|
||||
vector_score = max(0.0, min(float(vector_score or 0.0), 1.0))
|
||||
multihop_bonus = 0.035 if len(path.hops) >= 2 else 0.0
|
||||
terminal_relation_tokens = {
|
||||
token
|
||||
for token in _norm(path.hops[-1].relation if path.hops else "").split()
|
||||
if len(token) >= 4
|
||||
}
|
||||
terminal_relation_overlap = (
|
||||
len(terminal_relation_tokens & set(question_norm.split()))
|
||||
/ max(1, len(terminal_relation_tokens))
|
||||
)
|
||||
final_score = min(
|
||||
1.0,
|
||||
0.68 * base_score
|
||||
+ 0.32 * vector_score
|
||||
+ multihop_bonus
|
||||
+ 0.18 * terminal_relation_overlap,
|
||||
)
|
||||
breakdown.update(
|
||||
{
|
||||
"base_native_path_score": base_score,
|
||||
"vector_first_hop_score": vector_score,
|
||||
"vector_first_hop_weight": 0.32,
|
||||
"first_hop_locked": True,
|
||||
"multihop_bonus": multihop_bonus,
|
||||
"terminal_relation_question_overlap": terminal_relation_overlap,
|
||||
"final_score": final_score,
|
||||
}
|
||||
)
|
||||
return final_score, breakdown
|
||||
|
||||
@staticmethod
|
||||
def _row_to_hop(
|
||||
row: Dict[str, Any],
|
||||
*,
|
||||
hop_index: int,
|
||||
edge_score: float,
|
||||
edge_breakdown: Dict[str, Any],
|
||||
) -> GraphHop:
|
||||
return GraphHop(
|
||||
hop_index=hop_index,
|
||||
from_id=_text(row.get("from_id")),
|
||||
from_name=_text(row.get("from_name")),
|
||||
to_id=_text(row.get("to_id")),
|
||||
to_name=_text(row.get("to_name")),
|
||||
source=_text(row.get("source")),
|
||||
relation=_text(row.get("relation")),
|
||||
target=_text(row.get("target")),
|
||||
source_canonical_id=_text(row.get("source_canonical_id")),
|
||||
target_canonical_id=_text(row.get("target_canonical_id")),
|
||||
source_type=_text(row.get("source_type")),
|
||||
target_type=_text(row.get("target_type")),
|
||||
relation_category=_text(row.get("relation_category")),
|
||||
assertion_id=_text(row.get("assertion_id")),
|
||||
paper_id=_text(row.get("paper_id")),
|
||||
title=_text(row.get("title")),
|
||||
pmid=_text(row.get("pmid")),
|
||||
pmcid=_text(row.get("pmcid")),
|
||||
doi=_text(row.get("doi")),
|
||||
species=_text(row.get("species")),
|
||||
evidence=_text(row.get("evidence")),
|
||||
support_count=int(row.get("support_count") or 0),
|
||||
paper_count=int(row.get("paper_count") or 0),
|
||||
best_score=float(row.get("best_score") or 0.0),
|
||||
edge_score=edge_score,
|
||||
score_breakdown=edge_breakdown,
|
||||
passive_relation=False,
|
||||
structural_direction=(
|
||||
f"{_text(row.get('source'))} -> {_text(row.get('target'))}"
|
||||
),
|
||||
semantic_direction=(
|
||||
f"{_text(row.get('source'))} -> {_text(row.get('target'))}"
|
||||
),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _select_diverse_paths(
|
||||
paths: List[GraphPath], top_k: int
|
||||
) -> List[GraphPath]:
|
||||
ranked = sorted(
|
||||
paths,
|
||||
key=lambda path: (len(path.hops) >= 2, path.path_score),
|
||||
reverse=True,
|
||||
)
|
||||
selected: List[GraphPath] = []
|
||||
selected_keys: set[Tuple[str, ...]] = set()
|
||||
first_hop_counts: Dict[str, int] = {}
|
||||
for path in ranked:
|
||||
key = tuple(path.assertion_ids())
|
||||
if not key or key in selected_keys:
|
||||
continue
|
||||
first = key[0]
|
||||
if first_hop_counts.get(first, 0) >= 2:
|
||||
continue
|
||||
selected.append(path)
|
||||
selected_keys.add(key)
|
||||
first_hop_counts[first] = first_hop_counts.get(first, 0) + 1
|
||||
if len(selected) >= top_k:
|
||||
return selected
|
||||
for path in ranked:
|
||||
key = tuple(path.assertion_ids())
|
||||
if not key or key in selected_keys:
|
||||
continue
|
||||
selected.append(path)
|
||||
selected_keys.add(key)
|
||||
if len(selected) >= top_k:
|
||||
break
|
||||
return selected
|
||||
@@ -0,0 +1,28 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
class AuditWriter:
|
||||
def __init__(self, root: Path):
|
||||
self.root = root
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def new_query_id(self) -> str:
|
||||
return time.strftime("%Y%m%d_%H%M%S") + "_" + uuid.uuid4().hex[:8]
|
||||
|
||||
def write(self, query_id: str, payload: Dict[str, Any]) -> Path:
|
||||
path = self.root / query_id
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
target = path / "qa_audit.json"
|
||||
with self._lock:
|
||||
target.write_text(
|
||||
json.dumps(payload, ensure_ascii=False, indent=2, default=str),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return target
|
||||
@@ -0,0 +1,206 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Any, Dict
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
from neo4j import GraphDatabase
|
||||
|
||||
|
||||
ALLOWED_QDRANT_COLLECTIONS = {
|
||||
"plant_srt_direct_v1_bge_m3_20260702",
|
||||
"plant_srt_direct_v1_bge_m3_citations_20260717",
|
||||
}
|
||||
REQUIRED_NODE_LABELS = {
|
||||
"SRT2Paper",
|
||||
"SRT2CanonicalEntity",
|
||||
"SRT2CanonicalAssertion",
|
||||
}
|
||||
REQUIRED_RELATIONSHIP_TYPES = {
|
||||
"SRT2_CANONICAL_ASSERTS",
|
||||
"SRT2_SUPPORTED_BY",
|
||||
}
|
||||
FORBIDDEN_NODE_LABELS = {
|
||||
"Entity",
|
||||
"PCEntity",
|
||||
"PCRelationEvidence",
|
||||
"PCPaper",
|
||||
"SRT2Entity",
|
||||
"SRT2Assertion",
|
||||
}
|
||||
FORBIDDEN_RELATIONSHIP_TYPES = {
|
||||
"RELATION",
|
||||
"PC_REL",
|
||||
"HAS_SOURCE",
|
||||
"HAS_TARGET",
|
||||
"FROM_PAPER",
|
||||
"SRT2_ASSERTS",
|
||||
"SRT2_ASSERTION_SOURCE",
|
||||
"SRT2_ASSERTION_TARGET",
|
||||
}
|
||||
EXPECTED_NEO4J_PAPERS = 31_207
|
||||
EXPECTED_NEO4J_CANONICAL_ENTITIES = 3_820_996
|
||||
EXPECTED_NEO4J_CANONICAL_ASSERTIONS = 8_246_875
|
||||
EXPECTED_NEO4J_CANONICAL_ASSERTS = 8_246_875
|
||||
EXPECTED_NEO4J_SUPPORTED_BY = 8_202_284
|
||||
|
||||
|
||||
class BackendIdentityError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def _qdrant_json(url: str) -> Dict[str, Any]:
|
||||
with urlopen(url, timeout=30) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def _qdrant_sample(url: str, collection: str) -> Dict[str, Any]:
|
||||
body = json.dumps(
|
||||
{"limit": 1, "with_payload": True, "with_vector": False}
|
||||
).encode("utf-8")
|
||||
request = Request(
|
||||
f"{url.rstrip('/')}/collections/{collection}/points/scroll",
|
||||
data=body,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
with urlopen(request, timeout=30) as response:
|
||||
data = json.load(response)
|
||||
points = data.get("result", {}).get("points") or []
|
||||
return dict(points[0].get("payload") or {}) if points else {}
|
||||
|
||||
|
||||
def validate_backends() -> Dict[str, Any]:
|
||||
graph_schema = os.environ.get("QA_V1_GRAPH_SCHEMA", "canonical").strip().lower()
|
||||
if graph_schema != "canonical":
|
||||
raise BackendIdentityError(
|
||||
f"QA_V1_GRAPH_SCHEMA must be canonical, got {graph_schema!r}"
|
||||
)
|
||||
|
||||
collection = os.environ.get(
|
||||
"V15_QDRANT_COLLECTION", "plant_srt_direct_v1_bge_m3_20260702"
|
||||
).strip()
|
||||
if collection not in ALLOWED_QDRANT_COLLECTIONS:
|
||||
raise BackendIdentityError(
|
||||
f"Qdrant collection {collection!r} is not in the production allowlist"
|
||||
)
|
||||
|
||||
qdrant_url = os.environ.get(
|
||||
"V15_QDRANT_URL", os.environ.get("QDRANT_URL", "http://127.0.0.1:6333")
|
||||
).rstrip("/")
|
||||
qdrant_info = _qdrant_json(f"{qdrant_url}/collections/{collection}").get(
|
||||
"result", {}
|
||||
)
|
||||
qdrant_points = int(qdrant_info.get("points_count") or 0)
|
||||
if qdrant_points != 8_246_777:
|
||||
raise BackendIdentityError(
|
||||
f"Qdrant identity mismatch: expected 8246777 points, got {qdrant_points}"
|
||||
)
|
||||
sample_payload = _qdrant_sample(qdrant_url, collection)
|
||||
if sample_payload.get("source_system") != "V15_V16_canonical":
|
||||
raise BackendIdentityError(
|
||||
"Qdrant payload is not the V15/V16 canonical assertion collection"
|
||||
)
|
||||
|
||||
neo4j_uri = os.environ.get("NEO4J_URI", "bolt://127.0.0.1:18788")
|
||||
neo4j_user = os.environ.get("NEO4J_USER", "neo4j")
|
||||
neo4j_password = os.environ.get("NEO4J_PASSWORD", "")
|
||||
driver = GraphDatabase.driver(
|
||||
neo4j_uri,
|
||||
auth=(neo4j_user, neo4j_password),
|
||||
connection_timeout=15,
|
||||
)
|
||||
try:
|
||||
with driver.session() as session:
|
||||
labels = set(
|
||||
session.run(
|
||||
"CALL db.labels() YIELD label RETURN label"
|
||||
).value("label")
|
||||
)
|
||||
relationship_types = set(
|
||||
session.run(
|
||||
"CALL db.relationshipTypes() YIELD relationshipType "
|
||||
"RETURN relationshipType"
|
||||
).value("relationshipType")
|
||||
)
|
||||
counts = session.run(
|
||||
"""
|
||||
MATCH (p:SRT2Paper)
|
||||
WITH count(p) AS papers
|
||||
MATCH (c:SRT2CanonicalAssertion)
|
||||
WITH papers, count(c) AS canonical_assertions
|
||||
MATCH (e:SRT2CanonicalEntity)
|
||||
RETURN papers, canonical_assertions,
|
||||
count(e) AS canonical_entities
|
||||
"""
|
||||
).single()
|
||||
relationship_counts = {
|
||||
row["relationship_type"]: int(row["count"])
|
||||
for row in session.run(
|
||||
"""
|
||||
CALL db.relationshipTypes() YIELD relationshipType
|
||||
CALL {
|
||||
WITH relationshipType
|
||||
MATCH ()-[r]->()
|
||||
WHERE type(r) = relationshipType
|
||||
RETURN count(r) AS relation_count
|
||||
}
|
||||
RETURN relationshipType AS relationship_type,
|
||||
relation_count AS count
|
||||
"""
|
||||
)
|
||||
}
|
||||
finally:
|
||||
driver.close()
|
||||
|
||||
missing_labels = REQUIRED_NODE_LABELS - labels
|
||||
forbidden_labels = FORBIDDEN_NODE_LABELS & labels
|
||||
missing_relationships = REQUIRED_RELATIONSHIP_TYPES - relationship_types
|
||||
forbidden_relationships = FORBIDDEN_RELATIONSHIP_TYPES & relationship_types
|
||||
if missing_labels or forbidden_labels or missing_relationships or forbidden_relationships:
|
||||
raise BackendIdentityError(
|
||||
"Neo4j schema identity mismatch: "
|
||||
f"missing_labels={sorted(missing_labels)}, "
|
||||
f"forbidden_labels={sorted(forbidden_labels)}, "
|
||||
f"missing_relationships={sorted(missing_relationships)}, "
|
||||
f"forbidden_relationships={sorted(forbidden_relationships)}"
|
||||
)
|
||||
|
||||
papers = int(counts["papers"])
|
||||
canonical_entities = int(counts["canonical_entities"])
|
||||
canonical_assertions = int(counts["canonical_assertions"])
|
||||
canonical_asserts = relationship_counts.get("SRT2_CANONICAL_ASSERTS", 0)
|
||||
supported_by = relationship_counts.get("SRT2_SUPPORTED_BY", 0)
|
||||
expected_identity = (
|
||||
papers == EXPECTED_NEO4J_PAPERS
|
||||
and canonical_entities == EXPECTED_NEO4J_CANONICAL_ENTITIES
|
||||
and canonical_assertions == EXPECTED_NEO4J_CANONICAL_ASSERTIONS
|
||||
and canonical_asserts == EXPECTED_NEO4J_CANONICAL_ASSERTS
|
||||
and supported_by == EXPECTED_NEO4J_SUPPORTED_BY
|
||||
)
|
||||
if not expected_identity:
|
||||
raise BackendIdentityError(
|
||||
"Neo4j store does not match the July 2 direct canonical identity: "
|
||||
f"papers={papers}, canonical_entities={canonical_entities}, "
|
||||
f"canonical_assertions={canonical_assertions}, "
|
||||
f"canonical_asserts={canonical_asserts}, supported_by={supported_by}"
|
||||
)
|
||||
|
||||
return {
|
||||
"backend_policy": "neo4j_qdrant_only",
|
||||
"graph_schema": graph_schema,
|
||||
"neo4j_uri": neo4j_uri,
|
||||
"neo4j_store": "neo4j_plant_srt_direct_v1_20260702",
|
||||
"neo4j_papers": papers,
|
||||
"neo4j_canonical_entities": canonical_entities,
|
||||
"neo4j_canonical_assertions": canonical_assertions,
|
||||
"neo4j_canonical_asserts": canonical_asserts,
|
||||
"neo4j_supported_by": supported_by,
|
||||
"qdrant_url": qdrant_url,
|
||||
"qdrant_collection": collection,
|
||||
"qdrant_points": qdrant_points,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(json.dumps(validate_backends(), ensure_ascii=False, indent=2))
|
||||
@@ -0,0 +1,420 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from typing import Any, Dict, Iterable, List, Tuple
|
||||
|
||||
import requests
|
||||
from neo4j import Query
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.http import models
|
||||
|
||||
from .direct_flow_retriever import DirectFlowRetriever, _norm, _text
|
||||
from .graph_path_chain import NativeGraphPathRetriever
|
||||
from .schemas import EvidenceItem, RetrievalHypothesis
|
||||
|
||||
|
||||
class _BoundedCanonicalMixin:
|
||||
"""Generate bounded graph candidates from indexed Qdrant payload fields.
|
||||
|
||||
Neo4j is used only to validate exact cass_* assertion IDs. This avoids
|
||||
enumerating and sorting every relationship attached to a high-degree node.
|
||||
"""
|
||||
|
||||
def _bounded_init(self) -> None:
|
||||
self._bounded_qdrant = QdrantClient(
|
||||
url=os.environ.get(
|
||||
"V15_QDRANT_URL",
|
||||
os.environ.get("QDRANT_URL", "http://127.0.0.1:6333"),
|
||||
),
|
||||
timeout=float(os.environ.get("QA_V1_BOUNDED_QDRANT_TIMEOUT", "20")),
|
||||
)
|
||||
self._bounded_collection = os.environ.get(
|
||||
"V15_QDRANT_COLLECTION",
|
||||
"plant_srt_direct_v1_bge_m3_20260702",
|
||||
)
|
||||
self._bounded_embed_url = os.environ.get(
|
||||
"QA_V112_EMBED_URL",
|
||||
os.environ.get("QA_V1_EMBED_URL", "http://127.0.0.1:11434/api/embed"),
|
||||
)
|
||||
self._bounded_embed_model = os.environ.get(
|
||||
"QA_V112_EMBED_MODEL",
|
||||
os.environ.get("QA_V1_EMBED_MODEL", "bge-m3:567m"),
|
||||
)
|
||||
self._bounded_max_frontier = max(
|
||||
1, min(int(os.environ.get("QA_V1_BOUNDED_MAX_FRONTIER", "24")), 48)
|
||||
)
|
||||
self._bounded_per_node_cap = max(
|
||||
1, min(int(os.environ.get("QA_V1_BOUNDED_PER_NODE", "8")), 16)
|
||||
)
|
||||
self._bounded_max_hops = max(
|
||||
1, min(int(os.environ.get("QA_V1_BOUNDED_MAX_HOPS", "2")), 3)
|
||||
)
|
||||
self._bounded_neo4j_timeout = max(
|
||||
1.0,
|
||||
min(
|
||||
float(os.environ.get("QA_V1_BOUNDED_NEO4J_TIMEOUT", "5")),
|
||||
20.0,
|
||||
),
|
||||
)
|
||||
self._bounded_question = ""
|
||||
self._bounded_vector: List[float] = []
|
||||
self._bounded_metrics: Dict[str, Any] = {}
|
||||
self._bounded_reset_metrics()
|
||||
|
||||
def _bounded_reset_metrics(self) -> None:
|
||||
self._bounded_metrics = {
|
||||
"backend": "qdrant_payload_index_candidates_plus_neo4j_exact_assertion_validation",
|
||||
"collection": getattr(self, "_bounded_collection", ""),
|
||||
"max_frontier": getattr(self, "_bounded_max_frontier", 0),
|
||||
"per_node_cap": getattr(self, "_bounded_per_node_cap", 0),
|
||||
"max_hops": getattr(self, "_bounded_max_hops", 0),
|
||||
"qdrant_batch_calls": 0,
|
||||
"qdrant_requests": 0,
|
||||
"qdrant_candidates": 0,
|
||||
"neo4j_validation_calls": 0,
|
||||
"neo4j_validated": 0,
|
||||
"qdrant_seconds": 0.0,
|
||||
"neo4j_seconds": 0.0,
|
||||
}
|
||||
|
||||
def _bounded_prepare_question(self, question: str) -> None:
|
||||
self._bounded_question = _text(question)
|
||||
response = requests.post(
|
||||
self._bounded_embed_url,
|
||||
json={
|
||||
"model": self._bounded_embed_model,
|
||||
"input": self._bounded_question[:2400],
|
||||
},
|
||||
timeout=float(os.environ.get("QA_V1_BOUNDED_EMBED_TIMEOUT", "30")),
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
vectors = payload.get("embeddings") or []
|
||||
if vectors:
|
||||
self._bounded_vector = list(vectors[0])
|
||||
elif payload.get("embedding"):
|
||||
self._bounded_vector = list(payload["embedding"])
|
||||
else:
|
||||
raise RuntimeError("bounded graph embedding endpoint returned no vector")
|
||||
|
||||
def _bounded_candidates(
|
||||
self,
|
||||
frontier_ids: Iterable[str],
|
||||
per_node_expand: int,
|
||||
) -> List[Dict[str, Any]]:
|
||||
ids = []
|
||||
seen = set()
|
||||
for value in frontier_ids:
|
||||
canonical_id = _text(value)
|
||||
if (
|
||||
canonical_id.startswith("cent_")
|
||||
and canonical_id not in seen
|
||||
and len(ids) < self._bounded_max_frontier
|
||||
):
|
||||
ids.append(canonical_id)
|
||||
seen.add(canonical_id)
|
||||
if not ids or not self._bounded_vector:
|
||||
return []
|
||||
|
||||
per_node = max(
|
||||
1,
|
||||
min(
|
||||
int(per_node_expand or self._bounded_per_node_cap),
|
||||
self._bounded_per_node_cap,
|
||||
),
|
||||
)
|
||||
requests_batch = [
|
||||
models.QueryRequest(
|
||||
query=self._bounded_vector,
|
||||
filter=models.Filter(
|
||||
must=[
|
||||
models.FieldCondition(
|
||||
key="source_canonical_id",
|
||||
match=models.MatchValue(value=canonical_id),
|
||||
)
|
||||
]
|
||||
),
|
||||
limit=per_node,
|
||||
with_payload=True,
|
||||
with_vector=False,
|
||||
)
|
||||
for canonical_id in ids
|
||||
]
|
||||
|
||||
started = time.perf_counter()
|
||||
responses = self._bounded_qdrant.query_batch_points(
|
||||
collection_name=self._bounded_collection,
|
||||
requests=requests_batch,
|
||||
timeout=int(
|
||||
float(os.environ.get("QA_V1_BOUNDED_QDRANT_TIMEOUT", "20"))
|
||||
),
|
||||
)
|
||||
self._bounded_metrics["qdrant_seconds"] += time.perf_counter() - started
|
||||
self._bounded_metrics["qdrant_batch_calls"] += 1
|
||||
self._bounded_metrics["qdrant_requests"] += len(requests_batch)
|
||||
|
||||
candidates: List[Dict[str, Any]] = []
|
||||
for canonical_id, response in zip(ids, responses):
|
||||
for point in response.points:
|
||||
payload = dict(point.payload or {})
|
||||
if _text(payload.get("source_canonical_id")) != canonical_id:
|
||||
continue
|
||||
payload["_bounded_vector_score"] = float(point.score or 0.0)
|
||||
candidates.append(payload)
|
||||
self._bounded_metrics["qdrant_candidates"] += len(candidates)
|
||||
return self._bounded_validate_assertions(candidates)
|
||||
|
||||
def _bounded_validate_assertions(
|
||||
self, candidates: List[Dict[str, Any]]
|
||||
) -> List[Dict[str, Any]]:
|
||||
by_id: Dict[str, Dict[str, Any]] = {}
|
||||
for row in candidates:
|
||||
assertion_id = _text(row.get("assertion_id") or row.get("relation_id"))
|
||||
if assertion_id.startswith("cass_") and assertion_id not in by_id:
|
||||
by_id[assertion_id] = row
|
||||
if not by_id:
|
||||
return []
|
||||
|
||||
statement = Query(
|
||||
"""
|
||||
UNWIND $assertion_ids AS assertion_id
|
||||
MATCH (a:SRT2CanonicalAssertion {assertion_id: assertion_id})
|
||||
RETURN a.assertion_id AS assertion_id
|
||||
""",
|
||||
timeout=self._bounded_neo4j_timeout,
|
||||
)
|
||||
started = time.perf_counter()
|
||||
with self.driver.session() as session:
|
||||
validated = {
|
||||
_text(record["assertion_id"])
|
||||
for record in session.run(
|
||||
statement, assertion_ids=list(by_id)
|
||||
)
|
||||
}
|
||||
self._bounded_metrics["neo4j_seconds"] += time.perf_counter() - started
|
||||
self._bounded_metrics["neo4j_validation_calls"] += 1
|
||||
self._bounded_metrics["neo4j_validated"] += len(validated)
|
||||
return [by_id[assertion_id] for assertion_id in by_id if assertion_id in validated]
|
||||
|
||||
|
||||
class BoundedCanonicalGraphPathRetriever(
|
||||
_BoundedCanonicalMixin, NativeGraphPathRetriever
|
||||
):
|
||||
"""Native path retriever with bounded, indexed adjacency candidate lookup."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self._bounded_init()
|
||||
|
||||
def retrieve_chains(
|
||||
self,
|
||||
question: str,
|
||||
hypothesis: RetrievalHypothesis,
|
||||
seed_ids: Iterable[str],
|
||||
seed_names: Iterable[str],
|
||||
**kwargs: Any,
|
||||
) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
|
||||
self._bounded_reset_metrics()
|
||||
self._bounded_prepare_question(question)
|
||||
kwargs["max_hops"] = min(
|
||||
int(kwargs.get("max_hops") or self._bounded_max_hops),
|
||||
self._bounded_max_hops,
|
||||
)
|
||||
kwargs["beam_width"] = min(
|
||||
int(kwargs.get("beam_width") or self._bounded_max_frontier),
|
||||
self._bounded_max_frontier,
|
||||
)
|
||||
kwargs["per_node_expand"] = min(
|
||||
int(kwargs.get("per_node_expand") or self._bounded_per_node_cap),
|
||||
self._bounded_per_node_cap,
|
||||
)
|
||||
chains, audit = super().retrieve_chains(
|
||||
question,
|
||||
hypothesis,
|
||||
seed_ids,
|
||||
seed_names,
|
||||
**kwargs,
|
||||
)
|
||||
audit["bounded_graph_backend"] = dict(self._bounded_metrics)
|
||||
return chains, audit
|
||||
|
||||
def _fetch_neighbors(
|
||||
self,
|
||||
frontier_ids: List[str],
|
||||
allowed: set[str],
|
||||
*,
|
||||
relation_clues: List[str],
|
||||
priority_relation_clues: List[str],
|
||||
entity_clues: List[str],
|
||||
answer_type_groups: List[str],
|
||||
per_node_expand: int,
|
||||
) -> List[Dict[str, Any]]:
|
||||
candidates = self._bounded_candidates(frontier_ids, per_node_expand)
|
||||
rows: List[Dict[str, Any]] = []
|
||||
allowed_norm = {_text(value) for value in allowed}
|
||||
relation_terms = [
|
||||
_norm(value) for value in relation_clues if _norm(value)
|
||||
]
|
||||
priority_terms = [
|
||||
_norm(value) for value in priority_relation_clues if _norm(value)
|
||||
]
|
||||
entity_terms = {_norm(value) for value in entity_clues if _norm(value)}
|
||||
answer_terms = [
|
||||
_norm(value) for value in answer_type_groups if _norm(value)
|
||||
]
|
||||
for payload in candidates:
|
||||
compact = _norm(
|
||||
" ".join(
|
||||
[
|
||||
_text(payload.get("source")),
|
||||
_text(payload.get("relation_norm") or payload.get("relation")),
|
||||
_text(payload.get("target")),
|
||||
_text(payload.get("relation_category")),
|
||||
]
|
||||
)
|
||||
)
|
||||
evidence = _norm(
|
||||
payload.get("best_evidence") or payload.get("evidence")
|
||||
)
|
||||
target_norm = _norm(payload.get("target"))
|
||||
relation_score = 0
|
||||
if any(term in compact for term in priority_terms):
|
||||
relation_score = 4
|
||||
elif any(term in compact for term in relation_terms):
|
||||
relation_score = 2
|
||||
elif any(term in evidence for term in relation_terms):
|
||||
relation_score = 1
|
||||
target_type = _norm(payload.get("target_type"))
|
||||
rows.append(
|
||||
{
|
||||
"from_id": payload.get("source_canonical_id"),
|
||||
"from_name": payload.get("source"),
|
||||
"to_id": payload.get("target_canonical_id"),
|
||||
"to_name": payload.get("target"),
|
||||
"to_entity_type": payload.get("target_type"),
|
||||
"assertion_id": payload.get("assertion_id"),
|
||||
"relation_category": payload.get("relation_category") or "",
|
||||
"source": payload.get("source"),
|
||||
"source_type": payload.get("source_type"),
|
||||
"source_canonical_id": payload.get("source_canonical_id"),
|
||||
"relation": payload.get("relation_norm")
|
||||
or payload.get("relation"),
|
||||
"target": payload.get("target"),
|
||||
"target_type": payload.get("target_type"),
|
||||
"target_canonical_id": payload.get("target_canonical_id"),
|
||||
"evidence": payload.get("best_evidence")
|
||||
or payload.get("evidence"),
|
||||
"paper_id": payload.get("best_paper_id")
|
||||
or payload.get("paper_id"),
|
||||
"title": payload.get("title") or "",
|
||||
"pmid": payload.get("pmid") or "",
|
||||
"pmcid": payload.get("pmcid") or "",
|
||||
"doi": payload.get("doi") or "",
|
||||
"species": payload.get("species") or "",
|
||||
"support_count": payload.get("support_count") or 0,
|
||||
"paper_count": payload.get("paper_count") or 0,
|
||||
"best_score": payload.get("best_score")
|
||||
or payload.get("quality_score")
|
||||
or 0.0,
|
||||
"entity_clue_score": int(target_norm in entity_terms),
|
||||
"answer_type_score": int(
|
||||
any(term in target_type for term in answer_terms)
|
||||
),
|
||||
"relation_clue_score": relation_score,
|
||||
"category_match": int(
|
||||
not allowed_norm
|
||||
or _text(payload.get("relation_category"))
|
||||
in allowed_norm
|
||||
),
|
||||
"bounded_vector_score": payload.get(
|
||||
"_bounded_vector_score", 0.0
|
||||
),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
class BoundedCanonicalDirectFlowRetriever(
|
||||
_BoundedCanonicalMixin, DirectFlowRetriever
|
||||
):
|
||||
"""Direct retriever whose graph lane cannot scan complete node adjacency."""
|
||||
|
||||
def __init__(self, **kwargs: Any) -> None:
|
||||
super().__init__(**kwargs)
|
||||
self._bounded_init()
|
||||
self._bounded_seen_assertions: set[str] = set()
|
||||
|
||||
def retrieve(
|
||||
self,
|
||||
question: str,
|
||||
queries: Dict[str, List[str]],
|
||||
hypothesis: RetrievalHypothesis,
|
||||
**kwargs: Any,
|
||||
) -> Tuple[List[EvidenceItem], Dict[str, Any]]:
|
||||
self._bounded_reset_metrics()
|
||||
self._bounded_seen_assertions = set()
|
||||
self._bounded_prepare_question(question)
|
||||
items, audit = super().retrieve(
|
||||
question,
|
||||
queries,
|
||||
hypothesis,
|
||||
**kwargs,
|
||||
)
|
||||
audit["bounded_graph_backend"] = dict(self._bounded_metrics)
|
||||
return items, audit
|
||||
|
||||
def neo4j_seed_search(
|
||||
self,
|
||||
canonical_ids: List[str],
|
||||
names: List[str],
|
||||
limit: int,
|
||||
*,
|
||||
method: str,
|
||||
) -> List[EvidenceItem]:
|
||||
ids = [
|
||||
value
|
||||
for value in canonical_ids
|
||||
if _text(value).startswith("cent_")
|
||||
][: self._bounded_max_frontier]
|
||||
if not ids and names:
|
||||
statement = Query(
|
||||
"""
|
||||
UNWIND $names AS name
|
||||
MATCH (e:SRT2CanonicalEntity {name_norm: name})
|
||||
RETURN e.canonical_id AS canonical_id
|
||||
LIMIT $limit
|
||||
""",
|
||||
timeout=self._bounded_neo4j_timeout,
|
||||
)
|
||||
with self.driver.session() as session:
|
||||
ids = [
|
||||
_text(record["canonical_id"])
|
||||
for record in session.run(
|
||||
statement,
|
||||
names=[_norm(name) for name in names[:20]],
|
||||
limit=self._bounded_max_frontier,
|
||||
)
|
||||
]
|
||||
candidates = self._bounded_candidates(
|
||||
ids,
|
||||
min(
|
||||
self._bounded_per_node_cap,
|
||||
max(1, int(limit or 1) // max(1, len(ids))),
|
||||
),
|
||||
)
|
||||
output: List[EvidenceItem] = []
|
||||
for payload in candidates:
|
||||
assertion_id = _text(
|
||||
payload.get("assertion_id") or payload.get("relation_id")
|
||||
)
|
||||
if assertion_id in self._bounded_seen_assertions:
|
||||
continue
|
||||
self._bounded_seen_assertions.add(assertion_id)
|
||||
item = self.item_from_payload(payload, method, "bounded_canonical")
|
||||
item.vector_score = float(payload.get("_bounded_vector_score") or 0.0)
|
||||
item.rrf_score = max(item.rrf_score, item.vector_score)
|
||||
output.append(item)
|
||||
if len(output) >= min(int(limit or 1), 160):
|
||||
break
|
||||
return output
|
||||
@@ -0,0 +1,196 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
|
||||
_PMID = re.compile(r"^(?:PMID\s*[::]?\s*)?(\d{5,10})$", re.I)
|
||||
_VISIBLE_PMID = re.compile(r"\[PMID\s*[::]\s*(\d{5,10})\]", re.I)
|
||||
_PMCID = re.compile(r"^PMC\d+$", re.I)
|
||||
_DOI = re.compile(r"^10\.\d{4,9}/[-._;()/:A-Z0-9]+$", re.I)
|
||||
_VISIBLE_PMCID = re.compile(r"\[PMCID\s*[::]\s*(PMC\d+)\]", re.I)
|
||||
_VISIBLE_DOI = re.compile(r"\[DOI\s*[::]\s*(10\.\d{4,9}/[-._;()/:A-Z0-9]+)\]", re.I)
|
||||
_VISIBLE_PUBLIC = re.compile(
|
||||
r"\[(PMID|PMCID|DOI)\s*[::]\s*([^\]]+)\]",
|
||||
re.I,
|
||||
)
|
||||
_GROUPED_PUBLIC = re.compile(
|
||||
r"\[((?:PMID|PMCID|DOI)\s*[::]\s*[^\]\n]+)\]",
|
||||
re.I,
|
||||
)
|
||||
_GROUPED_PUBLIC_SEPARATOR = re.compile(
|
||||
r"\s*[,;]\s*(?=(?:PMID|PMCID|DOI)\s*[::])",
|
||||
re.I,
|
||||
)
|
||||
_GROUPED_PUBLIC_PART = re.compile(
|
||||
r"^\s*(PMID|PMCID|DOI)\s*[::]\s*(.+?)\s*$",
|
||||
re.I,
|
||||
)
|
||||
_CITATION_LIKE = re.compile(r"\[(PMID|PMCID|DOI)([^\]\n]*)(\]|$)", re.I)
|
||||
_FULL_WIDTH_PUBLIC_CITATION = re.compile(
|
||||
r"【(?:PMID|PMCID|DOI)\b[^】\n]*】",
|
||||
re.I,
|
||||
)
|
||||
_INTERNAL_LABELS = (
|
||||
re.compile(r"\[(?:S\d+|G\d+\.H\d+)\]", re.I),
|
||||
re.compile(r"\[Data\s*:\s*(?:Sources|GraphChains)[^\]]*\]", re.I),
|
||||
re.compile(r"\bG\d+\.H\d+\b", re.I),
|
||||
)
|
||||
|
||||
|
||||
def normalize_pmid(value: Any) -> str:
|
||||
match = _PMID.fullmatch(str(value or "").strip())
|
||||
return match.group(1) if match else ""
|
||||
|
||||
|
||||
def public_citation(*, evidence_id: str, pmid: str, pmcid: str, doi: str) -> str:
|
||||
normalized = normalize_pmid(pmid)
|
||||
if normalized:
|
||||
return f"[PMID: {normalized}]"
|
||||
clean_pmcid = str(pmcid or "").strip().upper()
|
||||
if _PMCID.fullmatch(clean_pmcid):
|
||||
return f"[PMCID: {clean_pmcid}]"
|
||||
clean_doi = str(doi or "").strip()
|
||||
if _DOI.fullmatch(clean_doi):
|
||||
return f"[DOI: {clean_doi}]"
|
||||
return ""
|
||||
|
||||
|
||||
def cited_pmids(answer: str) -> set[str]:
|
||||
return set(_VISIBLE_PMID.findall(str(answer or "")))
|
||||
|
||||
|
||||
def visible_public_citations(answer: str) -> list[str]:
|
||||
citations: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for kind, raw_value in _VISIBLE_PUBLIC.findall(str(answer or "")):
|
||||
kind = kind.upper()
|
||||
value = str(raw_value or "").strip()
|
||||
if kind == "PMID":
|
||||
normalized = normalize_pmid(value)
|
||||
citation = f"[PMID: {normalized}]" if normalized else ""
|
||||
elif kind == "PMCID":
|
||||
normalized = value.upper()
|
||||
citation = f"[PMCID: {normalized}]" if _PMCID.fullmatch(normalized) else ""
|
||||
else:
|
||||
citation = f"[DOI: {value}]" if _DOI.fullmatch(value) else ""
|
||||
if citation and citation not in seen:
|
||||
citations.append(citation)
|
||||
seen.add(citation)
|
||||
return citations
|
||||
|
||||
|
||||
def normalize_grouped_public_citations(
|
||||
answer: str,
|
||||
allowed_citations: set[str],
|
||||
) -> tuple[str, list[dict[str, object]]]:
|
||||
"""Split a citation group only when every identifier is already allowlisted."""
|
||||
|
||||
allowed = set(allowed_citations)
|
||||
audit: list[dict[str, object]] = []
|
||||
|
||||
def replace(match: re.Match[str]) -> str:
|
||||
original = match.group(0)
|
||||
parts = _GROUPED_PUBLIC_SEPARATOR.split(match.group(1))
|
||||
if len(parts) < 2:
|
||||
return original
|
||||
|
||||
normalized: list[str] = []
|
||||
for part in parts:
|
||||
parsed = _GROUPED_PUBLIC_PART.fullmatch(part)
|
||||
if not parsed:
|
||||
return original
|
||||
kind, value = parsed.groups()
|
||||
citations = visible_public_citations(
|
||||
f"[{kind.upper()}: {str(value or '').strip()}]"
|
||||
)
|
||||
if len(citations) != 1 or citations[0] not in allowed:
|
||||
return original
|
||||
if citations[0] not in normalized:
|
||||
normalized.append(citations[0])
|
||||
|
||||
replacement = " ".join(normalized)
|
||||
audit.append(
|
||||
{
|
||||
"original": original,
|
||||
"replacement": replacement,
|
||||
"citation_count": len(normalized),
|
||||
}
|
||||
)
|
||||
return replacement
|
||||
|
||||
normalized_answer = _GROUPED_PUBLIC.sub(replace, str(answer or ""))
|
||||
return normalized_answer, audit
|
||||
|
||||
|
||||
def internal_answer_labels(answer: str) -> list[str]:
|
||||
text = str(answer or "")
|
||||
return sorted({match.group(0) for pattern in _INTERNAL_LABELS for match in pattern.finditer(text)})
|
||||
|
||||
|
||||
def malformed_citation_labels(answer: str) -> list[str]:
|
||||
malformed: set[str] = set()
|
||||
text = str(answer or "")
|
||||
malformed.update(
|
||||
match.group(0) for match in _FULL_WIDTH_PUBLIC_CITATION.finditer(text)
|
||||
)
|
||||
for match in _CITATION_LIKE.finditer(text):
|
||||
kind = match.group(1).upper()
|
||||
tail = match.group(2).strip()
|
||||
closing = match.group(3)
|
||||
if closing != "]":
|
||||
malformed.add(match.group(0))
|
||||
continue
|
||||
value = re.sub(r"^[::]\s*", "", tail)
|
||||
if kind == "PMID":
|
||||
identifier_shaped = bool(re.fullmatch(r"\d{5,10}", value))
|
||||
elif kind == "PMCID":
|
||||
identifier_shaped = bool(re.fullmatch(r"PMC\d+", value, re.I))
|
||||
else:
|
||||
identifier_shaped = bool(re.fullmatch(r"10\.\d{4,9}/\S+", value, re.I))
|
||||
has_colon = bool(re.match(r"^[::]", tail))
|
||||
if not identifier_shaped or not has_colon:
|
||||
malformed.add(match.group(0))
|
||||
return sorted(malformed)
|
||||
|
||||
|
||||
def validate_answer_citations(answer: str, allowed_citations: set[str]) -> dict[str, object]:
|
||||
cited = visible_public_citations(answer)
|
||||
allowed = set(allowed_citations)
|
||||
unknown = [citation for citation in cited if citation not in allowed]
|
||||
invalid_public: list[str] = []
|
||||
for kind, value in _VISIBLE_PUBLIC.findall(str(answer or "")):
|
||||
raw_citation = f"[{kind.upper()}: {str(value or '').strip()}]"
|
||||
if not visible_public_citations(raw_citation):
|
||||
invalid_public.append(raw_citation)
|
||||
internal = internal_answer_labels(answer)
|
||||
malformed = malformed_citation_labels(answer)
|
||||
cited_pmids_values = [match for match in _VISIBLE_PMID.findall(str(answer or ""))]
|
||||
unknown_pmids = [value for value in cited_pmids_values if f"[PMID: {value}]" in unknown]
|
||||
cited_pmcids = [value.upper() for value in _VISIBLE_PMCID.findall(str(answer or ""))]
|
||||
unknown_pmcids = [value for value in cited_pmcids if f"[PMCID: {value}]" in unknown]
|
||||
cited_dois = list(_VISIBLE_DOI.findall(str(answer or "")))
|
||||
unknown_dois = [value for value in cited_dois if f"[DOI: {value}]" in unknown]
|
||||
return {
|
||||
"valid": not unknown and not invalid_public and not malformed and not internal,
|
||||
"cited_pmids": sorted(set(cited_pmids_values)),
|
||||
"unknown_pmids": sorted(set(unknown_pmids)),
|
||||
"cited_pmcids": sorted(set(cited_pmcids)),
|
||||
"unknown_pmcids": sorted(set(unknown_pmcids)),
|
||||
"cited_dois": sorted(set(cited_dois)),
|
||||
"unknown_dois": sorted(set(unknown_dois)),
|
||||
"invalid_public_citations": sorted(set(invalid_public)),
|
||||
"malformed_citation_labels": malformed,
|
||||
"internal_labels": internal,
|
||||
}
|
||||
|
||||
|
||||
def validate_answer_pmids(answer: str, allowed_pmids: set[str]) -> dict[str, object]:
|
||||
cited = cited_pmids(answer)
|
||||
allowed = {value for value in (normalize_pmid(x) for x in allowed_pmids) if value}
|
||||
unknown = sorted(cited - allowed)
|
||||
return {
|
||||
"valid": not unknown,
|
||||
"cited_pmids": sorted(cited),
|
||||
"unknown_pmids": unknown,
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Iterable
|
||||
|
||||
from .schemas import ClaimContext, EvidenceItem
|
||||
|
||||
|
||||
_HTML_TAG = re.compile(r"<[^>]+>")
|
||||
_BINOMIAL = r"[A-Z][a-z]{2,}\s+[a-z][a-z-]{2,}"
|
||||
_NON_TAXON_WORDS = {
|
||||
"development",
|
||||
"factors",
|
||||
"fruit",
|
||||
"growth",
|
||||
"hormones",
|
||||
"plant",
|
||||
"root",
|
||||
"tomato",
|
||||
"transcription",
|
||||
}
|
||||
|
||||
|
||||
def _valid_structured_species(value: str) -> bool:
|
||||
if not re.fullmatch(_BINOMIAL, value):
|
||||
return False
|
||||
return not ({part.casefold() for part in value.split()} & _NON_TAXON_WORDS)
|
||||
|
||||
|
||||
def _evidence_species(text: str, structured_species: str) -> list[str]:
|
||||
clean = _HTML_TAG.sub(" ", str(text or ""))
|
||||
if _valid_structured_species(structured_species) and structured_species in clean:
|
||||
return [structured_species]
|
||||
return []
|
||||
|
||||
|
||||
def _title_species(title: str) -> list[str]:
|
||||
clean = _HTML_TAG.sub(" ", str(title or ""))
|
||||
values: list[str] = []
|
||||
for match in re.finditer(rf"\(({_BINOMIAL})(?:\s+[A-Z]\.)?\)", clean):
|
||||
value = " ".join(match.group(1).split())
|
||||
if _valid_structured_species(value) and value not in values:
|
||||
values.append(value)
|
||||
return values
|
||||
|
||||
|
||||
def _first_raw_value(item: EvidenceItem, fields: Iterable[str]) -> str:
|
||||
raw = item.raw if isinstance(item.raw, dict) else {}
|
||||
for field in fields:
|
||||
value = " ".join(str(raw.get(field) or "").split())
|
||||
if value:
|
||||
return value
|
||||
return ""
|
||||
|
||||
|
||||
def build_claim_context(item: EvidenceItem) -> ClaimContext:
|
||||
evidence_text = str(item.evidence or "")
|
||||
supplied_species = " ".join(str(item.species or "").split())
|
||||
claim_species = _evidence_species(evidence_text, supplied_species)
|
||||
paper_species = _title_species(item.title)
|
||||
if (
|
||||
_valid_structured_species(supplied_species)
|
||||
and supplied_species in str(item.title or "")
|
||||
and supplied_species not in paper_species
|
||||
):
|
||||
paper_species.append(supplied_species)
|
||||
if claim_species:
|
||||
species_source = "evidence"
|
||||
confidence = 1.0
|
||||
elif paper_species:
|
||||
species_source = "title_only"
|
||||
confidence = 0.5
|
||||
else:
|
||||
species_source = "none"
|
||||
confidence = 0.0
|
||||
|
||||
return ClaimContext(
|
||||
claim_species=claim_species,
|
||||
paper_species=paper_species,
|
||||
species_source=species_source,
|
||||
subject=" ".join(str(item.source or "").split()),
|
||||
function_or_process=" ".join(str(item.target or "").split()),
|
||||
condition=_first_raw_value(item, ("condition", "treatment", "context")),
|
||||
direction=" ".join(str(item.relation or "").split()),
|
||||
confidence=confidence,
|
||||
)
|
||||
@@ -0,0 +1,49 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from .evidence_pack import format_evidence_pack
|
||||
from .llm_client import LLMClient
|
||||
from .schemas import AtomicClaim, EvidenceItem
|
||||
|
||||
|
||||
class ClaimVerifier:
|
||||
SYSTEM = """You verify claims from an unverified provisional plant-biology answer.
|
||||
|
||||
Use only supplied evidence. A claim may be supported, partially_supported,
|
||||
unsupported, contradicted, context_mismatch, or not_retrieved. Do not treat
|
||||
not_retrieved as false. Preserve species and experimental context. Return JSON
|
||||
only."""
|
||||
|
||||
def __init__(self, llm: LLMClient):
|
||||
self.llm = llm
|
||||
|
||||
def verify(
|
||||
self, claims: List[AtomicClaim], evidence: List[EvidenceItem]
|
||||
) -> List[Dict[str, Any]]:
|
||||
if not claims:
|
||||
return []
|
||||
payload = self.llm.complete_json(
|
||||
self.SYSTEM,
|
||||
f"""Claims:
|
||||
{[{"claim_id": c.claim_id, "text": c.text} for c in claims]}
|
||||
|
||||
Evidence:
|
||||
{format_evidence_pack(evidence)}
|
||||
|
||||
Return:
|
||||
{{
|
||||
"claims": [
|
||||
{{
|
||||
"claim_id":"H1",
|
||||
"status":"supported|partially_supported|unsupported|contradicted|context_mismatch|not_retrieved",
|
||||
"supporting_evidence_ids":["S1"],
|
||||
"contradicting_evidence_ids":[],
|
||||
"reason":""
|
||||
}}
|
||||
]
|
||||
}}""",
|
||||
max_tokens=2600,
|
||||
)
|
||||
rows = payload.get("claims") if isinstance(payload, dict) else []
|
||||
return rows if isinstance(rows, list) else []
|
||||
@@ -0,0 +1,150 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, Iterable, List, Tuple
|
||||
|
||||
from .schemas import EvidenceItem
|
||||
|
||||
|
||||
def _norm(value: Any) -> str:
|
||||
text = str(value or "").lower()
|
||||
text = re.sub(r"[^a-z0-9\u4e00-\u9fff]+", " ", text)
|
||||
return " ".join(text.split())
|
||||
|
||||
|
||||
def _cid(item: EvidenceItem, field: str, fallback: str) -> str:
|
||||
raw = item.raw or {}
|
||||
value = raw.get(field)
|
||||
if value:
|
||||
return str(value)
|
||||
return _norm(fallback)
|
||||
|
||||
|
||||
def relation_intent(item: EvidenceItem) -> str:
|
||||
raw = item.raw or {}
|
||||
rel = _norm(item.relation or raw.get("relation_norm"))
|
||||
cat = _norm(raw.get("relation_category"))
|
||||
text = f"{rel} {cat}"
|
||||
|
||||
if any(k in text for k in ["express", "upregulated", "downregulated", "accumulat", "induced"]):
|
||||
return "expression"
|
||||
if any(k in text for k in ["localiz", "located", "targeted_to", "subcellular"]):
|
||||
return "localization"
|
||||
if any(k in text for k in ["cis_element", "motif", "domain", "intron", "exon", "gene_structure", "structure"]):
|
||||
return "structure"
|
||||
if any(k in text for k in ["stress", "tolerance", "resistance", "pathogen", "defense", "salt", "drought", "cold", "heat"]):
|
||||
return "stress_response"
|
||||
if any(k in text for k in ["classification", "classified", "belongs", "member", "include", "contains", "comprise", "is_a", "clade"]):
|
||||
return "classification"
|
||||
if any(k in text for k in ["count", "catalog", "identified", "number"]):
|
||||
return "catalog_count"
|
||||
if any(k in text for k in ["ortholog", "homolog", "synten", "collinear"]):
|
||||
return "orthology_homology"
|
||||
if any(k in text for k in ["regulat", "promot", "inhibit", "affect", "function", "role", "involved", "participat"]):
|
||||
return "function_effect"
|
||||
return cat or rel or "other"
|
||||
|
||||
|
||||
def cluster_key(item: EvidenceItem) -> Tuple[str, str, str, str]:
|
||||
source = _cid(item, "source_canonical_id", item.source)
|
||||
target = _cid(item, "target_canonical_id", item.target)
|
||||
intent = relation_intent(item)
|
||||
layer = _norm(item.output_layer or (item.raw or {}).get("relation_category") or "")
|
||||
return source, intent, target, layer
|
||||
|
||||
|
||||
def cluster_id_for(key: Tuple[str, str, str, str]) -> str:
|
||||
digest = hashlib.sha1("|".join(key).encode("utf-8")).hexdigest()[:16]
|
||||
return f"clu_{digest}"
|
||||
|
||||
|
||||
def _representative_score(item: EvidenceItem) -> Tuple[float, float, float, float]:
|
||||
raw = item.raw or {}
|
||||
has_evidence = 1.0 if item.evidence else 0.0
|
||||
layer_score = 0.0
|
||||
layer = _norm(item.output_layer)
|
||||
if layer == "main_kg":
|
||||
layer_score = 1.0
|
||||
elif "background" in layer:
|
||||
layer_score = 0.7
|
||||
elif "candidate" in layer:
|
||||
layer_score = 0.35
|
||||
support = float(raw.get("support_count") or 0.0)
|
||||
paper_count = float(raw.get("paper_count") or 0.0)
|
||||
return (
|
||||
has_evidence,
|
||||
layer_score,
|
||||
support + 0.25 * paper_count,
|
||||
max(float(item.rrf_score or 0.0), float(item.vector_score or 0.0), float(item.final_score or 0.0)),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EvidenceCluster:
|
||||
cluster_id: str
|
||||
key: Tuple[str, str, str, str]
|
||||
representative: EvidenceItem
|
||||
members: List[EvidenceItem] = field(default_factory=list)
|
||||
|
||||
def finalize(self) -> EvidenceItem:
|
||||
self.members.sort(key=_representative_score, reverse=True)
|
||||
self.representative = self.members[0]
|
||||
member_ids = [m.relation_id for m in self.members if m.relation_id]
|
||||
pmids = []
|
||||
paper_ids = []
|
||||
for m in self.members:
|
||||
if m.pmid and m.pmid not in pmids:
|
||||
pmids.append(m.pmid)
|
||||
if m.paper_id and m.paper_id not in paper_ids:
|
||||
paper_ids.append(m.paper_id)
|
||||
raw = dict(self.representative.raw or {})
|
||||
raw["cluster_id"] = self.cluster_id
|
||||
raw["cluster_key"] = list(self.key)
|
||||
raw["cluster_member_assertion_ids"] = member_ids
|
||||
raw["cluster_member_count"] = len(member_ids)
|
||||
raw["cluster_member_paper_ids"] = paper_ids[:30]
|
||||
raw["cluster_member_pmids"] = pmids[:30]
|
||||
self.representative.raw = raw
|
||||
self.representative.cluster_id = self.cluster_id
|
||||
self.representative.cluster_member_count = len(member_ids)
|
||||
self.representative.cluster_member_assertion_ids = member_ids
|
||||
self.representative.cluster_member_paper_ids = paper_ids[:30]
|
||||
self.representative.cluster_member_pmids = pmids[:30]
|
||||
return self.representative
|
||||
|
||||
|
||||
def build_clusters(items: Iterable[EvidenceItem]) -> Tuple[List[EvidenceItem], List[Dict[str, Any]]]:
|
||||
clusters: Dict[Tuple[str, str, str, str], EvidenceCluster] = {}
|
||||
for item in items:
|
||||
key = cluster_key(item)
|
||||
cid = cluster_id_for(key)
|
||||
cluster = clusters.get(key)
|
||||
if cluster is None:
|
||||
cluster = EvidenceCluster(cluster_id=cid, key=key, representative=item)
|
||||
clusters[key] = cluster
|
||||
cluster.members.append(item)
|
||||
|
||||
representatives: List[EvidenceItem] = []
|
||||
audit: List[Dict[str, Any]] = []
|
||||
for cluster in clusters.values():
|
||||
rep = cluster.finalize()
|
||||
representatives.append(rep)
|
||||
audit.append(
|
||||
{
|
||||
"cluster_id": cluster.cluster_id,
|
||||
"cluster_key": list(cluster.key),
|
||||
"representative_assertion_id": rep.relation_id,
|
||||
"representative_srt": {
|
||||
"source": rep.source,
|
||||
"relation": rep.relation,
|
||||
"target": rep.target,
|
||||
},
|
||||
"member_count": len(cluster.members),
|
||||
"member_assertion_ids": [m.relation_id for m in cluster.members if m.relation_id],
|
||||
"member_paper_ids": list(dict.fromkeys([m.paper_id for m in cluster.members if m.paper_id]))[:30],
|
||||
}
|
||||
)
|
||||
representatives.sort(key=_representative_score, reverse=True)
|
||||
return representatives, audit
|
||||
@@ -0,0 +1,102 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QAConfig:
|
||||
project_root: Path
|
||||
gfanalysis_root: Path
|
||||
alias_path: Path
|
||||
taxonomy_path: Path
|
||||
reranker_model: Path
|
||||
initial_recall_per_query: int
|
||||
max_query_variants: int
|
||||
max_evidence: int
|
||||
min_evidence: int
|
||||
max_rounds: int
|
||||
graph_hops: int
|
||||
include_candidate: bool
|
||||
enable_provisional_draft: bool
|
||||
enable_second_round: bool
|
||||
enable_bge_rerank: bool
|
||||
graph_schema: str
|
||||
enable_agentic_controller: bool = True
|
||||
agentic_max_replans: int = 2
|
||||
enable_question_decomposition: bool = True
|
||||
subquestion_workers: int = 3
|
||||
vector_query_workers: int = 2
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "QAConfig":
|
||||
root = Path(os.environ.get("GENE_FAMILY_ROOT", "/data/gene-family-mas-v2"))
|
||||
graph_schema = os.environ.get(
|
||||
"QA_V1_GRAPH_SCHEMA", "canonical"
|
||||
).strip().lower()
|
||||
if graph_schema != "canonical":
|
||||
raise RuntimeError(
|
||||
"QA V1.1.73 production only permits the canonical graph schema"
|
||||
)
|
||||
return cls(
|
||||
project_root=root,
|
||||
gfanalysis_root=Path(
|
||||
os.environ.get("GFANALYSIS_ROOT", "/data/wutx/00test/gfAnalysis")
|
||||
),
|
||||
alias_path=Path(
|
||||
os.environ.get(
|
||||
"QA_V1_ALIAS_CONFIG",
|
||||
root / "configs" / "rag" / "entity_alias.yaml",
|
||||
)
|
||||
),
|
||||
taxonomy_path=Path(
|
||||
os.environ.get(
|
||||
"QA_V1_TAXONOMY_CONFIG",
|
||||
root / "configs" / "rag" / "relation_taxonomy.yaml",
|
||||
)
|
||||
),
|
||||
reranker_model=Path(
|
||||
os.environ.get(
|
||||
"QA_V1_BGE_RERANKER_MODEL",
|
||||
root / "bge-reranker-base",
|
||||
)
|
||||
),
|
||||
initial_recall_per_query=int(
|
||||
os.environ.get("QA_V1_INITIAL_RECALL_PER_QUERY", "50")
|
||||
),
|
||||
max_query_variants=int(
|
||||
os.environ.get("QA_V1_MAX_QUERY_VARIANTS", "6")
|
||||
),
|
||||
max_evidence=int(os.environ.get("QA_V1_MAX_EVIDENCE", "40")),
|
||||
min_evidence=int(os.environ.get("QA_V1_MIN_EVIDENCE", "8")),
|
||||
max_rounds=max(1, min(int(os.environ.get("QA_V1_MAX_ROUNDS", "2")), 2)),
|
||||
graph_hops=max(1, min(int(os.environ.get("QA_V1_GRAPH_HOPS", "2")), 2)),
|
||||
include_candidate=os.environ.get("QA_V1_INCLUDE_CANDIDATE", "0") == "1",
|
||||
enable_provisional_draft=os.environ.get(
|
||||
"QA_V1_ENABLE_PROVISIONAL_DRAFT", "1"
|
||||
)
|
||||
== "1",
|
||||
enable_second_round=os.environ.get("QA_V1_ENABLE_SECOND_ROUND", "1")
|
||||
== "1",
|
||||
enable_bge_rerank=os.environ.get("QA_V1_ENABLE_BGE_RERANK", "1")
|
||||
== "1",
|
||||
graph_schema=graph_schema,
|
||||
enable_agentic_controller=os.environ.get(
|
||||
"QA_V1_ENABLE_AGENTIC_CONTROLLER", "1"
|
||||
)
|
||||
== "1",
|
||||
agentic_max_replans=max(
|
||||
0, min(int(os.environ.get("QA_V1_AGENTIC_MAX_REPLANS", "2")), 2)
|
||||
),
|
||||
enable_question_decomposition=os.environ.get(
|
||||
"QA_ENABLE_QUESTION_DECOMPOSITION", "1"
|
||||
)
|
||||
== "1",
|
||||
subquestion_workers=max(
|
||||
1, min(int(os.environ.get("QA_V1_SUBQUESTION_WORKERS", "3")), 3)
|
||||
),
|
||||
vector_query_workers=max(
|
||||
1, min(int(os.environ.get("QA_V1_VECTOR_QUERY_WORKERS", "2")), 2)
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,406 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Tuple
|
||||
|
||||
import requests
|
||||
from neo4j import GraphDatabase
|
||||
from qdrant_client import QdrantClient
|
||||
|
||||
from .schemas import EvidenceItem, RetrievalHypothesis
|
||||
|
||||
ALLOWED_QDRANT_COLLECTIONS = {
|
||||
"plant_srt_direct_v1_bge_m3_20260702",
|
||||
"plant_srt_direct_v1_bge_m3_citations_20260717",
|
||||
}
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return " ".join(str(value or "").split())
|
||||
|
||||
|
||||
def _norm(value: Any) -> str:
|
||||
import re
|
||||
|
||||
text = str(value or "").lower()
|
||||
text = re.sub(r"[^a-z0-9\u4e00-\u9fff]+", " ", text)
|
||||
return " ".join(text.split())
|
||||
|
||||
|
||||
def load_paper_metadata_registry(path: Path) -> Dict[str, Dict[str, Any]]:
|
||||
records: Dict[str, Dict[str, Any]] = {}
|
||||
if not path.is_file():
|
||||
return records
|
||||
with path.open("r", encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
if not line.strip():
|
||||
continue
|
||||
row = json.loads(line)
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
paper_id = _text(row.get("paper_id"))
|
||||
if paper_id and paper_id not in records:
|
||||
records[paper_id] = row
|
||||
return records
|
||||
|
||||
|
||||
class DirectFlowRetriever:
|
||||
"""Direct V15/V16 canonical Qdrant + Neo4j retriever.
|
||||
|
||||
This retriever deliberately avoids legacy SQLite/PC lanes and avoids graph
|
||||
scans. Neo4j expansion starts only from canonical IDs or exact normalized
|
||||
names. Qdrant and Neo4j share the cass_* assertion_id/relation_id.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
graph_schema: str | None = None,
|
||||
vector_query_workers: int = 2,
|
||||
) -> None:
|
||||
self.graph_schema = (
|
||||
graph_schema or os.environ.get("QA_V1_GRAPH_SCHEMA", "canonical")
|
||||
).strip().lower()
|
||||
if self.graph_schema != "canonical":
|
||||
raise RuntimeError(
|
||||
"Legacy Entity/RELATION and PCEntity/PC_REL graphs are disabled"
|
||||
)
|
||||
self.vector_query_workers = max(1, min(int(vector_query_workers), 2))
|
||||
self.qdrant_url = os.environ.get("V15_QDRANT_URL", os.environ.get("QDRANT_URL", "http://127.0.0.1:6333"))
|
||||
self.collection = os.environ.get("V15_QDRANT_COLLECTION", "plant_srt_direct_v1_bge_m3_20260702")
|
||||
if self.collection not in ALLOWED_QDRANT_COLLECTIONS:
|
||||
raise RuntimeError(
|
||||
f"Qdrant collection {self.collection!r} is not allowed in production"
|
||||
)
|
||||
self.embed_url = os.environ.get("QA_V112_EMBED_URL", os.environ.get("QA_V1_EMBED_URL", "http://127.0.0.1:11434/api/embed"))
|
||||
self.embed_model = os.environ.get("QA_V112_EMBED_MODEL", os.environ.get("QA_V1_EMBED_MODEL", "bge-m3:567m"))
|
||||
self.neo4j_uri = os.environ.get("NEO4J_URI", "bolt://127.0.0.1:18788")
|
||||
self.neo4j_user = os.environ.get("NEO4J_USER", "neo4j")
|
||||
self.neo4j_password = os.environ.get("NEO4J_PASSWORD", "plant_srt_direct_password")
|
||||
registry_path = os.environ.get("QA_PAPER_METADATA_REGISTRY", "").strip()
|
||||
self.paper_metadata = (
|
||||
load_paper_metadata_registry(Path(registry_path)) if registry_path else {}
|
||||
)
|
||||
self.qdrant = QdrantClient(url=self.qdrant_url, timeout=60)
|
||||
self.driver = GraphDatabase.driver(
|
||||
self.neo4j_uri,
|
||||
auth=(self.neo4j_user, self.neo4j_password),
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
if self.driver is not None:
|
||||
self.driver.close()
|
||||
|
||||
def retrieve(
|
||||
self,
|
||||
question: str,
|
||||
queries: Dict[str, List[str]],
|
||||
hypothesis: RetrievalHypothesis,
|
||||
*,
|
||||
qdrant_top_k: int = 120,
|
||||
graph_top_k: int = 80,
|
||||
use_qdrant: bool = True,
|
||||
use_neo4j: bool = True,
|
||||
use_graph_expansion: bool = True,
|
||||
) -> Tuple[List[EvidenceItem], Dict[str, Any]]:
|
||||
effective_use_neo4j = bool(use_neo4j) and self.graph_schema == "canonical"
|
||||
effective_use_graph_expansion = (
|
||||
bool(use_graph_expansion) and effective_use_neo4j
|
||||
)
|
||||
audit: Dict[str, Any] = {
|
||||
"qdrant": [],
|
||||
"seed_extraction": {},
|
||||
"neo4j": [],
|
||||
"dedup": {},
|
||||
"options": {
|
||||
"qdrant_top_k": qdrant_top_k,
|
||||
"graph_top_k": graph_top_k,
|
||||
"use_qdrant": use_qdrant,
|
||||
"use_neo4j": use_neo4j,
|
||||
"use_graph_expansion": use_graph_expansion,
|
||||
"effective_use_neo4j": effective_use_neo4j,
|
||||
"effective_use_graph_expansion": effective_use_graph_expansion,
|
||||
},
|
||||
"neo4j_status": (
|
||||
"enabled"
|
||||
if effective_use_neo4j
|
||||
else "disabled_for_legacy_schema"
|
||||
if self.graph_schema == "legacy"
|
||||
else "disabled_by_request"
|
||||
),
|
||||
}
|
||||
all_queries = self._flatten_queries(question, queries)
|
||||
qdrant_items: List[EvidenceItem] = []
|
||||
if use_qdrant:
|
||||
per_query = max(10, min(qdrant_top_k // max(1, len(all_queries)), 50))
|
||||
if self.vector_query_workers == 1 or len(all_queries) == 1:
|
||||
query_rows = [
|
||||
(query, self.qdrant_search(query, per_query))
|
||||
for query in all_queries
|
||||
]
|
||||
else:
|
||||
with ThreadPoolExecutor(
|
||||
max_workers=min(self.vector_query_workers, len(all_queries)),
|
||||
thread_name_prefix="qa-vector",
|
||||
) as executor:
|
||||
futures = [
|
||||
executor.submit(self.qdrant_search, query, per_query)
|
||||
for query in all_queries
|
||||
]
|
||||
query_rows = [
|
||||
(query, future.result())
|
||||
for query, future in zip(all_queries, futures)
|
||||
]
|
||||
for query, rows in query_rows:
|
||||
audit["qdrant"].append({"query": query, "rows": len(rows)})
|
||||
qdrant_items.extend(rows)
|
||||
|
||||
seed_ids, seed_names = self.extract_seeds(hypothesis, qdrant_items)
|
||||
audit["seed_extraction"] = {
|
||||
"canonical_ids": seed_ids[:50],
|
||||
"names": seed_names[:50],
|
||||
"canonical_id_count": len(seed_ids),
|
||||
"name_count": len(seed_names),
|
||||
}
|
||||
|
||||
graph_items: List[EvidenceItem] = []
|
||||
if effective_use_neo4j:
|
||||
direct_rows = self.neo4j_seed_search(seed_ids, seed_names, graph_top_k, method="neo4j_seed_exact")
|
||||
audit["neo4j"].append({"stage": "seed_exact", "rows": len(direct_rows)})
|
||||
graph_items.extend(direct_rows)
|
||||
if effective_use_graph_expansion:
|
||||
expansion_seed_ids = self._ids_from_items((qdrant_items[:30] + direct_rows[:30]))
|
||||
expansion_rows = self.neo4j_seed_search(expansion_seed_ids, [], graph_top_k, method="neo4j_bounded_expand")
|
||||
audit["neo4j"].append({"stage": "bounded_expand", "rows": len(expansion_rows)})
|
||||
graph_items.extend(expansion_rows)
|
||||
|
||||
merged = self.dedup(qdrant_items + graph_items)
|
||||
audit["dedup"] = {
|
||||
"qdrant_items": len(qdrant_items),
|
||||
"graph_items": len(graph_items),
|
||||
"merged_items": len(merged),
|
||||
}
|
||||
return merged, audit
|
||||
|
||||
def qdrant_search(self, query: str, limit: int) -> List[EvidenceItem]:
|
||||
try:
|
||||
vector = self.embed(query)
|
||||
response = self.qdrant.query_points(
|
||||
collection_name=self.collection,
|
||||
query=vector,
|
||||
limit=limit,
|
||||
with_payload=True,
|
||||
)
|
||||
except Exception:
|
||||
return []
|
||||
output: List[EvidenceItem] = []
|
||||
for point in response.points:
|
||||
payload = dict(point.payload or {})
|
||||
item = self.item_from_payload(payload, "qdrant_vector", query)
|
||||
item.vector_score = float(point.score or 0.0)
|
||||
item.rrf_score = max(item.rrf_score, item.vector_score)
|
||||
output.append(item)
|
||||
return output
|
||||
|
||||
def embed(self, text: str) -> List[float]:
|
||||
response = requests.post(
|
||||
self.embed_url,
|
||||
json={"model": self.embed_model, "input": text[:2400]},
|
||||
timeout=60,
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
embeddings = data.get("embeddings") or []
|
||||
if embeddings:
|
||||
return embeddings[0]
|
||||
if "embedding" in data:
|
||||
return data["embedding"]
|
||||
raise RuntimeError("embedding endpoint returned no vector")
|
||||
|
||||
def neo4j_seed_search(
|
||||
self,
|
||||
canonical_ids: List[str],
|
||||
names: List[str],
|
||||
limit: int,
|
||||
*,
|
||||
method: str,
|
||||
) -> List[EvidenceItem]:
|
||||
if not canonical_ids and not names:
|
||||
return []
|
||||
if self.driver is None:
|
||||
raise RuntimeError("canonical Neo4j search is unavailable for legacy schema")
|
||||
statement = """
|
||||
MATCH (e:SRT2CanonicalEntity)
|
||||
WHERE e.canonical_id IN $ids OR e.name_norm IN $names OR toLower(e.name) IN $names
|
||||
WITH e LIMIT 30
|
||||
MATCH (e)-[r:SRT2_CANONICAL_ASSERTS]-(n:SRT2CanonicalEntity)
|
||||
MATCH (a:SRT2CanonicalAssertion {assertion_id: r.assertion_id})
|
||||
RETURN a.assertion_id AS assertion_id,
|
||||
a.source AS source,
|
||||
a.source_type AS source_type,
|
||||
a.source_canonical_id AS source_canonical_id,
|
||||
a.relation_norm AS relation,
|
||||
a.relation_category AS relation_category,
|
||||
a.target AS target,
|
||||
a.target_type AS target_type,
|
||||
a.target_canonical_id AS target_canonical_id,
|
||||
a.best_evidence AS evidence,
|
||||
a.best_paper_id AS paper_id,
|
||||
a.title AS title,
|
||||
a.doi AS doi,
|
||||
a.pmid AS pmid,
|
||||
a.pmcid AS pmcid,
|
||||
a.best_score AS score,
|
||||
a.support_count AS support_count,
|
||||
a.paper_count AS paper_count,
|
||||
e.canonical_id AS matched_canonical_id,
|
||||
e.name AS matched_canonical_name
|
||||
ORDER BY coalesce(a.paper_count, 0) DESC,
|
||||
coalesce(a.support_count, 0) DESC,
|
||||
coalesce(a.best_score, 0.0) DESC
|
||||
LIMIT $limit
|
||||
"""
|
||||
try:
|
||||
with self.driver.session() as session:
|
||||
rows = [
|
||||
dict(record)
|
||||
for record in session.run(
|
||||
statement,
|
||||
ids=canonical_ids[:80],
|
||||
names=names[:80],
|
||||
limit=min(limit, 160),
|
||||
)
|
||||
]
|
||||
except Exception:
|
||||
return []
|
||||
output: List[EvidenceItem] = []
|
||||
for row in rows:
|
||||
payload = {
|
||||
"assertion_id": row.get("assertion_id"),
|
||||
"relation_id": row.get("assertion_id"),
|
||||
"source": row.get("source"),
|
||||
"source_type": row.get("source_type"),
|
||||
"source_canonical_id": row.get("source_canonical_id"),
|
||||
"relation": row.get("relation"),
|
||||
"relation_norm": row.get("relation"),
|
||||
"relation_category": row.get("relation_category"),
|
||||
"target": row.get("target"),
|
||||
"target_type": row.get("target_type"),
|
||||
"target_canonical_id": row.get("target_canonical_id"),
|
||||
"evidence": row.get("evidence"),
|
||||
"paper_id": row.get("paper_id"),
|
||||
"title": row.get("title"),
|
||||
"doi": row.get("doi"),
|
||||
"pmid": row.get("pmid"),
|
||||
"pmcid": row.get("pmcid"),
|
||||
"quality_score": row.get("score") or 0.0,
|
||||
}
|
||||
item = self.item_from_payload(payload, method, "canonical_seed")
|
||||
item.vector_score = float(row.get("score") or 0.8)
|
||||
item.rrf_score = max(float(row.get("paper_count") or 0) * 0.01, float(row.get("support_count") or 0) * 0.01, item.vector_score)
|
||||
output.append(item)
|
||||
return output
|
||||
|
||||
def item_from_payload(self, payload: Dict[str, Any], method: str, query: str) -> EvidenceItem:
|
||||
relation_id = _text(payload.get("relation_id") or payload.get("assertion_id"))
|
||||
metadata = payload.get("metadata") if isinstance(payload.get("metadata"), dict) else {}
|
||||
paper_id = _text(payload.get("paper_id") or payload.get("best_paper_id"))
|
||||
registry_metadata = getattr(self, "paper_metadata", {}).get(paper_id, {})
|
||||
return EvidenceItem(
|
||||
evidence_id="",
|
||||
kb="V16/V15",
|
||||
source=_text(payload.get("source")),
|
||||
relation=_text(payload.get("relation") or payload.get("relation_norm")),
|
||||
target=_text(payload.get("target")),
|
||||
evidence=_text(payload.get("evidence") or payload.get("best_evidence")),
|
||||
paper_id=paper_id,
|
||||
title=_text(payload.get("title") or payload.get("paper_title") or metadata.get("title") or registry_metadata.get("title")),
|
||||
pmid=_text(payload.get("pmid") or payload.get("pubmed_id") or metadata.get("pmid") or registry_metadata.get("pmid")),
|
||||
pmcid=_text(payload.get("pmcid") or metadata.get("pmcid") or registry_metadata.get("pmcid")),
|
||||
doi=_text(payload.get("doi") or metadata.get("doi") or registry_metadata.get("doi")),
|
||||
species=_text(payload.get("species") or metadata.get("species")),
|
||||
section=_text(payload.get("section") or metadata.get("section")),
|
||||
output_layer=_text(payload.get("output_layer") or "main_kg"),
|
||||
claim_scope=_text(payload.get("claim_scope") or metadata.get("claim_scope")),
|
||||
relation_id=relation_id,
|
||||
retrieval_methods=[method],
|
||||
retrieval_queries=[query],
|
||||
vector_score=float(payload.get("quality_score") or payload.get("best_score") or 0.0),
|
||||
rrf_score=float(payload.get("quality_score") or payload.get("best_score") or 0.0),
|
||||
raw=dict(payload),
|
||||
)
|
||||
|
||||
def extract_seeds(self, hypothesis: RetrievalHypothesis, items: List[EvidenceItem]) -> Tuple[List[str], List[str]]:
|
||||
ids: List[str] = []
|
||||
names: List[str] = []
|
||||
|
||||
def add_id(value: Any) -> None:
|
||||
text = _text(value)
|
||||
if text and text.startswith("cent_") and text not in ids:
|
||||
ids.append(text)
|
||||
|
||||
def add_name(value: Any) -> None:
|
||||
text = _norm(value)
|
||||
if 2 <= len(text) <= 160 and text not in names:
|
||||
names.append(text)
|
||||
|
||||
for entity in hypothesis.direct_question_entities + hypothesis.hypothesized_entities:
|
||||
add_name(entity.canonical_candidate)
|
||||
add_name(entity.mention)
|
||||
|
||||
for item in items[:50]:
|
||||
raw = item.raw or {}
|
||||
add_id(raw.get("source_canonical_id"))
|
||||
add_id(raw.get("target_canonical_id"))
|
||||
add_name(item.source)
|
||||
add_name(item.target)
|
||||
|
||||
return ids, names
|
||||
|
||||
@staticmethod
|
||||
def _ids_from_items(items: Iterable[EvidenceItem]) -> List[str]:
|
||||
ids: List[str] = []
|
||||
for item in items:
|
||||
raw = item.raw or {}
|
||||
for value in (raw.get("source_canonical_id"), raw.get("target_canonical_id")):
|
||||
text = _text(value)
|
||||
if text.startswith("cent_") and text not in ids:
|
||||
ids.append(text)
|
||||
return ids
|
||||
|
||||
@staticmethod
|
||||
def _flatten_queries(question: str, queries: Dict[str, List[str]]) -> List[str]:
|
||||
output: List[str] = []
|
||||
for value in [question]:
|
||||
if value and value not in output:
|
||||
output.append(value)
|
||||
for lane_values in queries.values():
|
||||
for value in lane_values:
|
||||
clean = _text(value)
|
||||
if clean and clean not in output:
|
||||
output.append(clean)
|
||||
return output[:10]
|
||||
|
||||
@staticmethod
|
||||
def dedup(items: List[EvidenceItem]) -> List[EvidenceItem]:
|
||||
by_id: Dict[str, EvidenceItem] = {}
|
||||
no_id: List[EvidenceItem] = []
|
||||
for item in items:
|
||||
key = item.relation_id
|
||||
if not key:
|
||||
no_id.append(item)
|
||||
continue
|
||||
existing = by_id.get(key)
|
||||
if existing is None:
|
||||
by_id[key] = item
|
||||
continue
|
||||
existing.retrieval_methods = list(dict.fromkeys(existing.retrieval_methods + item.retrieval_methods))
|
||||
existing.retrieval_queries = list(dict.fromkeys(existing.retrieval_queries + item.retrieval_queries))
|
||||
existing.vector_score = max(existing.vector_score, item.vector_score)
|
||||
existing.rrf_score = max(existing.rrf_score, item.rrf_score)
|
||||
if not existing.evidence and item.evidence:
|
||||
existing.evidence = item.evidence
|
||||
return list(by_id.values()) + no_id
|
||||
@@ -0,0 +1,187 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List
|
||||
|
||||
import yaml
|
||||
|
||||
from .schemas import HypothesisEntity, RetrievalHypothesis
|
||||
|
||||
|
||||
def _norm(value: str) -> str:
|
||||
return re.sub(r"\s+", " ", str(value or "").strip().lower())
|
||||
|
||||
|
||||
def _contains_alias(text: str, alias: str) -> bool:
|
||||
normalized_alias = _norm(alias)
|
||||
if not normalized_alias:
|
||||
return False
|
||||
if re.search(r"[\u4e00-\u9fff]", normalized_alias):
|
||||
return normalized_alias in text
|
||||
return bool(
|
||||
re.search(
|
||||
rf"(?<![A-Za-z0-9]){re.escape(normalized_alias)}(?![A-Za-z0-9])",
|
||||
text,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class DomainConfig:
|
||||
def __init__(self, alias_path: Path, taxonomy_path: Path):
|
||||
self.alias_path = alias_path
|
||||
self.taxonomy_path = taxonomy_path
|
||||
self.alias_entries: Dict[str, Dict[str, Any]] = {}
|
||||
self.alias_lookup: Dict[str, str] = {}
|
||||
self.taxonomy: Dict[str, List[str]] = {}
|
||||
self._load_aliases()
|
||||
self._load_taxonomy()
|
||||
|
||||
def _load_aliases(self) -> None:
|
||||
data = yaml.safe_load(self.alias_path.read_text(encoding="utf-8")) or {}
|
||||
for canonical, payload in data.items():
|
||||
payload = payload if isinstance(payload, dict) else {}
|
||||
aliases = [str(item) for item in payload.get("aliases", []) or []]
|
||||
aliases.append(str(canonical))
|
||||
raw_retrieval_label = payload.get("retrieval_label")
|
||||
retrieval_label = (
|
||||
raw_retrieval_label.strip()
|
||||
if isinstance(raw_retrieval_label, str)
|
||||
else ""
|
||||
)
|
||||
raw_retrieval_terms = payload.get("retrieval_terms")
|
||||
retrieval_terms = (
|
||||
[
|
||||
item.strip()
|
||||
for item in raw_retrieval_terms
|
||||
if isinstance(item, str) and item.strip()
|
||||
]
|
||||
if isinstance(raw_retrieval_terms, list)
|
||||
else []
|
||||
)
|
||||
entry = {
|
||||
"canonical": str(canonical),
|
||||
"type": str(payload.get("type") or ""),
|
||||
"aliases": list(dict.fromkeys(aliases)),
|
||||
"retrieval_label": retrieval_label,
|
||||
"retrieval_terms": list(dict.fromkeys(retrieval_terms)),
|
||||
"forbidden_expansions": [
|
||||
str(item)
|
||||
for item in payload.get("forbidden_expansions", []) or []
|
||||
],
|
||||
}
|
||||
self.alias_entries[str(canonical)] = entry
|
||||
for alias in entry["aliases"]:
|
||||
self.alias_lookup[_norm(alias)] = str(canonical)
|
||||
|
||||
def _load_taxonomy(self) -> None:
|
||||
data = yaml.safe_load(self.taxonomy_path.read_text(encoding="utf-8")) or {}
|
||||
for intent, payload in data.items():
|
||||
values: List[str] = []
|
||||
if isinstance(payload, list):
|
||||
values.extend(str(item) for item in payload)
|
||||
elif isinstance(payload, dict):
|
||||
for group in payload.values():
|
||||
if isinstance(group, list):
|
||||
values.extend(str(item) for item in group)
|
||||
self.taxonomy[str(intent)] = list(dict.fromkeys(values))
|
||||
|
||||
def canonicalize_entity(self, entity: HypothesisEntity) -> HypothesisEntity:
|
||||
canonical = self.alias_lookup.get(
|
||||
_norm(entity.canonical_candidate),
|
||||
self.alias_lookup.get(_norm(entity.mention), entity.canonical_candidate),
|
||||
)
|
||||
entry = self.alias_entries.get(canonical, {})
|
||||
entity.canonical_candidate = canonical
|
||||
if not entity.entity_type or entity.entity_type == "unknown":
|
||||
entity.entity_type = str(entry.get("type") or "unknown")
|
||||
if entry:
|
||||
entity.verification_status = "verified_alias"
|
||||
entity.matched_kb_entity = canonical
|
||||
entity.use_as_graph_seed = True
|
||||
elif entity.origin == "question":
|
||||
entity.verification_status = "question_direct_unverified"
|
||||
entity.use_as_graph_seed = True
|
||||
return entity
|
||||
|
||||
def normalize_hypothesis(self, hypothesis: RetrievalHypothesis) -> Dict[str, Any]:
|
||||
audit = []
|
||||
for entity in (
|
||||
hypothesis.direct_question_entities + hypothesis.hypothesized_entities
|
||||
):
|
||||
before = entity.canonical_candidate
|
||||
self.canonicalize_entity(entity)
|
||||
audit.append(
|
||||
{
|
||||
"mention": entity.mention,
|
||||
"before": before,
|
||||
"canonical": entity.canonical_candidate,
|
||||
"origin": entity.origin,
|
||||
"verification_status": entity.verification_status,
|
||||
"use_as_graph_seed": entity.use_as_graph_seed,
|
||||
}
|
||||
)
|
||||
return {"entities": audit, "alias_config": str(self.alias_path)}
|
||||
|
||||
def direct_entities_from_question(self, question: str) -> List[HypothesisEntity]:
|
||||
normalized_question = _norm(question)
|
||||
entities: List[HypothesisEntity] = []
|
||||
for canonical, entry in self.alias_entries.items():
|
||||
matched_alias = next(
|
||||
(
|
||||
alias
|
||||
for alias in sorted(entry["aliases"], key=len, reverse=True)
|
||||
if _contains_alias(normalized_question, alias)
|
||||
),
|
||||
"",
|
||||
)
|
||||
if not matched_alias:
|
||||
continue
|
||||
entities.append(
|
||||
HypothesisEntity(
|
||||
mention=matched_alias,
|
||||
canonical_candidate=canonical,
|
||||
entity_type=str(entry.get("type") or "unknown"),
|
||||
origin="question",
|
||||
confidence=1.0,
|
||||
verification_status="verified_alias",
|
||||
matched_kb_entity=canonical,
|
||||
use_as_graph_seed=True,
|
||||
)
|
||||
)
|
||||
return entities
|
||||
|
||||
def expanded_relations(self, intents: Iterable[str]) -> List[str]:
|
||||
output: List[str] = []
|
||||
for intent in intents:
|
||||
output.extend(self.taxonomy.get(str(intent), []))
|
||||
return list(dict.fromkeys(output))
|
||||
|
||||
def evidence_query_terms(
|
||||
self, entities: Iterable[HypothesisEntity]
|
||||
) -> List[str]:
|
||||
verified_entries = [
|
||||
self.alias_entries.get(entity.canonical_candidate, {})
|
||||
for entity in entities
|
||||
if entity.verification_status == "verified_alias"
|
||||
]
|
||||
evidence_terms = [
|
||||
term
|
||||
for entry in verified_entries
|
||||
for term in entry.get("retrieval_terms", [])
|
||||
]
|
||||
if not evidence_terms:
|
||||
return []
|
||||
labels = [
|
||||
entry.get("retrieval_label", "")
|
||||
for entry in verified_entries
|
||||
if entry.get("retrieval_label", "")
|
||||
]
|
||||
return list(dict.fromkeys(labels + evidence_terms))
|
||||
|
||||
def forbidden_terms(self, entities: Iterable[HypothesisEntity]) -> List[str]:
|
||||
output: List[str] = []
|
||||
for entity in entities:
|
||||
entry = self.alias_entries.get(entity.canonical_candidate, {})
|
||||
output.extend(entry.get("forbidden_expansions", []) or [])
|
||||
return list(dict.fromkeys(output))
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user